douqiao7958
douqiao7958
2015-03-25 12:31
浏览 541
已采纳

如何在Golang中使用utf8将[] rune编码为[] byte?

So it's really easy to decode a []byte into a []rune (simply cast to string, then cast to []rune works very nicely, I'm assuming it defaults to utf8 and with filler bytes for invalids). My question is - how are you suppose to decode this []rune back to []byte in utf8 form?

Am I missing something or do I have manually call EncodeRune for every single rune in my []rune? Surely there is an encoder that I can simply pass a Writer to.

图片转代码服务由CSDN问答提供 功能建议

因此,将 [] byte 解码为 []真的很容易 符文(只需强制转换为 string ,然后强制转换为 [] rune 效果非常好,我假设它默认为utf8并带有无效字符的填充字节) 。 我的问题是-您打算如何以utf8格式将此 [] rune 解码回 [] byte

我 缺少某些东西还是我为我中的每个符文手动调用 EncodeRune []符文? 当然,有一个编码器可以简单地将 Writer 传递给。

  • 点赞
  • 写回答
  • 关注问题
  • 收藏
  • 邀请回答

1条回答 默认 最新

  • duanhuo7441
    duanhuo7441 2015-03-25 12:35
    已采纳

    You can simply convert a rune slice ([]rune) to string which you can convert back to []byte.

    Example:

    rs := []rune{'H', 'e', 'l', 'l', 'o', ' ', '世', '界'}
    bs := []byte(string(rs))
    
    fmt.Printf("%s
    ", bs)
    fmt.Println(string(bs))
    

    Output (try it on the Go Playground):

    Hello 世界
    Hello 世界
    

    The Go Specification: Conversions mentions this case explicitly: Conversions to and from a string type, point #3:

    Converting a slice of runes to a string type yields a string that is the concatenation of the individual rune values converted to strings.

    Note that the above solution–although may be the simplest–might not be the most efficient. And the reason is because it first creates a string value that will hold a "copy" of the runes in UTF-8 encoded form, then it copies the backing slice of the string to the result byte slice (a copy has to be made because string values are immutable, and if the result slice would share data with the string, we would be able to modify the content of the string; for details, see golang: []byte(string) vs []byte(*string) and Immutable string and pointer address).

    Note that a smart compiler could detect that the intermediate string value cannot be referred to and thus eliminate one of the copies.

    We may get better performance by allocating a single byte slice, and encode the runes one-by-one into it. And we're done. To easily do this, we may call the unicode/utf8 package to our aid:

    rs := []rune{'H', 'e', 'l', 'l', 'o', ' ', '世', '界'}
    bs := make([]byte, len(rs)*utf8.UTFMax)
    
    count := 0
    for _, r := range rs {
        count += utf8.EncodeRune(bs[count:], r)
    }
    bs = bs[:count]
    
    fmt.Printf("%s
    ", bs)
    fmt.Println(string(bs))
    

    Output of the above is the same. Try it on the Go Playground.

    Note that in order to create the result slice, we had to guess how big the result slice will be. We used a maximum estimation, which is the number of runes multiplied by the max number of bytes a rune may be encoded to (utf8.UTFMax). In most cases, this will be bigger than needed.

    We may create a third version where we first calculate the exact size needed. For this, we may use the utf8.RuneLen() function. The gain will be that we will not "waste" memory, and we won't have to do a final slicing (bs = bs[:count]).

    Let's compare the performances. The 3 functions (3 versions) to compare:

    func runesToUTF8(rs []rune) []byte {
        return []byte(string(rs))
    }
    
    func runesToUTF8Manual(rs []rune) []byte {
        bs := make([]byte, len(rs)*utf8.UTFMax)
    
        count := 0
        for _, r := range rs {
            count += utf8.EncodeRune(bs[count:], r)
        }
    
        return bs[:count]
    }
    
    func runesToUTF8Manual2(rs []rune) []byte {
        size := 0
        for _, r := range rs {
            size += utf8.RuneLen(r)
        }
    
        bs := make([]byte, size)
    
        count := 0
        for _, r := range rs {
            count += utf8.EncodeRune(bs[count:], r)
        }
    
        return bs
    }
    

    And the benchmarking code:

    var rs = []rune{'H', 'e', 'l', 'l', 'o', ' ', '世', '界'}
    
    func BenchmarkFirst(b *testing.B) {
        for i := 0; i < b.N; i++ {
            runesToUTF8(rs)
        }
    }
    
    func BenchmarkSecond(b *testing.B) {
        for i := 0; i < b.N; i++ {
            runesToUTF8Manual(rs)
        }
    }
    
    func BenchmarkThird(b *testing.B) {
        for i := 0; i < b.N; i++ {
            runesToUTF8Manual2(rs)
        }
    }
    

    And the results:

    BenchmarkFirst-4        20000000                95.8 ns/op
    BenchmarkSecond-4       20000000                84.4 ns/op
    BenchmarkThird-4        20000000                81.2 ns/op
    

    As suspected, the second version is faster and the third version is the fastest, although the performance gain is not huge. In general the first, simplest solution is preferred, but if this is in some critical part of your app (and is executed many-many times), the third version might worth it to be used.

    点赞 评论

相关推荐