dongpangzan6425
2014-12-04 15:07
采纳率: 100%
浏览 686

如何在Go中将所有编码都转换为UTF 8?

我正在通过 IMAP 下载消息。接下来,我将解析消息添加到 MongoDB 中,但我遇到了一个问题,因为 MongoDB 只支持 UTF 8, 我想把所有编码都转换成 UTF 8的话该怎么做?

我知道可以转换为二进制,但我必须有正常的文本,因为我必须在数据库中搜索短语。除非——我可以搜索二进制的正常文本吗?

图片转代码服务由CSDN问答提供 功能建议

我正在通过IMAP下载邮件。 接下来,我将已解析的消息添加到MongoDB中。 我有一个问题,因为MongoDB仅支持UTF8。而且我想将任何编码转换为UTF8。代码多种多样。 如何将每个字符串转换为UTF 8?

我知道可以转换为二进制,但是我必须使用普通文本,因为我必须在数据库中搜索短语。 除非,我可以用二进制搜索普通文本吗?

  • 写回答
  • 关注问题
  • 收藏
  • 邀请回答

3条回答 默认 最新

  • doulierong0334 2014-12-04 15:37
    已采纳

    I'm using the go-charset project to do this: https://code.google.com/p/go-charset/

    It's pretty straightforward, you create a reader from a charset and it translates to utf-8 automatically. example from the library:

    r, err := charset.NewReader(strings.NewReader("\xa35 for Pepp\xe9"), "latin1")
    if err != nil {
        log.Fatal(err)
    }
    result, err := ioutil.ReadAll(r)
    if err != nil {
        log.Fatal(err)
    }
    fmt.Printf("%s
    ", result)  //outputs £5 for Peppé
    

    Now, in my case I know the charset because it comes from web pages and I read the headers/meta tags. If you need to detect the charset automatically by heuristics, you'll need another library for that, such as this one: https://github.com/saintfish/chardet

    I haven't used it but it also looks pretty simple to use:

    detector := chardet.NewTextDetector()
    result, err := detector.DetectBest(some_text)
    if err == nil {
        fmt.Printf(
            "Detected charset is %s, language is %s",
            result.Charset,
            result.Language)
    }
    
    已采纳该答案
    打赏 评论
  • douliang6563 2014-12-10 07:15

    I've found a better package, which uses iconv. Usage is trivial, it is described in the documentation. For example:

    output,_ := iconv.ConvertString("Hello World!", "windows-1252", "utf-8")
    

    Link to the package: https://github.com/djimenez/iconv-go

    打赏 评论
  • doujuxin7392 2017-06-01 03:57

    charset.NewReader in package golang.org/x/net/html/charset can't deal with encoding gb2312. charset.NewReaderLabel can deal with it.

    import  (
        "io/ioutil"
        "golang.org/x/net/html/charset"
    )
    
    func convrtToUTF8(str string, origEncoding string) string {
        strBytes := []byte(str)
        byteReader := bytes.NewReader(strBytes)
        reader, _ := charset.NewReaderLabel(origEncoding, byteReader)
        strBytes, _ = ioutil.ReadAll(reader)
        return string(strBytes)
    }
    
    打赏 评论

相关推荐 更多相似问题