我正在通过 IMAP 下载消息。接下来,我将解析消息添加到 MongoDB 中,但我遇到了一个问题,因为 MongoDB 只支持 UTF 8, 我想把所有编码都转换成 UTF 8的话该怎么做?
我知道可以转换为二进制,但我必须有正常的文本,因为我必须在数据库中搜索短语。除非——我可以搜索二进制的正常文本吗?
我正在通过 IMAP 下载消息。接下来,我将解析消息添加到 MongoDB 中,但我遇到了一个问题,因为 MongoDB 只支持 UTF 8, 我想把所有编码都转换成 UTF 8的话该怎么做?
我知道可以转换为二进制,但我必须有正常的文本,因为我必须在数据库中搜索短语。除非——我可以搜索二进制的正常文本吗?
I'm using the go-charset
project to do this: https://code.google.com/p/go-charset/
It's pretty straightforward, you create a reader from a charset and it translates to utf-8 automatically. example from the library:
r, err := charset.NewReader(strings.NewReader("\xa35 for Pepp\xe9"), "latin1")
if err != nil {
log.Fatal(err)
}
result, err := ioutil.ReadAll(r)
if err != nil {
log.Fatal(err)
}
fmt.Printf("%s
", result) //outputs £5 for Peppé
Now, in my case I know the charset because it comes from web pages and I read the headers/meta tags. If you need to detect the charset automatically by heuristics, you'll need another library for that, such as this one: https://github.com/saintfish/chardet
I haven't used it but it also looks pretty simple to use:
detector := chardet.NewTextDetector()
result, err := detector.DetectBest(some_text)
if err == nil {
fmt.Printf(
"Detected charset is %s, language is %s",
result.Charset,
result.Language)
}