在Golang中将文本输入标准化为ASCII

Question

I am building a small tool which parses a user's input and finds common pitfalls in writing and flags them so the user can improve their text. So far everything works well except for text that has curly quotes compared to normal ASCII straight quotes. I have a hack now which will do a string replacement for opening (and closing) single curly quotes and double opening (and close) curly quotes like so:

cleanedData := bytes.Replace([]byte(data), []byte("’"), []byte("'"), -1)

I feel like there must be a better way to handle this in the stdlib so I can also convert other non-ascii characters to an ascii equivalent. Any help would be greatly appreciated.

Answer 1

The strings.Map function looks to me like what you want.

I don't know of a generic 'ToAscii' type function, but Map has a nice approach for mapping runes to other runes.

Example (updated):

func main() {
    data := "Hello “Frank” or ‹François› as you like to be ‘called’"
    fmt.Printf("Original: %s
", data)
    cleanedData := strings.Map(normalize, data)
    fmt.Printf("Cleaned: %s
", cleanedData)
}

func normalize(in rune) rune {
    switch in {
    case '“', '‹', '”', '›':
        return '"'
    case '‘', '’':
        return '\''
    }
    return in
}

Output:

Original: Hello “Frank” or ‹François› as you like to be ‘called’
Cleaned: Hello "Frank" or "François" as you like to be 'called'