duanhe6464 2019-02-28 13:22
浏览 547
已采纳

特殊äö字符打破UTF-8编码

A user on my site inputted special characters into a text field: ä ö

These apparently are not the same ä ö characters I can input from my keyboard because when I paste them into Programmer's Notepad, they split into two: a¨ o¨

On my site's server side I have a PHP script that identifies illegal special characters in user input and highligts them in an html error message with preg_replace.

The character splitting happens there too so I get a normal letter a and o with a weird lone xCC character that breaks the UTF-8 string encoding and json_encode function fails as a result.

What would be the best way to handle these characters? Should I try to replace the special ä ö chars and replace them with the regular ones or can I somehow catch the broken UTF-8 chars and remove or replace them?

  • 写回答

1条回答 默认 最新

  • douniewei6346 2019-02-28 13:37
    关注

    It's not that these characters have broken the encoding, it's just that Unicode is really complicated.

    Commonly used accented letters have their own code points in the Unicode standard, in this case:

    • U+00E4 "LATIN SMALL LETTER A WITH DIAERESIS"
    • U+00F6 "LATIN SMALL LETTER O WITH DIAERESIS"

    However, to avoid encoding every possibility, particularly when multiple diacritics (accents) need to be placed on the same letter, Unicode includes "combining diacritics", such as:

    • U+0308 "COMBINING DIAERESIS"

    When placed after the code point for a normal letter, these code points add a diacritic to it when displaying.

    As you've seen, this means there's two different ways to represent the same letter. To help with this, Unicode includes "normalization forms" defined in an annex to the Unicode standard:

    • Normalization Form D (NFD): Canonical Decomposition
    • Normalization Form C (NFC): Canonical Decomposition, followed by Canonical Composition
    • Normalization Form KD (NFKD): Compatibility Decomposition
    • Normalization Form KC (NFKC): Compatibility Decomposition, followed by Canonical Composition

    Ignoring the "Compatibility" forms for now, we have two options:

    • Decomposition, which uses combining diacritics as often as possible
    • Composition, which uses specific code points as often as possible

    So one possibility is to convert your input into NFC, which in PHP can be achieved with the Normalizer class in the intl extension.

    However, not all combinations can be normalised to a form with no separate diacritics, so this doesn't solve all your problems. You'll also need to look at what characters exactly you want to allow, probably by matching Unicode character properties.

    You might also want to learn about "grapheme clusters" and use the relevant PHP functions. A "grapheme cluster", or just "grapheme", is what most readers will think of as "a character" - e.g. a letter with all its diacritics, or a full ideogram.

    本回答被题主选为最佳回答 , 对您是否有帮助呢?
    评论

报告相同问题?

悬赏问题

  • ¥15 如何用QDomDocument读取内容为空格的xml数据
  • ¥15 请阅读下面代码,帮我修改下代码
  • ¥15 关于#microsoft#的问题:电脑启动后不显示桌面图标和窗口,除任务栏外无法操作任何东西
  • ¥15 如何输入百度,显示本地下载的html文件页面,地址栏还显示百度的地址
  • ¥15 通过kinect制作换装程序但是服装不贴合(标签-ar)
  • ¥20 matlab如何绘制三维瀑布图
  • ¥15 关于用abap来解决动态规划的问题,但是要求输出索引值,这个是难点
  • ¥15 在ISIS中什么是IP从地址
  • ¥15 压测时,并发量过高时,响应时间出现尖刺
  • ¥15 关于vmprotect3.8.4虚拟文件一项