doupingdiao3546 2017-07-06 11:20
浏览 97
已采纳

io.Reader和涉及CSV文件的换行问题

I have an application which deals with CSV's being delivered via RabbitMQ from many different upstream applications - typically 5000-15,000 rows per file. Most of the time it works great. However a couple of these upstream applications are old (12-15 years) and the people who wrote them are long gone.

I'm unable to read CSV files from these older aplications due to the line breaks. I'm finding this a bit weird as the line breaks see to map to UTF-8 Carriage Returns (http://www.fileformat.info/info/unicode/char/000d/index.htm). Typically the app reads in only the headers from those older files and nothing else.

If I open one of these files in a text editor and save as utf-8 encoding overwriting the exiting file then it works with no issues at all.

Things I've tried I expected to work:

-Using a Reader:

    ba := make([]byte, 262144000)
    if _, err := file.Read(ba); err != nil {
        return nil, err
    }
    ba = bytes.Trim(ba, "\x00")
    bb := bytes.NewBuffer(ba)
    reader := csv.NewReader(bb)
    records, err := reader.ReadAll()
    if err != nil {
        return nil, err
    }

-Using the Scanner to read line by line (get a bufio.Scanner: token too long)

    scanner := bufio.NewScanner(file)
    var bb bytes.Buffer
    for scanner.Scan() {
        bb.WriteString(fmt.Sprintf("%s
", scanner.Text()))
    }

    // check for errors
    if err = scanner.Err(); err != nil {
        return nil, err
    }


reader := csv.NewReader(&bb)
records, err := reader.ReadAll()
if err != nil {
    return nil, err
}

Things I tried I expected not to work (and didn't):

  • Writing file contents to a new file (.txt) and reading the file back in (including running dos2unix against the created txt file)
  • Reading file into a standard string (hoping Go's UTF-8 encoding would magically kick in which of course it doesn't)
  • Reading file to Rune slice, then transforming to a string via byte slice

I'm aware of the https://godoc.org/golang.org/x/text/transform package but not too sure of a viable approach - it looks like the src encoding needs to be known to transform.

Am I stupidly overlooking something? Are there any suggestions how to transform these files into UTF-8 or update the line endings without knowing the file encoding whilst keeping the application working for all the other valid CSV files being delivered? Are there any options that don't involve me going byte to byte and doing a bytes.Replace I've not considered? I'm hoping there's something really obvious I've overlooked.

Apologies - I can't share the CSV files for obvious reasons.

  • 写回答

2条回答 默认 最新

  • duanbei7005 2018-02-27 00:27
    关注

    For anyone who's stumbled on this and wants an answer that doesn't involve strings.Replace, here's a method that wraps an io.Reader to replace solo carriage returns. It could probably be more efficient, but works better with huge files than a strings.Replace-based solution.

    https://gist.github.com/b5/78edaae9e6a4248ea06b45d089c277d6

    // ReplaceSoloCarriageReturns wraps an io.Reader, on every call of Read it
    // for instances of lonely  replacing them with 
     before returning to the end customer
    // lots of files in the wild will come without "proper" line breaks, which irritates go's
    // standard csv package. This'll fix by wrapping the reader passed to csv.NewReader:
    //    rdr, err := csv.NewReader(ReplaceSoloCarriageReturns(r))
    //
    func ReplaceSoloCarriageReturns(data io.Reader) io.Reader {
        return crlfReplaceReader{
            rdr: bufio.NewReader(data),
        }
    }
    
    // crlfReplaceReader wraps a reader
    type crlfReplaceReader struct {
        rdr *bufio.Reader
    }
    
    // Read implements io.Reader for crlfReplaceReader
    func (c crlfReplaceReader) Read(p []byte) (n int, err error) {
        if len(p) == 0 {
            return
        }
    
        for {
            if n == len(p) {
                return
            }
    
            p[n], err = c.rdr.ReadByte()
            if err != nil {
                return
            }
    
            // any time we encounter  & still have space, check to see if 
     follows
            // if next char is not 
    , add it in manually
            if p[n] == '' && n < len(p) {
                if pk, err := c.rdr.Peek(1); (err == nil && pk[0] != '
    ') || (err != nil && err.Error() == io.EOF.Error()) {
                    n++
                    p[n] = '
    '
                }
            }
    
            n++
        }
        return
    }
    
    本回答被题主选为最佳回答 , 对您是否有帮助呢?
    评论
查看更多回答(1条)

报告相同问题?

悬赏问题

  • ¥15 用联想小新14Pro
  • ¥15 multisim中关于74ls192n和DSWPK开关仿真图分析(减法计数器)
  • ¥15 基于8255的交通灯设计
  • ¥15 w3wp,exe 中发生未处理的 Microsoft ,NETFramework 异常。
  • ¥20 C51单片机程序及仿真(加减器)
  • ¥15 AQWA | 水动力分析 二阶波浪力
  • ¥15 c语言题目:成绩管理系统
  • ¥22 可否请教一下matlab未定义与 'double' 类型的输入参数相对应的函数 'finverse'是什么情况
  • ¥15 利用数据集,用机器学习模型对股市预测,私我
  • ¥15 tauri + vue3 如何实现在一个页面上局部加载外部网页