切分语料脚本报错、请问代码要如何修改呢？

Question

 

Answer 1

 

你用这个试试。如果文件内容不对会提示出来，

如果中文是乱码，把 encoding='utf-8' 改成 encoding='gbk' 试试

 

import sys
def cut2(fpath, new_data_dir, nsrc='zh', ntgt='en'):
    fp = open(fpath, 'r', encoding='utf-8')
    src_fp = open(new_data_dir+'raw.'+nsrc, 'w', encoding='utf-8')
    tgt_fp = open(new_data_dir+'raw.'+ntgt, 'w', encoding='utf-8')
    for line in fp.readlines():
        li = line.replace('\n', '').split('\t')
        if len(li)!=2:
            print("出错的行：",line)
            continue
        tgt_line = li[0]
        src_line = li[1]
        src_fp.write(src_line + '\n')
        tgt_fp.write(tgt_line + '\n')
    src_fp.close()
    tgt_fp.close()