训练集文本分类预处理

Question

怎么给训练集，测试集去除特殊符号，去停用词，提前关键词呢？像训练集分词一样的代码：
X_train_word=[jieba.cut(words ,cut_all=False)for words in X_train]
X_train_cut=[" ".join(word)for word in X_train_word]
X_train_cut[:1]
训练集、测试集里有多个txt文件，没篇TXT文件里有一篇文章

Answer 1

读取文档为字符串，使用re.sub剔除不需要的特殊符号，然后再剔除停用词，参考例子：

import re
import jieba
data = 'Are these datasets? yes, they are datesets for demonstrating. 怎么给训练集，测试集去除特殊符号，去除停用词，提取关键词呢？像训练集分词一样的代码。'
stopwords=['are','for','的','呢']
data = re.sub('[^\u4e00-\u9fa5A-Za-z0-9]+',' ',data)
c = jieba.cut(data,cut_all=False)
a=list(set([w for w in c if w not in stopwords and w!=' ']))
print(a)