CountVectorizer

Question

用CountVectorizer做情感分析，报错为：InvalidParameterError: The 'stop_words' parameter of CountVectorizer must be a str among {'english'}, an instance of 'list' or None. 求大神指点！

from sklearn.feature_extraction.text import CountVectorizer

def get_custom_stopwords(stop_words_file):
    with open(stop_words_file) as f:
        stopwords = f.read()
    stopwords_list = stopwords.split('\n')
    custom_stopwords_list = [i for i in stopwords_list]
    return custom_stopwords_list

stop_words_file = '哈工大停用词表.txt'  #设置停用词
stopwords = get_custom_stopwords(stop_words_file)

vect = CountVectorizer(max_df = 0.8, 
                       min_df = 3, 
                       token_pattern=u'(?u)\\b[^\\d\\W]\\w+\\b', 
                       stop_words=frozenset(stopwords))

#划分数据集
X = data['cut_comment']
y = data.sentiment

from sklearn.model_selection import train_test_split

X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=22)

test = pd.DataFrame(vect.fit_transform(X_train).toarray(), columns=vect.get_feature_names())
test.head()

Answer 1

引自免费微信小程序：皆我百晓生

在使用CountVectorizer时，你尝试从一个自定义的停用词文件中加载停用词列表，并将其传递给stop_words参数。然而，stop_words参数期望接收的是字符串（'english'）、列表实例或None，而你的代码将停用词处理为了一个frozenset对象。

修正这个问题的方法是直接将读取到的自定义停用词列表传递给stop_words参数，而不是转换为frozenset。请参考以下修改后的代码：

from sklearn.feature_extraction.text import CountVectorizer

def get_custom_stopwords(stop_words_file):
    with open(stop_words_file, 'r', encoding='utf-8') as f:  # 增加编码方式以确保正确读取文本文件
        stopwords = f.read()
    stopwords_list = stopwords.split('\n')
    custom_stopwords_list = [i for i in stopwords_list if i]  # 移除可能存在的空行
    return custom_stopwords_list

stop_words_file = '哈工大停用词表.txt'  # 设置停用词
stopwords = get_custom_stopwords(stop_words_file)

vect = CountVectorizer(max_df=0.8,
                       min_df=3,
                       token_pattern=u'(?u)\\b[^\\d\\W]\\w+\\b',
                       stop_words=stopwords)  # 将stopwords修改为list类型

现在vect实例将会正确使用你自定义的停用词列表来进行文本特征提取。