jieba词频统计中去除停用词的问题

Question

import jieba
file_userDict = 'add_word_list.txt'
jieba.load_userdict(file_userDict)
import re
stop_file = "stopwordlist.txt"
try:
    stopword_list = open(stop_file,encoding ='utf-8')
except:
    stopword_list = []
    print("error in stop_file")
stop_list = []
for line in stopword_list:
    line = re.sub(u'\n|\\r', '', line)
    stop_list.append(line)
f=open("法律（新）\服务贸易\中华人民共和国海南自由贸易港法(FBM-CLI.1.5015177).txt","r",encoding='utf-8')
txt=f.read()
words = jieba.lcut(txt)
counts = {}
for word in words:
    if len(word) ==1:
        continue
    else:
        counts[word] = counts.get(word,0) + 1
items = list(counts.items())
items.sort(key=lambda x:x[1], reverse=True)
items

我想做个先去除停用词然后对文本进行分词并词频统计的代码，然而这个代码最后得出的词频统计并没有去除停用词，请问该怎么修改呢？

Answer 1

参考gpt和自己的思路，在代码中，你已经成功读取并生成了停用词列表stop_list，可以在统计词频时添加一个判断，如果单词出现在停用词列表中，则不加入词频统计。下面是修改后的代码：

import jieba
import re

file_userDict = 'add_word_list.txt'
jieba.load_userdict(file_userDict)

stop_file = "stopwordlist.txt"
try:
stopword_list = open(stop_file, encoding='utf-8')
except:
stopword_list = []
print("error in stop_file")
stop_list = []
for line in stopword_list:
line = re.sub(u'\n|\r', '', line)
stop_list.append(line)

f = open("法律（新）\服务贸易\中华人民共和国海南自由贸易港法(FBM-CLI.1.5015177).txt", "r", encoding='utf-8')
txt = f.read()
words = jieba.lcut(txt)
counts = {}
for word in words:
if len(word) == 1 or word in stop_list: # 判断单词长度是否为1或者是否出现在停用词列表中
continue
else:
counts[word] = counts.get(word, 0) + 1
items = list(counts.items())
items.sort(key=lambda x: x[1], reverse=True)
print(items) # 输出词频统计结果