m0_61104322
2022-01-07 20:46
采纳率: 88.2%
浏览 40

深度学习XLnet模型出现问题,如何解决

在运行XLnet进行文本分类的时候,出现了下面的问题,该如何解决,请大家赐教:

def tokenize(text):
    tokenized = tokenizer(text,
              padding=True,
              truncation=True,
              return_tensors='tf',
              return_token_type_ids=False,
              return_attention_mask=False,
              verbose=True)
    return tokenized

x_train_enc = tokenize(x_train)
x_test_enc = tokenize(x_test)

y_train_enc_one_hot = to_categorical(y_train_enc, num_classes=num_classes)
y_test_enc_one_hot = to_categorical(y_test_enc, num_classes=num_classes)


ValueError Traceback (most recent call last)
/tmp/ipykernel_10776/2763515379.py in
11 return tokenized
12
---> 13 x_train_enc = tokenize(x_train)
14 x_test_enc = tokenize(x_test)
15

/tmp/ipykernel_10776/2763515379.py in tokenize(text)
8 return_token_type_ids=False,
9 return_attention_mask=False,
---> 10 verbose=True)
11 return tokenized
12

/environment/miniconda3/lib/python3.7/site-packages/transformers/tokenization_utils_base.py in call(self, text, text_pair, add_special_tokens, padding, truncation, max_length, stride, is_split_into_words, pad_to_multiple_of, return_tensors, return_token_type_ids, return_attention_mask, return_overflowing_tokens, return_special_tokens_mask, return_offsets_mapping, return_length, verbose, **kwargs)
2391 if not _is_valid_text_input(text):
2392 raise ValueError(
-> 2393 "text input must of type str (single example), List[str] (batch or single pretokenized example) "
2394 "or List[List[str]] (batch of pretokenized examples)."
2395 )

ValueError: text input must of type str (single example), List[str] (batch or single pretokenized example) or List[List[str]] (batch of pretokenized examples).

  • 写回答
  • 好问题 提建议
  • 追加酬金
  • 关注问题
  • 邀请回答

1条回答 默认 最新

  • CSDN专家-HGJ 2022-01-08 13:49
    最佳回答

    参数text应该是一个字符串类型或字符串列表形式。检查一下传入的参数text。
    试试改一下:
    for i, text in enumerate(df['abstract2'].tolist())

    评论
    解决 1 无用
    打赏 举报 编辑记录

相关推荐 更多相似问题