weixin_39928940
weixin_39928940
2021-01-10 12:29

Why does my utterance's score keep decreasing?

Hello,

I have built an app where users constantly train new intents/utterances & answers into the model. And then the model constantly re-trains itself every 4 minutes.

I have noticed that the scores of utterances that users ask keep decreasing.

For example, one user created an utterance/intent for "What is your favorite color?". Typing the exact utterance used to return ~.9 score. Now it returns ~.3 score.

What is the cause of this and how can I reliably solve this problem?

该提问来源于开源项目:axa-group/nlp.js

  • 点赞
  • 写回答
  • 关注问题
  • 收藏
  • 复制链接分享
  • 邀请回答

7条回答

  • weixin_39915700 weixin_39915700 4月前

    Hello,

    I tested to retrain 100 times one model with 52 intents. Conclussions about that retraining: - Having only Logistic Regression Classifier, scores are always the same - Having also neural, scores are slightly different, but not decreasing, simply moving near the same values

    About your problem, without the corpus to check, it's impossible to see what's happening. But I can guess that: - You are having so many utterances - Utterances that should be in the same intent are in different intents. - You train intents with only one utterance.

    That's why usully nobody let's a chatbot to be trained by the users. The trainers are choosen by those that knows the business of the chatbot. Because something like this can happen:

    user1> train "What is your favourite color" as "color" user2> Train "What is your favourite color" as "favourite_color" user3> Train "What is your color favourite" as "whatever" ...

    So perhaps you can share your corpus to check.

    点赞 评论 复制链接分享
  • weixin_39928940 weixin_39928940 4月前

    You are right, I was actually about to reply that my post was wrong, the math was off and it was something wrong I did.

    I would really hate to challenge the integrity of this great package. I'm terribly sorry for my mistake.

    I still do have utterance related accuracy issues but it's not as severe as I originally reported.

    Also, i did not make any code modifications.

    I'm going to look at the numbers again and re-post my issue in a little bit.

    点赞 评论 复制链接分享
  • weixin_39928940 weixin_39928940 4月前

    Ok, to conclude this issue, it seems short messages results in very small score values (~.2 average). But since it's universally affected to all intents and consistent, it's not a major problem. It was only a problem when I expected all messages to result in similar .7 ~ .9 scores.

    Thank you very much for your work in this package.

    点赞 评论 复制链接分享
  • weixin_39928940 weixin_39928940 4月前

    To add to this issue/question ... I'm not sure if my score is decreasing, but it certainly looks like my intents with "conditions" have a significantly lower score value... even though the utterance matches letter by letter...

    Any guidance on how and why the score is so low would be deeply appreciated.

    点赞 评论 复制链接分享
  • weixin_39901439 weixin_39901439 4月前

    I noticed the same thing and think it's because the more utterances you add, the closer they become to each another, like "what is your favorite car", "what is your favorite color". It dilutes the score for these ones. In my example above, you have 2 "what is your favorite", and scoring is computed based on words presence (weights). May be in this case, the best is to use an entity and only keep one "what is your favorite %userfavorite%".

    点赞 评论 复制链接分享
  • weixin_39928940 weixin_39928940 4月前

    Problem is, I cannot keep just one. I have many users who constantly add new utterances and intents. And a lot of them have common utterances.

    As for dilution, I understand that but a drop from .9 to .1 ~ .3 is too severe.

    点赞 评论 复制链接分享
  • weixin_39901439 weixin_39901439 4月前

    May be I didn't pinpoint your problem correctly. Can you share your code modifications?

    点赞 评论 复制链接分享

相关推荐