Why you should do traffic analysis

Chatbot model too large?  Struggling with performance? Consider doing a traffic analysis

Sometimes a chatbot can get out of hand – perhaps it’s because new intents regularly get added to the chatbot model, simply because the odd question is asked that you want the chatbot to answer.  Or maybe a chatbot is taking the place of a whole area of the business and it has a large domain to begin with (and perhaps the management team stipulate that all subject areas need to be covered in the chatbot).

The problem is, large chatbot models can be very hard to train and maintain.  They often tend to be very finely balanced, so by making just a small tweak to the training data of one intent, it can have disastrous knock-on effects to other intents in the model.  So chatbot trainers need to regularly spend a lot of time fixing these regressions.  And in the meantime, the chatbot model is losing performance.

STOP!

Wouldn’t it be best to have a smaller, better performing chatbot than a “Jack of all trades, master of none” chatbot?

HOW?

By doing a traffic analysis to find out what your most frequently returned intents are from your customer logs in a set period, say 3-6 months’ worth of logs.  You can then concentrate efforts on improving the performance in these short tail (frequently returned) intents and consider cutting out some of your long tail (infrequently returned) intents. 

We’ve done numerous traffic analyses for our clients and a typical analysis shows that more than 50% of traffic to their chatbot is covered by only 10-20 intents.  So, we always recommend these short tail intents should be focused on as a priority.  We also look at the intents that are barely returning any hits at all in the given period, and recommend these ones are removed, as the traffic analysis proves they’re not really needed.

In summary, you need to think about where best to put your efforts into making improvements.  The traffic analysis exercise helps to create a clear plan of action - prioritise fixing weaknesses in your short tail intents and discard some of your long tail intents; a smaller chatbot model is easier to train and maintain.  Once you’re getting good performance out of the chatbot, you can then consider adding some of your long tail intents back in IF they’re really needed.