Part 2 of my Lex chatbot journey is about making the chatbot smarter with Q&A, to read previous about getting started visit part 1.

The use case is to make the chatbot discussed more helpful.

Options considered for powering the document search:

  1. Solr or OpenSearch on EC2 - whilst a multicore setup Solr would have provided a robust solution for the search API felt that setting up an EC2 instance might be excessive for use case. Interestingly OpenSearch doesn't recommend multicore.
  2. tfjs-models - TensorflowJS tfjs-models/qna and tfjs-models/universal-sentence-encoder models looked promising and returned great results, the only issue was slow speed when run with @tensorflow/tfjs and not @tensorflow/tfjs-core. Assumption is that @tensorflow/tfjs will be required for AWS Lambda because of large unpacked size of @tensorflow/tfjs-core.
  3. string-similarity - with an unpacked size of 11kB and quick run speed string-similarity  appears also a good fit for powering the document search. Whilst results worked they felt a little bit "off". With string-similarity additional work will be required to make Q&A search feel a little bit more natural with technique like selectively preserving only stemmed keywords in the query/index.
  4. LunrJS - with the tagline "a bit like Solr, but much smaller and not as bright" it feels like LunrJS might be a good fit. It also took a couple of moments for the penny to drop - Lunr is a play on Solr 😉

And the winner is => LunrJS

Currently the chat Q&A document search on influencerTips is powered by LunrJS.

Share a comment with a suggestion.