The interaction between humans and machines is continuously evolving and improving. In the search for the improvement of this communication, numerous studies try to shorten the distance between natural language (languages we speak) and algorithms (computer codes).
It’s no wonder that increasingly complex and sophisticated natural language processing (NLP) systems have emerged.
We have already told you about our work – started in July 2017 – on the creation of Bothub, a global, collaborative and open source PLN system developed under the expectation of being transformed into the biggest and best in the world.
You probably know about renowned software such as IBM Watson, Wit.ai, and Google Cloud Platform, but what you still don’t know is that the “newborn” Bothub has achieved impressive rates and a more efficient interpretation of Brazilian Portuguese (PT-BR) than that of Wit.ai’s Portuguese (PT).
How did we discover this? Follow the post and understand everything!
Tests
We decided to analyze the effectiveness of our system and we thought that the best way to do this would be to compare it with some consolidated platform in the market. We chose Wit.ai because it’s one of the tools most used by developers. In addition to belonging to Facebook, one of the largest investors in chatbots on the globe, and be connected to Facebook Messenger.
We started with the elaboration of 100 phrases in PT-BR, our native language. They covered 10 themes: geography, health, soccer, pets, games, cinema, elections, automobiles, financial and food. For each category, 10 phrases with definitions of entities (species of “keywords” that help extract useful information from the phrase) and intentions were distributed.
Then we developed a script in Python (a programming language) capable of training and testing phrases automatically.
The 100 sentences formulated underwent training at Bothub and Wit.ai. Then, without determining intentions and entities, 100 new sentences were also tested on both platforms.
From there, two CSV files were generated, which can be read in Excel or other spreadsheet readers, with the results of the predictions of the systems separately.
These files presented columns with the tested phrases, expected intentions, found intentions, percentages of intentions´ reliability, results of comparisons between expected and found intentions, expected entities, found entities, results of comparisons between expected and found entities, and percentages of entities´reliability.
Just below the table was the total number of sentences tested and the final percentage of correct predictions.
Results
In the most recent test, in September of this year, we reached the stimulating result of 62% of correct predictions of Bothub, against 38% of Wit.ai.
An important point to note was the result achieved by Bothub in each intention. The “financial” category hit 9 predictions. Followed by “cinema” and “automobiles” that got 8 hits. With the same intentions, Wit.ai hit 6, 4 and 8, respectively.
Phrases that found the “financial” intention, not identified by Wit.ai, such as “I have already paid”, “Could you reverse the difference of my purchase?” and “I would like to cancel my purchase”, highlight this category as our best performance.
Do you want to test too? Access https://github.com/Ilhasoft/bothub/tree/master/benchmark.
Share it with friends!😉
