november 2020

How did I end up with testing Google Assistant with Botium?

During my daily software tests I verify a lot of SaaS apps and platforms.

Examples are Chatbots and virtual assistants.

And recently, clients asked me more about testing Google Assistant.

This can be done manually. This takes a long time and errors can occur when testing the same cases over a longer period of time. 

Is it also possible to use test automation?

Yes, with the help of Botium.

This article shows my first experiences when testing Google Assistant automatically with Botium.

But first what is Google Assistant?

What is Google Assistant?

Google Assistant is Google’s voice assistant. By voice commands, voice searching, and voice-activated device control you can finish a number of tasks after you’ve said the “OK Google” or “Hey Google” wake words. Conversational interactions through text or speech is its main goal, making Google Assistant a chatbot.

After the wake words you can start talking to Google without using a ‘trigger word’. Google listens and gives a response.

Google can even recognize different voices, knowing who is talking to it and responding accordingly. 

You can also ask for multiple things in one sentence.

Great, now we know what Google Assistant is. 

But do we want to test Google Assistant manually – with our own voice, or multiple voices – or can this testing be done automatically? 

Yes, by using Botium, the tool for Testing, training and quality assurance for chatbots.

What is Botium?

Selenium is the de-facto-standard for testing web applications. 

Appium is the de-facto-standard for testing smartphone applications. 

Botium is the de-facto-standard for testing conversational AI. 

And just as Selenium and Appium, Botium is free and Open Source, and available on Github.

I can tell you a lot about Botium’s architecture, but that’s out of scope for this article.

Next to testing the conversation flow of a chatbot you can do a lot more:

• Testing NLP model of a chatbot
• E2E testing of a chatbot based on Selenium and Appium
• Load- and Stress testing
• GDPR testing
• Security testing
• CI/CD integration (Jenkins, Bamboo, Azure DevOps Pipelines, IBM Toolchain, …)
• and many more

For now we will focus on testing Google Assistant with Botium.

How can I test Google Assistant with Botium?

Instead of thinking how I could test Google Assistant with Botium I did a very modern thing.

I Googled the question.

And I got lucky. Botium already made a video on this with the appropriate name ‘Setting Up a Google Assistant Project in Botium Box’

Awesome, now I can copy this completely for my own testing.

The steps in Google Actions, Assistant and Botium are very straightforward.

And soon I was making my own Botium tests and creating scripts for testing Google Assistant.

But, as with all software, it is not always as easy as it seems.

My experiences with Testing Google Assistant with Botium

From what I saw when testing Google Assistant, Botium as a tool is very intuitive.

What I really like is that you can see and hear (!)  the results as they occur when using Google Assistant. Just search in your test result for the following links.

The sound and screen-links overlap, but that’s a cosmetic issue.

The important thing is, it works. 

You can hear the text as it is displayed by using the play button as seen above.

And by clicking the weblink you are forwarded to a page resembling the results you get when using Google Assistant.

One of the things I noticed when testing Google Assistant it can be different in their response than what you expected.

For example, see this result after saying ‘Hallo’ to Google Assistant:

Always update your testcases regularly, Google Assistant is changing fast, and as you can see, it even gets more personalized.

Also, what happens when the answer to your question can be random? Better said, the answer can be given in a different sequence.

Google Assistant will give appropriate prompts, depending on the question you ask.

But see what happens when I ask Google Assistant via Botium the following question:

‘Who is Jack Nicholson’?

Great, the testcase worked fine, but let’s see what happens when I repeat this testcase?

In the second test different buttons appear in the result, failing the testcase.

Now you can delete the expected test results from your Botium code in your testcase, but remember you are testing a voice assistant and Google Assistant returns these buttons so you can click them for more information or an answer to a question.

Therefore in Botium you really have to define your testcases and their expected results to avoid situations like this.

Wrap up

This article showed my first experience testing Google Assistant with Botium.

These results are not exhaustive, I only mentioned the findings important when testing Google Assistant with Botium for the first time.

Other future test cases could be multiple voice testing, asking multiple questions at the same time etc. But that’s for a future article.

Botium is intuitive, easy to use and can be used for different testing processes with chatbots.

As with every tool, you have to stay critical as a real tester should be.

Test automating a chatbot is never easy, but with Botium you have a great tool to work with.

It should be in every chatbot tester’s toolbox!