Tutorial: First Steps in Botium Box

So, you followed the installation instructions and you can see the Botium Dashboard, but you don’t know how to start ? Follow the steps in this tutorial to get your first Botium Box test project up and running!

This tutorial is available on Youtube as well!

Step 1: Connect to a Chatbot

The first step is to connect Botium Box to your chatbot. Open the Chatbots menu and enter the Register a new chatbot section.

Have a look at the Connector/Chatbot Technology selection, here you can see what chatbot technologies are supported by Botium Box. You can get an overview and more details for each technology here.

For the sake of simplicity, select to Botium Sample Chatbot (Echo) - an integrated chatbot you can use for start exploring Botium Box features.

Click Save and afterwards on Live Chat to have a quick chat with the chatbot. This is for testing if the connection parameters (called Capabilities in Botium) actually allow talking to the chatbot or if there is configuration missing or wrong.

After starting the connection with the Connect button, you can start to chat with the chatbot - enter anyhting you want, the Echo bot will reply.

Don’t close this window, we will now use the chat you just had as a test case for automated replay.

Step 2: Compose a Test Set

In Botium, a Test Set is a collection of test cases. A test case is basically a predefined conversation with your chatbot. In the Live Chat window, click on the Save as Test Case button to save the conversation you just had as test case. Let’s create a new test set, see screenshot below, and when ready click on OK:

Afterwards, navigate to the test set we just created by using the Test Sets menu and clicking on the First Steps Test Set in the list.

In the Local Repository section you can see a single test case, the one we saved before. Click to open and see the BotiumScript code for this test case. BotiumScript is a very easy to learn text based domain-specific file format (or programming language) for defining test case for a conversational interface (or chatbot).

The BotiumScript for our first test case looks like this:

1 2 3 4 5 6 7 8 9 10 11 12 13 say hi and ask for help #me hi bot #bot default You said: hi bot #me help #bot default Recognized commands: help, fail, add to cart, show cart, clear cart, weak intent, duplicate intent, buttons, show me buttons, show me some buttons, give me buttons, picture, show me a picture, give me a picture, card, show me a card, give me a card, attachment, audio attachment

You can read it like:

  • The name of the test case is say hi and ask for help

  • As a starter, send the text hi bot to the chatbot

  • We expect the chatbot to answer with the text You said: hi bot - if the answer from the chatbot is different, the chatbot doesn’t behave as expected, and the test case is failed

  • Then, send the text help to the chatbot

  • And again we expect a very specific answer from the chatbot

This is a simple text-based test case, BotiumScripts supports many more features like

Step 3: Run the Test

So, we are connected to the chatbot and we recorded a first test case. Now let’s run this test case.

Go to the Test Results menu and in the Start Test Session section, select our First Steps Chatbot and our First Steps Test Set, and click on Start Test Session.

You can now see Botium doing the hard work in the background, and after several seconds, the test result is ready. As expected, the test case succeeded. On the bottom of the screen, you can see all the executed test cases - for our first test set, we only had one single test case.

Click on the test case to open the conversation transcript. This looks very similar to the live chat we had earlier - but the conversation happened without any manual intervention.

Step 4: Extend the Test Case

Our test case covered exactly one possible greeting a user can send - hi bot, nothing else. Now let’s extend this test case with other possible greetings like hello and good morning

In BotiumScript these synonyms are called utterances. Navigate to your test set and select the option Write New Test Script.

Select Utterances File as Script Type, enter the list and save it.

1 2 3 4 UTT_USER_HELLO hi hello good afternoon

We have to prepare an utterances list for the bot responses as well:

1 2 3 4 UTT_BOT_HELLO You said: hi You said: hello You said: good afternoon

While the first list contains all the possible things to send to the bot, the second list contains all the possible things the bot could answer.

The first line is the reference code of the utterance, you need it in the next step for including this list in a convo file. You can use any wording you like, but we recommend:

  • use UPPERCASE letters

  • start it with UTT_USER (for user utterances) or UTT_BOT (for bot utterances)

  • No whitespace

Read more about utterances here.

Now open the say hi and ask for help convo again and change it like this - note that we now used the reference code for the utterances list:

1 2 3 4 5 6 7 8 9 10 11 12 13 say hi and ask for help #me UTT_USER_HELLO #bot default UTT_BOT_HELLO #me help #bot default Recognized commands: help, fail, add to cart, show cart, clear cart, weak intent, duplicate intent, buttons, show me buttons, show me some buttons, give me buttons, picture, show me a picture, give me a picture, card, show me a card, give me a card, attachment, audio attachment

To make Botium Box consider the utterances files, you have to enable the Expand Conversations options in the Advanced Scripting Settings. See here for details.

Running the test set again as in Step 3, you will now see that there are actually 3 test cases with 3 conversation transcript - one for each possible greeting from UTT_USER_HELLO.


So before you actually start writing more test cases, let’s create a new test project.

Step 5: Create a Test Project

In the previous step, you had to select the chatbot, select the test set and only then Botium was able to start the test run. A test project is a permanent link between a chatbot and one or more test sets in Botium Box.

Click the Quickstart menu, enter a name for the test project and select the First Steps Chatbot, afterwards proceed to the next screen.

In the second screen, select the First Steps Test Set, and proceed to the final screen.

On the final screen, click the Save button to be directed to the test project screen. You can also navigate there with the Test Projects menu.

When clicking the Save and Start Testing button, the test project is created and the first test run is immediately started, navigating you to the test results screen you know from before.

In the test project view, you can now see all test results for this test project, you can start a test session, you can do performance testing with your chatbot, and you can integrate Botium Box with your CI/CD pipeline.

Step 6: Connect to a real chatbot (import sample project).

There is a Github Repository available with some example configurations for most important chatbot technologies available. For example, with this configuration you will connect to a Microsoft Bot Framework chatbot. Follow the instructions to import this sample project and start chatting with a real chatbot.

Next Steps

Now it is time to connect to a real chatbot and to write real test cases.