Performance Testing

What questions can be answered with a Performance test ?

Functional tests answering a simple question: is it working, or not. Performance tests are simulating human users, so they can answer more sensitive questions:

How can the chatbot deal with many users?

Before publishing a bot, good to know what are its limits, and how it behaves beyond its limits. Is it possible to kill it? Does it recover after a hevy load?

Stress test can help you to answer this question.

Do we have some (memory) leak?

The conversation is stateless, so executing the same conversation on the same server should take about the same time. If the catbot starts a never ending process, or does not releases some resource like memory, or background storage, then the system will be slower after time, or will die suddenly.

Load test is for detecting this.

Are the conversations stateless?

Chatbots are nowdays mostly static, we got the same answer for the same qestion. So it is a good test to repeat a question. It is even possible, that first time it works well, second time we got error.

Every Performance test can detect this problem, because the nature of the Performance Tests is repeating a convo.

Are the conversations thread safe?

Many problems can be detected with parallel execution of a conversation. For example deadlock, wrong resource sharing, or incorrect synchronization.

Performance test is not the ultimate tool for those mistakes. It can happen that everything goes well 10 times parallel, then the next time we got an error. The system is too complex to do exact the same thing every time. So lets say Botium can accidentally such problems as side effect.

Every Performance test can detect this problem, but Stress test fits better for this.