Wrapping Up

Acceptance test limitations

Acceptance tests are very powerful when used properly, but they can also lead to various problems. These problems are manageable, but they could end up taking more of our time than desired in an extensive test suite. However, the result is worth it because we can be more confident that our application works appropriately end-to-end. The two major problems affecting acceptance tests are speed and maintainability.

Acceptance tests flex the entire application stack—a browser starts up, executes tasks, navigates to one or more pages, and then shuts down. This process is more expensive than a traditional test that doesn’t leverage a browser. A large acceptance test suite could take many times longer to run than a large unit/integration test suite, so we may want to run our acceptance tests nightly or on-demand rather than with every single build of our application. It might appear that this problem would only affect large applications, but the performance cost of tests can quickly add up in smaller applications too.

The maintainability of an acceptance test suite can be challenging to achieve due to the brittleness of front-end interfaces. It’s common for a design to evolve, for CSS classes to change, and for the order of elements to shift. Any of these occurrences will most likely cause tests to break in the suite. If the suite takes a while to run and is not run on every change, the breakages can add up. A large front-end redesign might involve changing all the existing acceptance tests. There are strategies to deal with the inherent maintenance issues of an acceptance suite, but the problems will always exist in some form. It’s outside the scope of this course to cover maintainable acceptance test strategies, but we can find resources online that can help us with this.

Despite the challenge of building and maintaining a robust acceptance test suite, the result can be worth it. It’s often the goal of QA teams to have a complete acceptance test suite, but it can be difficult if the application doesn’t start with acceptance tests.

Note: We’ll need to weigh the costs and benefits to decide if it’s the right choice and time to build an acceptance suite for our application.

Putting it all together

Software systems are difficult to write correctly, and real-time systems are even more difficult to write correctly. The challenge of persistent connections that must run for a long time without failure adds to the difficulty of developing a real-time application. Acceptance tests help ensure that applications we develop are deployed with minimal bugs.

There are multiple approaches we can take in the quality assurance process. The most accessible approach is to behave like a user while checking that the application works as expected. Simple things like going forward or backward in the browser history, putting a computer to sleep, or experiencing an internet disconnection could cause problems in an application. A different approach to QA testing is to force issues that might be very rare, such as a database going down or random Elixir processes crashing. When testing an application, follow a simple framework to keep ourself honest and focused in the test:

• Define the test.
• Write steps for the test.
• Write expectations for the test.
• Then, execute the test.

Manual acceptance testing is extremely valuable, but it can also be tedious and time-consuming. We can use WebDriver based automation tests via the Hound library to write automated and repeatable end-to-end tests. These tests can be challenging to write and maintain, but they are the strongest way to repeatedly guarantee that our application works as expected in different scenarios. In practice, we are likely to use a mixture of both testing strategies to cover an application entirely.

Now that we’re thinking like a user and trying to break our applications before our users do, we’re going to build a more advanced real-time feature into our application.