Main E2E Test Defects: Utility in Case of Failure

Utility in case of failure

A written test is useless when it fails. There are many reasons that could cause the signup flow to fail. Some of these include:

1. The host does not work.
2. The page does not respond.
3. One of the input fields is not rendered.
4. The button is not rendered.
5. The button does not trigger the correct Redux action.
6. The AJAX call does not start.
7. The AJAX call has the wrong request payload.
8. The API does not work and it does not respond.
9. The API returns the wrong payload.
10. The functions that manage the authentication do not perform the expected job.
11. The user already exists.
12. Everything is managed correctly but the redirect to the home page does not work.
13. The home page does not show the “No articles are here” text.
14. The home page shows a slightly different message to the user (ex: “Click on ‘New Post’ to add your first contribution”).

Implementation details errors

Some of the possible errors that can occur at the beginning of the test include:

5. The button does not trigger the correct Redux action.

6. The functions that manage the authentication do not do the expected job.

These errors fall under. “implementation details”. We are testing the whole front-end application from an outer perspective, meaning we are testing whether the user signup works, not how the signup works.

A more practical example: Say you are testing whether pushing the accelerator causes the car to move. To test this, you push the accelerator and make sure that the car moves! It doesn’t matter if the accelerator is wired or electronic. You are testing its effectiveness in getting the car to move, and not how the accelerator gets the car to move. The “how” will be tested with other types of tests.

Remember testing rules an E2E test can not provide useful feedback for a single function, and a unit test can not tell you if your app works. When applied to our signup flow, we are not testing why the form component does not receive the right dispatch prop or why it does not call it. We are just checking that the form works and allows the user to register.! Do the users care about Redux actions? No! They just want to register!

Error feedback

Imagine you are working on a new feature. You inadvertently break the signup flow and you discover as much once you run the existing E2E tests. This test identifies the regression, but what does it tell us about what is broken in the signup flow. Is the E2E test useful to identify what we need to fix?

Take a look at the feedback the Test Runner gives us for all the previously listed errors.

✅= Yes

❌= No

⚠️ = Somewhere in the middle between yes and no, it depends

Error	Test Runner feedback	Is it useful?
1. the host does not work		✅
2. the page does not respond		⚠️
3. one of the input fields is not rendered		❌
4. the button is not rendered		✅️
All the remaining errors: 6. the AJAX call does not start 7. the AJAX call has the wrong request payload 8. the API does not work and it does not respond 9. the API returns the wrong payload 11. the user already exists 12. everything is managed correctly but the redirect to the home does not work 13. the home page does not show the “No articles are here” text 14. the home page shows a slight different message to the user		❌

Take the most useless two errors and inverting the comparison, do you think that the feedback on the first column helps you understand what happened in the second column?

Test Runner feedback	Error
	3. one of the input fields is not rendered
	One of 6. the AJAX call does not start 7. the AJAX call has the wrong request payload 8. the API does not work and it does not respond 9. the API returns the wrong payload 11. the user already exists 12. everything is managed correctly but the redirect to the home does not work 13. the home page does not show the “No articles are here” text 14. the home page shows a slight different message to the user

The errors in the table tell us nothing about what happened.

What to do in case of failures

Tests aren’t only about utility, but also about preventing useless and long debugging processes. How do you proceed when a test fails? There are a couple steps you should take:

• Read the test logs.

• Look for the failing test and read the output error.

• If the output error speaks for itself, you can fix the regression you introduced. If the error is unclear, you open the test file.

• Identify the failing test and run it only if you have violated the “tests must not share the state” rule. You are going to waste a lot of time at this step.

• Re-run the test looking at the web app.

• Based on the error, there are a few more steps you can take as well:

  • If the error is a “visual one” (something not rendered etc.) then a look at the screenshots and videos that Cypress saves automatically could be enough.
  • If the error is not a visual (wrong DOM selectors, wrong AJAX payloads, etc.), you should open the DevTools, re-run the test, and inspect the DOM and network activities until you find the error.

This type of E2E test has no meaningful feedback and it requires you to debug it when it fails. remember, debugging a failing test is harder than debugging a standard script/app failure. If you are having to debug your tests to such an extreme extent, it is important to stop and rethink them..We will rethink this test in the following lessons.