Preparing for End-to-End Tests

We will continue to use Ginkgo and Gomega as our testing framework, but we will invoke multi-git as a command-line program and will not test at the Go-program level. Along the way, you will discover a bug and fix it. This demonstrates nicely that your multiple layers of tests are useful. Let’s get into it.

What is end-to-end testing?

End-to-end testing is arguably the most important kind of testing. If I can write just one kind of test, I would always choose end-to-end testing. An end-to-end test is a test that runs against the full-fledged system “from the outside”. This means the system is tested just like it is being used by users or other systems.

Within end-to-end tests, there are different flavors:

• Functional/acceptance testing
• Performance/stress/load testing
• Regression testing

Functional/acceptance tests verify that the system behaves as it is accepted to behave. If you want to be comprehensive, you may want to test more than just the results of queries that are returned to callers, but also verify internal state, persistent stores, and logs.

It is very important to test how the system behaves under various failure modes and not just test the happy path.

Performance, stress, and load testing are very important for large systems. If there are SLAs (service level agreements) in place, those SLAs are actually business requirements and should be considered part of acceptance testing. However, even if there is no official SLA, every system has to be able to handle its load, and you need to test it. Even if you run in the cloud, with its infinite capacity, and your system is designed to dynamically take advantage of cloud resources, you still need to test your system’s performance. First, it’s difficult to build a fully dynamic system. Second, you can run into cloud provider limits and quotas, and finally, you don’t want to just throw hardware at your problems because you have to pay for it.

Regression testing is all about ensuring that changes to your code don’t break expected behavior. The tests run the new and old code on the same inputs (sometimes randomly generated inputs) to ensure you get the same results.

What’s the difference between integration testing and end-to-end testing?

In the previous lesson, we focused on unit tests, and in this lesson, we are focusing on end-to-end tests. Integration tests live in the middle. They test several units or modules that interact with each other and sometimes also involve third-party dependencies. They are very useful for complex systems. Integration tests often require a lot of configuration and setup to hook up the modules under test and mock the other dependencies. End-to-end tests on the other hand, test the complete system from the outside.

Writing end-to-end tests for Multi-git

For multi-git, we will focus on functional/acceptance tests. As a thin layer on top of git, there isn’t a lot of value from performance tests. Regression tests are not very useful either as this is mostly a personal project for demonstration purposes.

How to write an end-to-end test?

The idea is to access the program like a user would and then verify the response. For a command-line program like multi-git, this process involves:

• Setting proper environment variables
• Invoking multi-git with proper command-line arguments
• Checking the output

To perform all these tasks, we can add a helper function to the helpers package.

Extending the helpers package

The helpers package already has two useful functions, CreateDir() and AddFiles(), that we’ve used in the repo_manager unit tests. Let’s add a RunMultiGit() function. This function takes an input the git command to run, the ignoreErrors Boolean flag that determines if multi-git should bail out on the first error or continue to other repos, the root directory multi-git operates on, and the list of sub-directories to work on. It returns the output of the command for each repo or an error:

func RunMultiGit(command string, 
                 ignoreErrors bool, 
                 mgRoot string, 
                 mgRepos string) (output string, err error) {

RunMultiGit() requires that mg, the name of the multi-git executable be in your PATH. It checks it using the which mg command. If it’s not in the PATH, you get an error:

    out, err := exec.Command("which", "mg").CombinedOutput()
    if err != nil {
        return
    }

    if len(out) == 0 {
        err = errors.New("mg is not in the PATH")
        return
    }

The next part is preparing the command-line arguments for multi-git. The --command argument is mandatory. If ignoreErrors is true, it adds the --ignoreErrors command-line argument:

    components := []string{"--command", command}
    if ignoreErrors {
        components = append(components, "--ignore-errors")
    }

The next part prepares the exec.Command for mg and then adds the MG_ROOT and MG_REPOS environment variables to the command environment.

    cmd := exec.Command("mg", components...)
    cmd.Env = os.Environ()
    cmd.Env = append(cmd.Env, "MG_ROOT="+mgRoot, "MG_REPOS="+mgRepos)

Next, it runs the command and captures both the standard output and standard error using the CombinedOutput() method. Finally, it converts the output from a byte array to a string and returns.

    out, err = cmd.CombinedOutput()
    output = string(out)
    return
}

Note that RunMultiGit() doesn’t deal with directories or files. This is the responsibility of the tests themselves.