Learn how to design a test automation framework for automating web-based applications.

Test Automation

Testing the user interface for any application is generally time-consuming when done for each new release. UI test automation is the most common and effective way to simulate the actual user journey and business use cases for any application.

In this module, we’ll learn to design a test automation framework from scratch to automate any web-based application using Java, Selenium, and TestNG. We’ll start by learning the basics of Selenium and TestNG and then learn about build management, logging, and reporting, all of which are key concepts for any QA engineer or developer. By the end of this module, we will successfully create a UI test automation framework.

Design a Test Automation Framework with Selenium and Java

### List of default TestNG reports

TestNG adds a few basic reports of the test run by default. The following are some reporters:

* `org.testng.reporters.SuiteHTMLReporter` -- generates a HTML reporter for suites.
* `org.testng.reporters.FailedReporter` -- generates ***testng-failed.xml*** containing only the failed tests.
* `org.testng.reporters.XMLReporter` -- generates summary of test output in xml format.
* `org.testng.reporters.EmailableReporter2` -- generates a single-page HTML report ***emailable-report.html*** of the test results.
* `org.testng.reporters.JUnitReportReporter` -- generates Junit test output xml for each of the test suites. 


### Custom reporters

We can have our own reporters by implementing `org.testng.IReporter`.

Sample Implementation of `IReporter` for generating a summary report of the test run:

```java
package com.example;

import java.io.File;
import java.io.FileWriter;
import java.io.IOException;
import java.io.Writer;
import java.text.DecimalFormat;
import java.util.List;

import org.testng.IReporter;
import org.testng.ISuite;
import org.testng.ITestContext;
import org.testng.xml.XmlSuite;

public class SummaryReport implements IReporter {

	private DecimalFormat decimalFormat = new DecimalFormat("00.##");

	@Override
	public void generateReport(List<XmlSuite> xmlSuites, List<ISuite> suites, String outputDirectory) {

		File file = new File(outputDirectory, "summary.html");
		try (Writer writer = new FileWriter(file)) {
			writer.write("<html><body>");
			String title = "<table border='1' style='background-color:yellow;border:1px dotted black;width:80%;border-collapse:collapse;'><caption style='margin: 1em; font-weight: bolder'>Test Execution Summary Report</caption><tbody>";
			writer.write(title);
			writer.write(
					"<tr><th style='padding:3px;'align='center'>Suite Name</th><th style='padding:3px;'align='center'>Total</th><th style='padding:3px;'align='center'>Passed</th><th style='padding:3px;'align='center'>Failed</th><th style='padding:3px;'align='center'>Skipped</th><th style='padding:3px;'align='center'>% Tests Passed</th><th style='padding:3px;'align='center'>Total Execution Time (Sec)</th></tr>");

			for (ISuite suite : suites) {
				suite.getResults().forEach((k, v) -> {

					ITestContext context = v.getTestContext();
					Integer passed = context.getPassedTests().size();
					Integer failed = context.getFailedTests().size();
					Integer skipped = context.getSkippedTests().size();
					Integer total = passed + failed + skipped;
					Double execTime = (context.getEndDate().getTime() - context.getStartDate().getTime()) * 0.001;

					if (total > 0) {
						try {
							String percent = decimalFormat
									.format((passed.doubleValue() / total.doubleValue()) * 100.0D);
							writer.write(String.format(
									"<tr><td style='text-align: center'>%s</td><td style='text-align: center'>%s</td><td style='text-align: center'>%s</td><td style='text-align: center'>%s</td><td style='text-align: center'>%s</td><td style='text-align: center'>%s</td><td style='text-align: center'>%s</td></tr>",
									suite.getName(), total, passed, failed, skipped, percent,
									decimalFormat.format(execTime)));
						} catch (IOException e) {
							e.printStackTrace();
						}
					}
				});
			}
			writer.write("</tbody></table></body></html>");
		} catch (IOException ex) {
			ex.printStackTrace();
		}
		System.out.println("writing summary report to " + file.getAbsolutePath());
	}
}
```

This custom implementation of `IReporter` needs to be added to ***testng.xml*** like:

```xml
<listeners> 
    <listener class-name="com.example.SummaryReportListener" />
</listeners> 
```
Or in test class like:
```
@Listeners({ com.example.SummaryReportListener.class })
public class TestClass {}
```

Adding listeners is already discussed in the [TestNG Listener](https://www.educative.io/courses/test-automation-framework-selenium-java/using-listeners/) lesson.

The default test output folder is `${user.dir}/test-output`.

Custom Reporter produces HTML report ***summary.html*** like below:

![](/api/collection/5455247108472832/4603030201696256/page/6347598035156992/image/6452618382540800)

Below is the sample of ***emailable-report.html***:


# List of default TestNG reports

TestNG adds a few basic reports of the test run by default. The following are some reporters:

* `org.testng.reporters.SuiteHTMLReporter` -- generates a HTML reporter for suites.
* `org.testng.reporters.FailedReporter` -- generates ***testng-failed.xml*** containing only the failed tests.
* `org.testng.reporters.XMLReporter` -- generates summary of test output in xml format.
* `org.testng.reporters.EmailableReporter2` -- generates a single-page HTML report ***emailable-report.html*** of the test results.
* `org.testng.reporters.JUnitReportReporter` -- generates Junit test output xml for each of the test suites. 


# Custom reporters

We can have our own reporters by implementing `org.testng.IReporter`.

Sample Implementation of `IReporter` for generating a summary report of the test run:

```java
package com.example;

import java.io.File;
import java.io.FileWriter;
import java.io.IOException;
import java.io.Writer;
import java.text.DecimalFormat;
import java.util.List;

import org.testng.IReporter;
import org.testng.ISuite;
import org.testng.ITestContext;
import org.testng.xml.XmlSuite;

public class SummaryReport implements IReporter {

	private DecimalFormat decimalFormat = new DecimalFormat("00.##");

	@Override
	public void generateReport(List<XmlSuite> xmlSuites, List<ISuite> suites, String outputDirectory) {

		File file = new File(outputDirectory, "summary.html");
		try (Writer writer = new FileWriter(file)) {
			writer.write("<html><body>");
			String title = "<table border='1' style='background-color:yellow;border:1px dotted black;width:80%;border-collapse:collapse;'><caption style='margin: 1em; font-weight: bolder'>Test Execution Summary Report</caption><tbody>";
			writer.write(title);
			writer.write(
					"<tr><th style='padding:3px;'align='center'>Suite Name</th><th style='padding:3px;'align='center'>Total</th><th style='padding:3px;'align='center'>Passed</th><th style='padding:3px;'align='center'>Failed</th><th style='padding:3px;'align='center'>Skipped</th><th style='padding:3px;'align='center'>% Tests Passed</th><th style='padding:3px;'align='center'>Total Execution Time (Sec)</th></tr>");

			for (ISuite suite : suites) {
				suite.getResults().forEach((k, v) -> {

					ITestContext context = v.getTestContext();
					Integer passed = context.getPassedTests().size();
					Integer failed = context.getFailedTests().size();
					Integer skipped = context.getSkippedTests().size();
					Integer total = passed + failed + skipped;
					Double execTime = (context.getEndDate().getTime() - context.getStartDate().getTime()) * 0.001;

					if (total > 0) {
						try {
							String percent = decimalFormat
									.format((passed.doubleValue() / total.doubleValue()) * 100.0D);
							writer.write(String.format(
									"<tr><td style='text-align: center'>%s</td><td style='text-align: center'>%s</td><td style='text-align: center'>%s</td><td style='text-align: center'>%s</td><td style='text-align: center'>%s</td><td style='text-align: center'>%s</td><td style='text-align: center'>%s</td></tr>",
									suite.getName(), total, passed, failed, skipped, percent,
									decimalFormat.format(execTime)));
						} catch (IOException e) {
							e.printStackTrace();
						}
					}
				});
			}
			writer.write("</tbody></table></body></html>");
		} catch (IOException ex) {
			ex.printStackTrace();
		}
		System.out.println("writing summary report to " + file.getAbsolutePath());
	}
}
```

This custom implementation of `IReporter` needs to be added to ***testng.xml*** like:

```xml
<listeners> 
    <listener class-name="com.example.SummaryReportListener" />
</listeners> 
```
Or in test class like:
```
@Listeners({ com.example.SummaryReportListener.class })
public class TestClass {}
```

Adding listeners is already discussed in the [TestNG Listener](https://www.educative.io/courses/test-automation-framework-selenium-java/using-listeners/) lesson.

The default test output folder is `${user.dir}/test-output`.

Custom Reporter produces HTML report ***summary.html*** like below:

![](/api/collection/5455247108472832/4603030201696256/page/6347598035156992/image/6452618382540800)

Below is the sample of ***emailable-report.html***:


# List of default TestNG reports

TestNG adds a few basic reports of the test run by default. The following are some reporters:

* `org.testng.reporters.SuiteHTMLReporter` -- generates a HTML reporter for suites.
* `org.testng.reporters.FailedReporter` -- generates ***testng-failed.xml*** containing only the failed tests.
* `org.testng.reporters.XMLReporter` -- generates summary of test output in xml format.
* `org.testng.reporters.EmailableReporter2` -- generates a single-page HTML report ***emailable-report.html*** of the test results.
* `org.testng.reporters.JUnitReportReporter` -- generates Junit test output xml for each of the test suites. 


# Custom reporters

We can have our own reporters by implementing `org.testng.IReporter`.

Sample Implementation of `IReporter` for generating a summary report of the test run:

```java
package com.example;

import java.io.File;
import java.io.FileWriter;
import java.io.IOException;
import java.io.Writer;
import java.text.DecimalFormat;
import java.util.List;

import org.testng.IReporter;
import org.testng.ISuite;
import org.testng.ITestContext;
import org.testng.xml.XmlSuite;

public class SummaryReport implements IReporter {

	private DecimalFormat decimalFormat = new DecimalFormat("00.##");

	@Override
	public void generateReport(List<XmlSuite> xmlSuites, List<ISuite> suites, String outputDirectory) {

		File file = new File(outputDirectory, "summary.html");
		try (Writer writer = new FileWriter(file)) {
			writer.write("<html><body>");
			String title = "<table border='1'><caption>Test Execution Summary Report</caption><tbody>";
			writer.write(title);
			writer.write(
					"<tr><thalign='center'>Suite Name</th><thalign='center'>Total</th><thalign='center'>Passed</th><thalign='center'>Failed</th><thalign='center'>Skipped</th><thalign='center'>% Tests Passed</th><thalign='center'>Total Execution Time (Sec)</th></tr>");

			for (ISuite suite : suites) {
				suite.getResults().forEach((k, v) -> {

					ITestContext context = v.getTestContext();
					Integer passed = context.getPassedTests().size();
					Integer failed = context.getFailedTests().size();
					Integer skipped = context.getSkippedTests().size();
					Integer total = passed + failed + skipped;
					Double execTime = (context.getEndDate().getTime() - context.getStartDate().getTime()) * 0.001;

					if (total > 0) {
						try {
							String percent = decimalFormat
									.format((passed.doubleValue() / total.doubleValue()) * 100.0D);
							writer.write(String.format(
									"<tr><td>%s</td><td>%s</td><td>%s</td><td>%s</td><td>%s</td><td>%s</td><td>%s</td></tr>",
									suite.getName(), total, passed, failed, skipped, percent,
									decimalFormat.format(execTime)));
						} catch (IOException e) {
							e.printStackTrace();
						}
					}
				});
			}
			writer.write("</tbody></table></body></html>");
		} catch (IOException ex) {
			ex.printStackTrace();
		}
		System.out.println("writing summary report to " + file.getAbsolutePath());
	}
}
```

This custom implementation of `IReporter` needs to be added to ***testng.xml*** like:

```xml
<listeners> 
    <listener class-name="com.example.SummaryReportListener" />
</listeners> 
```
Or in test class like:
```
@Listeners({ com.example.SummaryReportListener.class })
public class TestClass {}
```

Adding listeners is already discussed in the [TestNG Listener](https://www.educative.io/courses/test-automation-framework-selenium-java/using-listeners/) lesson.

The default test output folder is `＄{user.dir}/test-output`.

Custom Reporter produces HTML report ***summary.html*** like below:

![](/api/collection/5455247108472832/4603030201696256/page/6347598035156992/image/6452618382540800)

Below is the sample of ***emailable-report.html***:


In this lesson, you'll understand TestNG Reporting.

Introduction to UI Test Automation

Selenium WebDriver Basics

TestNG Framework

Build Management

Configuration Management

Logging

Reporting

Designing the UI Test Automation Framework

Sample UI Test Framework

Conclusion

TestNG Reporting

List of default TestNG reports #

Custom reporters #