Gain insights into property-based testing with PropEr in Erlang. Delve into foundational principles, common frameworks, custom data generators, and applying testing in realistic projects for robust code validation.

Property Based Testing with PropEr in Erlang_468 x 60 copy.png

propert-based-testing.tar.gz

Erlang

Elixer

Erlang-Chap2

Elixir-Chap2

Erlang-Shell

erlang-user

elixir-user

elixir-shell

elrang-eunit

erlang-chap5-employee

elixir-cover_test

erlang-cover_test

erlang-chap10-databaseinit

erlang-chap10-user

elixir-chap10-user

erlang-chap10-shell

elixir-chap10-shell

erlang-chap10-dbsetup

elixir-chap10-dbSetup

erlang-chap5-templating

erlang-chap5-last

elixir-chap5-last

erlang-chap10-default

elixir-chap10-default

elixir-chap10-default-copy

Erlang-Chap2-copy

Elixir-Chap2-copy

Erlang-Shell-copy

elixir-shell-copy

erlang-default-force

elixir-default-force

erlang-test-shell

elxir-test-shell

Property-based testing relies on specifying some property of code, rather than unit tests which specify the expected output in response to some inputs. In this course, you’ll cover the concepts you need to get started, up to the point where you feel confident enough to use the most advanced features of PropEr with Erlang.

You’ll start smoothly with the basic and foundational principles of property-based testing. From there, you will cover some of the common frameworks, how to think in properties, how to write your own custom data generators, and more. You'll also see how you can use Property-Based Testing in a realistic project.

By the time you finish this course, you will be comfortable testing real-world applications using properties.

Property-Based Testing with PropEr in Erlang

> **Order of failures:** \
Property tests are probabilistic. If you are following along, you may find different issues, or the same issues in a different order, than those in this text. This is normal, and rerunning the property a few times may yield different errors each time.

# Code
Let’s start off with the code and make fixes to it as we go through the lesson.

> **Note:** To run an updated version of the code, press "Run", and then call `rebar3 proper` in the terminal that opens. 

import React from 'react';
import { shallow, configure } from 'enzyme';
import Adapter from 'enzyme-adapter-react-16';

configure({ adapter: new Adapter() });

import HelloWorld from './app';

var TestResult = function() {
    this.succeeded = false;
    this.reason = "";
    this.input = "";
    this.expected_output = "";
    this.actual_output = "";
}

export const executeTests = function() {

  var results = [];

  result = new TestResult();
  result.input = 'HelloWorld Component';
  result.expected_output = "span containing text 'Hello World'"

  let wrapper = shallow(<HelloWorld />);

  // Call your Challenge function here.

  let type = wrapper.type();
  let testSuccessful = true;
  let failureReason;

  if (type !== 'span') {
    testSuccessful = false;
    failureReason = "You need to render exactly one span HTML element";
  } else if (wrapper.props().children != "Hello World") {
     testSuccessful = false;
     failureReason = "You have rendered wrong message in your span element";
  }

  result.actual_output = wrapper.html();

  if (testSuccessful) {
    result.succeeded = true;
    result.reason = "Succeeded"
  } else {
    result.succeeded = false;
    result.reason = failureReason;
  }

  results.push(result);

  return results;
}


React

Dart

GoJS React

Typescript React

Vue.js

# Understanding the bug

Running the tests then fails on Unicode values of `0x00`, meaning that the protocol likely uses null-terminated strings, which causes failures. Anyone using this PostgreSQL driver needs to protect themselves against such strings.

For now, we’ll assume that those invalid Unicode values should either be filtered somewhere else or that a bunch of other properties or unit tests will handle these, and we’ll instead work around it by making sure our generators don’t generate that data anymore. In fact, we’ll find a lot of other troublesome characters: `%` and `_` will influence PostgreSQL search in ways that `string:find/2` wouldn’t, and `\` will mess with escaping in SQL whereas our model won’t care.

# Refining the tests

We have to ask ourselves whether what we want to test here is the minutiae of the SQL string handling, or whether these strings are a tool we use to validate the state transitions of our stateful model. This is the point where you can decide to split your property into two distinct properties:

1. One property tests the search patterns explicitly to see if there are risky things happening with special characters.
2. One property tests the state transitions and various matches.

We’ll focus on the second one by ignoring all kinds of special characters in searches, titles, and authors.
```erlang
title() -> friendly_unicode().

author() -> friendly_unicode().

friendly_unicode() ->
    ?LET(X, ?SUCHTHAT(S, string(),
                      not lists:member(0, S) andalso
                      nomatch =:= string:find(S, "\\") andalso
                      nomatch =:= string:find(S, "_") andalso
                      nomatch =:= string:find(S, "%") andalso
                      string:length(S) < 256),
         elements([X, unicode:characters_to_binary(X)])).
```
> **Note:** Replace the `title` and `author` generators in the generators section of the `test/prop_bookstore.erl` file with the above.



This makes sure that character `0` never makes it through the generator, but also blocks backslashes and other troublesome characters. This will let us test our transitions without worrying about edge cases that we decided should belong in another property. Let’s run the test again and see what we get.

We fell through a case where `return_copy_unknown` returns an error for zero updated rows. Our postcondition for this specific deterministic function expects this: 
```erlang
postcondition(_, {_, _, return_copy_unknown, _}, {error, not_found}) -> 
    true;
```


The original code for this is in `src/bookstore_db.erl` and can be fixed by adding one clause to the `handle_single_update/1` returning `{error, not_found}` as expected, and fixing all the functions using it.

```erlang
handle_single_update({{update,1}, _}) -> ok; 
handle_single_update({{update,0}, _}) -> {error, not_found}; 
handle_single_update({error, Reason}) -> {error, Reason}; 
handle_single_update(Other) -> {error, Other}.
```
> **Note:** Replace the set of `handle_single_update` functions in the `src/bookstore_db.erl` file with the above.

This is all that we need for our application to work at a basic level and more than enough for us to test. Let's run the tests again.

This failure is interesting. If we do the check by hand, things seem to be reasonable. We’re looking for a book written by an author whose name contains the character `13` in it. The database returns one book with that value, written by `<<2,13>>`, whereas our state contains the book author `[2,13]`. Those are the same author! We’ve just been hit by a bug in our test caused by our desire to support both UTF-8 binaries and character lists in the same interface.

The fix this problem , we should revisit our postconditions. We have to make sure that the string comparisons they contain won’t care whether the string is a binary or a list of characters. We can devise some custom helper functions that will do that for us. 

```elixir
books_equal([], []) -> 
    true;
books_equal([A|As], [B|Bs]) ->
    book_equal(A, B) andalso books_equal(As, Bs);
books_equal(_, _) -> 
    false.

book_equal({ISBNA, TitleA, AuthorA, OwnedA, AvailA}, 
           {ISBNB, TitleB, AuthorB, OwnedB, AvailB}) -> 
    {ISBNA, OwnedA, AvailA} =:= {ISBNB, OwnedB, AvailB}
    andalso
    string:equal(TitleA, TitleB) andalso string:equal(AuthorA, AuthorB).
```
> **Note:** Add these into the helper's sections at the end of the `test/prop_bookstore.erl` file.

The first function takes two lists of books and checks that all of their elements are the same, with the help of the second function. The latter one tests the equality of every field with a format known to be stable, and then uses `string:equal/2` on the fields with varying Unicode encodings to safely compare them for equality. The functions can be edited in our existing postconditions. We'll have to update the lookup postconditions that we defined in the previous lesson. They should now look like this: 

```erlang
postcondition(S, {_, _, find_book_by_isbn_exists, [ISBN]}, {ok, [Res]}) ->
    book_equal(Res, maps:get(ISBN, S, undefined));
postcondition(_, {_, _, find_book_by_isbn_unknown, _}, {ok, []}) ->
    true;
postcondition(S, {_, _, find_book_by_author_matching, [Auth]}, {ok,Res}) ->
    Map = maps:filter(fun(_, {_,_,A,_,_}) ->
                              nomatch =/= string:find(A, Auth)
                      end, S),
    books_equal(lists:sort(Res), lists:sort(maps:values(Map)));
postcondition(_, {_, _, find_book_by_author_unknown, _}, {ok, []}) ->
    true;
postcondition(S, {_, _, find_book_by_title_matching, [Title]}, {ok,Res}) ->
    Map = maps:filter(fun(_, {_,T,_,_,_}) ->
                             nomatch =/= string:find(T, Title)
                      end, S),
    books_equal(lists:sort(Res), lists:sort(maps:values(Map)));
postcondition(_, {_, _, find_book_by_title_unknown, _}, {ok, []}) ->
    true;
```
> **Note:** These are the 6 cases before the final case between lines 116 and 133 in the original form in the `test/bookstore_test.exs` file. 


The first edit replaces one of our existing clauses and makes it use the new comparison function. The last two clauses work on lists and use `books_equal/2` instead of `book_equal/2`. Let's run the tests again. 

The good news is that we have a failure. Take a look at the shrunken counterexample: 

1. `add_book_new(ISBNA, Title, Author, 1, 1) -> ok`
2. `borrow_copy_unavail(ISBNA) -> ok`

It appears our problem is that our model allows calling `book_shim:borrow_copy_unavail/1` when the copy is in fact available. Currently, it is validated with the last precondition that we have. But it visibly doesn’t check for book availability. Let’s add clauses for this call specifically, and the `bookstore_db:return_copy` calls too, since they have the same problem. Let's add the following preconditions to the `test/bookstore_test.exs` file:

```erlang
precondition(S, {call, _, borrow_copy_avail, [ISBN]}) -> 
    0 < element(5, maps:get(ISBN, S));
precondition(S, {call, _, borrow_copy_unavail, [ISBN]}) -> 
    0 =:= element(5, maps:get(ISBN, S));
precondition(S, {call, _, return_copy_full, [ISBN]}) -> 
    {_, _, _, Owned, Avail} = maps:get(ISBN, S),
    Owned =:= Avail;
precondition(S, {call, _, return_copy_existing, [ISBN]}) -> 
    {_, _, _, Owned, Avail} = maps:get(ISBN, S),
    Owned =/= Avail;
```
> **Note:** We won’t add these at the end of the preconditions because they must be added before the final precondition.

Let's try to run the code one more time.


Understand how refining the tests we have written for the application will yield better results.

Refining the Tests 

Foundations of Property-Based Testing

Writing Properties

Thinking in Properties

Custom Generators

Responsible Testing

Properties-Driven Development

Shrinking

Targeted Properties

Stateful Properties

Case Study: Bookstore

State Machine Properties

Conclusion

Refining the Tests

Code