Gain insights into property-based testing with PropEr in Elixir. Learn foundational principles, custom data generators, and advanced concepts for effective real-world application testing.

Property Based Testing with PropEr in Elixr_468 x 60 copy.png

propert-based-testing.tar.gz

Erlang

Elixer

Erlang-Chap2

Elixir-Chap2

Erlang-Shell

erlang-user

elixir-user

elixir-shell

elrang-eunit

erlang-chap5-employee

elixir-cover_test

erlang-cover_test

erlang-chap10-databaseinit

erlang-chap10-user

elixir-chap10-user

erlang-chap10-shell

elixir-chap10-shell

erlang-chap10-dbsetup

elixir-chap10-dbSetup

erlang-chap5-templating

erlang-chap5-last

elixir-chap5-last

erlang-chap10-default

elixir-chap10-default

elixir-chap10-default-copy

Erlang-Chap2-copy

Elixir-Chap2-copy

Erlang-Shell-copy

elixir-shell-copy

erlang-default-force

elixir-default-force

erlang-test-shell

elxir-test-shell

Property-based testing relies on specifying some property of code, unlike unit tests, which specify the expected output in response to some inputs. In this course, we'll cover the concepts we need to feel confident about using even the most advanced features of PropEr with Elixir.

We'll start with the basic and foundational principles of property-based testing, see what the framework offers us to get started, make our way through thinking in properties, write our own custom data generators, and then learn more advanced concepts. We'll also see how you can use property-based testing in a realistic project.

By the time we finish this course, we should feel comfortable testing real-world applications using properties.

Property-Based Testing with PropEr in Elixir

> **Order of failures:** \
Property tests are probabilistic. We may find different issues, or the same issues in a different order, than those in this text. This is normal, and rerunning the property a few times may yield different errors each time. \
The first error in this section can’t be reproduced easily in Elixir, where the `string()` generator is not exported by PropCheck. Instead, `utf8()` is used. However, `utf8()` more frequently returns well-formed Unicode data that doesn’t cause the failure.

# Code
Let's start off with the code and make fixes to it as we go through the lesson.

> **Note**: To run an updated version of the code, press run, and then call `mix test` in the terminal that opens. 

import React from 'react';
import { shallow, configure } from 'enzyme';
import Adapter from 'enzyme-adapter-react-16';

configure({ adapter: new Adapter() });

import HelloWorld from './app';

var TestResult = function() {
    this.succeeded = false;
    this.reason = "";
    this.input = "";
    this.expected_output = "";
    this.actual_output = "";
}

export const executeTests = function() {

  var results = [];

  result = new TestResult();
  result.input = 'HelloWorld Component';
  result.expected_output = "span containing text 'Hello World'"

  let wrapper = shallow(<HelloWorld />);

  // Call your Challenge function here.

  let type = wrapper.type();
  let testSuccessful = true;
  let failureReason;

  if (type !== 'span') {
    testSuccessful = false;
    failureReason = "You need to render exactly one span HTML element";
  } else if (wrapper.props().children != "Hello World") {
     testSuccessful = false;
     failureReason = "You have rendered wrong message in your span element";
  }

  result.actual_output = wrapper.html();

  if (testSuccessful) {
    result.succeeded = true;
    result.reason = "Succeeded"
  } else {
    result.succeeded = false;
    result.reason = failureReason;
  }

  results.push(result);

  return results;
}


React

Dart

GoJS React

Typescript React

Vue.js

# Understanding the bug

Running the tests now fails on Unicode values of `0x00`, meaning that the protocol likely uses null-terminated strings, which causes failures. Anyone using this PostgreSQL driver needs to protect themselves against such strings.

For now, we’ll assume that those invalid Unicode values should either be filtered somewhere else or that a bunch of other properties or unit tests will handle these, and we’ll instead work around it by making sure our generators don’t generate that data anymore. In fact, we’ll find a lot of other troublesome characters: `%` and `_` will influence PostgreSQL search in ways that `string:find/2` wouldn’t, and `\` will mess with escaping in SQL whereas our model won’t care.

# Refining the tests

We have to ask ourselves whether what we want to test here is the minutiae of the SQL string handling, or whether these strings are a tool we use to validate the state transitions of our stateful model. This is the point where we can decide to split our property into two distinct properties:

1. One property tests the search patterns explicitly to see if there are risky things happening with special characters.
2. One property tests the state transitions and various matches.

We’ll focus on the second one by ignoring all kinds of special characters in searches, titles, and authors.

```elixir
def title(), do: friendly_unicode()

def author(), do: friendly_unicode()

def friendly_unicode() do
  bad_chars = [<<0>>, "\\", "_", "%"]
  friendly_gen =
    such_that s <- utf8(), when: (not contains_any?(s, bad_chars)) &&
      String.length(s) < 256
  let x <- friendly_gen do
      elements([x, String.to_charlist(x)])
  end
end
```
> **Note**: Replace our `title` and `author` generators in the generators section of the `test/bookstore_test.exs` file with the above.


This makes sure that character `0` never makes it through the generator, but also blocks backslashes and other troublesome characters. This will let us test our transitions without worrying about edge cases that we decided should belong in another property. 
> Let’s run the test again and see what we get

We fell through a case where `return_copy_unknown` returns an error for zero updated rows. Our postcondition for this specific deterministic function expects this: 
```elixir
def postcondition(_, {_,_,:return_copy_unknown,_}, {:error,:not_found}) do
  true
end
```


The original code for this is in `src/bookstore_db.erl` and can be fixed by adding one clause to the `handle_single_update/1` returning `{error, not_found}` as expected, and fixing all the functions using it.

```elixir
defp handle_single_update({{:update, 1}, _}), do: :ok
defp handle_single_update({{:update, 0}, _}), do: {:error, :not_found} 
defp handle_single_update({:error, reason}), do: {:error, reason}
defp handle_single_update(other), do: {:error, other}
```

> **Note**: Replace the set of `handle_single_update` functions in the `lib/bookstore/db.ex` file with the following.

This is all that we need for our application to work at a basic level, and more than enough for us to test.

> Let's run the tests again.

This failure is interesting. If we do the check by hand, things seem to be reasonable. We’re looking for a book written by an author whose name contains the character `13` in it. The database returns one book with that value, written by `<<2,13>>`, whereas our state contains the book author `[2,13]`. Those are the same author! We’ve just been hit by a bug in our test caused by our desire to support both UTF-8 binaries and character lists in the same interface.

The fix this problem, we should revisit our postconditions. We have to make sure that the string comparisons they contain won’t care whether the string is a binary or a list of characters. We can devise some custom helper functions that will do that for us.

```elixir
defp books_equal([], []) do
  true
end
defp books_equal([a | as], [b | bs]) do
  book_equal(a, b) && books_equal(as, bs)
end
defp books_equal(_, _) do
  false
end

defp book_equal(
        {isbn_a, title_a, author_a, owned_a, avail_a},
        {isbn_b, title_b, author_b, owned_b, avail_b}
      ) do
  {isbn_a, owned_a, avail_a} == {isbn_b, owned_b, avail_b} &&
    String.equivalent?(
      IO.chardata_to_string(title_a),
      IO.chardata_to_string(title_b)
    ) &&
    String.equivalent?(
      IO.chardata_to_string(author_a),
      IO.chardata_to_string(author_b)
    )
end
```

> **Note**: Add these in the helper's sections at the end of the `test/bookstore_test.exs` file.

The first function takes two lists of books and checks that all of their elements are the same, with the help of the second function. The latter one tests the equality of every field with a format known to be stable, and then uses `string:equal/2` on the fields with varying Unicode encodings to safely compare them for equality. The functions can be edited in our existing postconditions. We'll have to update the lookup postconditions that we defined in the previous lesson. They should now look like this: 

```elixir
def postcondition(
  state,
  {_, _, :find_book_by_isbn_exists, [isbn]},
  {:ok, [res]})
do
  book_equal(res, Map.get(state, isbn, nil))
end
def postcondition(_, {_, _, :find_book_by_isbn_unknown, _}, {:ok, []}) do
  true
end
def postcondition(
  state,
  {_, _, :find_book_by_author_matching, [auth]},
  {:ok, res}
) do
  map = :maps.filter(
    fn _, {_,_,a,_,_} -> contains?(a, auth) end,
    state
  )
  books_equal(Enum.sort(res), Enum.sort(Map.values(map)))
end
def postcondition(_, {_,_,:find_book_by_author_unknown,_}, {:ok,[]}) do
  true
end
def postcondition(
  state,
  {_, _, :find_book_by_title_matching, [title]},
  {:ok, res}
) do
  map = :maps.filter(
    fn _, {_,t,_,_,_} -> contains?(t, title) end,
    state
  )
  books_equal(Enum.sort(res), Enum.sort(Map.values(map)))
end
def postcondition(_, {_,_,:find_book_by_title_unknown,_}, {:ok,[]}) do
  true
end
```
> **Note**: These are the 6 cases before the final case between lines 133 and 166 in the original form in the `test/bookstore_test.exs` file. 

The first edit replaces one of our existing clauses and makes it use the new comparison function. The last two clauses work on lists and use `books_equal/2` instead of `book_equal/2`.

> Let's run the tests again. 

The good news is that we have a failure. Take a look at the shrunken counterexample: 

1. `add_book_new(ISBNA, Title, Author, 1, 1) -> ok`
2. `borrow_copy_unavail(ISBNA) -> ok`

It appears our problem is that our model allows calling `book_shim:borrow_copy_unavail/1` when the copy is in fact available. Currently, it is validated with the last precondition that we have. But it visibly doesn’t check for book availability. Let’s add clauses for this call specifically, and the `bookstore_db:return_copy` calls too, since they have the same problem. Let's add the following preconditions to the `test/bookstore_test.exs` file:


```elixir
def precondition(s, {:call, _, :borrow_copy_avail, [isbn]}) do 
  0 < elem(Map.get(s, isbn), 4)
end
def precondition(s, {:call, _, :borrow_copy_unavail, [isbn]}) do
    0 == elem(Map.get(s, isbn), 4)
end
def precondition(s, {:call, _, :return_copy_full, [isbn]}) do
    {_, _, _, owned, avail} = Map.get(s, isbn)
    owned == avail
end
def precondition(s, {:call, _, :return_copy_existing, [isbn]}) do
    {_, _, _, owned, avail} = Map.get(s, isbn)
    owned != avail
end
```
> **Note**: We won’t add these at the end of the preconditions because they must be added before the final precondition.

Let's try the run the code one more time.


Understand how refining the tests we have written for the application will yield better results.

Refining the Tests

Foundations of Property-Based Testing

Writing Properties

Thinking in Properties

Custom Generators

Responsible Testing

Properties-Driven Development

Shrinking

Stateful Properties

Case Study: Bookstore

State Machine Properties

Conclusion

Refining the Tests