By Eric Torreborre
We saw in a previous blog post that the Assembly platform provides a very expressive and type-safe smart contracts language, SymPL, to power the enterprise applications deployed on the platform.
SymPL is being developed with Haskell in order to provide full control over the execution model and ensure both determinism and correctness, though the language itself is actually Pythonic in its syntax and semantics Haskell, is an excellent tool for implementing side-effect-free, mission critical software, while its programming paradigm is less familiar to many developers.
At Symbiont, we also use Haskell for the implementation of another crucial component of the platform: the distributed state machine. This state machine is the "orchestrator" guaranteeing that each node of the network computes distributed transactions in a consistent way, in accordance with our privacy model. We chose Haskell for its support for functional programming and its robust type system.
However a type system is not everything and we still need to write tests to check for correctness! Fortunately Haskell comes with great testing libraries supporting a variety of approaches. In this post, we focus on one testing library, called `tasty,` and how to make it a little bit more flexible.
A taste of `tasty`
tasty is a Haskell testing library which is quite interesting. Contrary to other testing libraries it does not really care how you write your tests. You can use many different "styles":
1. You can use classical test unit assertions:
2. You can use a more behavior-driven development (BDD) way of describing your tests:
3. You can use property-based testing with either QuickCheck or Hedgehog
So what does `tasty` do exactly?
`tasty` provides a way to run any test which conforms to a given "interface":
It must have a `TestName`
It must conform to the IsTest typeclass, which basically means that you can run it and get a fail/pass result back
`tasty` then offers the possibility to group tests in TestTrees:
Selecting tests
Grouping tests in "trees" often feels like a natural thing to do. With tests grouped in named trees you can select on the command line which test group you want to run:
The `tasty` option `--pattern` actually provides a powerful "language" for selecting tests based on group names. Let's take the example from the documentation, and create 2 nested test groups:
When evaluating if the test case "Three" should be executed you can specify that the variable `$0`, which is the full test name `One.Two.Three`, needs to contain `Two` for example:
Or you can expect this test to be executed because its top-level group, `$1`, does not contains the word "SKIP":
This is all nice but there are situations where we want to be able to select test cases at different levels of the test hierarchy based on various criteria:
Does the test use a database?
Is the test fast?
Does the test require a migration?
Does the test require tracing?
...
Fortunately `tasty` has a solution for us in the form of custom options.
Tagging options
Custom options
You can indeed create any "option" with tasty and use that option to drive the behaviour of your tests. For example the tasty-hedgehog library defines a `HedgehogTestLimit` option. Every `testProperty` is equipped with that option and when a value is entered on the command line, the property will use the passed value to control the number of times a property must succeed before the test case can be considered "PASSED":
A Database option
Let's use this to select tests which require a database. First we need to declare our new `Database` option type:
Then we need a way to express that tests can use this option:
This says:
Ask for the `Database` option value
If it is set, return the current test tree
Otherwise return an empty test group
Now any test adorned with `withDatabase` can be controlled from the command line:
This test will only be executed if we pass the `database` option on the command line:
Combining options
This is working quite well but the reality can be more complicated! Sometimes the tests we want to run depend on several factors:
Run all the tests which require a database
Run all the tests which are slow
Run all the tests which require database and are slow
Run all the tests which require a database or are slow
Run all the tests which are slow and don't require database
Run all the tests which are fast integration tests but don't need tracing
...
In that case specifying several distinct options will not work because the selection of tests depend on the possible *combinations* of those options.
A "mega-option"
One solution to this problem is to aggregate all the options into one large option! The option below is created so that:
It collects all of our custom option values
It adds a `--any` flag to decide if we can select a test based on all the flags (the default), or just one of them
This `CustomOptions` data type is integrated in `tasty` with:
But this only declares that we can use this option with `tasty`, we still need to:
Populate the value of `CustomOptions` with the individual tags set on each test for `Database`, `Tracing` etc...
Collect the individual tags set on the command line
Implement the logic for deciding if a test must be executed or not
Collect test "tags"
For 1. we can redefine `withDatabase` as:
Now tagging a test with `withDatabase` will adorn it with a `CustomOptions` value where `Database` is set to `True`. This is what `localOption` does. It says: "now on the test group `t`, the custom options have a specific value for `Database`".
Collect command-line values
For point 2. we can collect all the values set on the command line with some nested `askOption`:
Adorning a test with `withCustomOptions` allows us to:
Get the value of the `Database` option as set on the command line
Get the value of the `Tracing` option as set on the command line
Get the value of the `Slow` option as set on the command line
Get the value of the `CustomOptions` as set on the test tree using `withDatabase`, `withTracing`, ...
Select tests using some logic
We are now in a position to decide if the current test (actually `TestTree`) can be executed or not by comparing the values passed on the command line to the options set on that particular test (this is point 3.).
We want to give the following meaning to command-line values:
If the values are `--a --b --c` we want to select any test which has all those tags
If `--any` is added to the list: `--any --a --b --c` we want to select the tests having at least one of those 3 tags (possibly more/others)
We can encode this logic with the following:
A default configuration
We also need to specify what happens when nothing is passed on the command line. In that case all the options have their default values, `False`, and we want to select tests which have no tags:
Conclusion
`tasty` is a library which can take some time to dig into but which is quite open for customizations. In this blog post we have shown how to:
Create options for "tagging" tests
Aggregate their values both on the command line and on tests
Select tests based on some arbitrary logic for combining the option values
Happy testing in Haskell!