I'll start by just saying I've never introduced property tests without finding a bug in either the specification or implementation. I have repeatedly spent time figuring out where the bug was in my property testing library before realising that it was correct and the obviously correct code I was testing was actually wrong.
I often find it easier in tests to start with the default strategies, then in the test construct the world. An example would be instead of creating strategies for user accounts and monetary transfers I made list of (integer, integer, integer) and interpret that as (from_account, to_account, amount).
You could have some assertions like (I'm afraid I don't know what benefits adjustment means but I know it's just an example)
* This benefits adjustment will always be 0 or positive
* It will only ever return a value or raise one of the following explicit exceptions <- "my code doesn't crash" is a good base assertion
* Benefits adjustments over longer periods are never less (e.g. adjustment for the time range 2020-2023 >= 2020-2021)
* If I create a second organisation with employees, the result of checking the first one never changes.
Your tests don't have to check exact results, otherwise you end up rebuilding the logic you're trying to test just to work out what the result should be. You can check general properties that should remain true.
You can extend this out as well making it more stateful in a sense. I built a UI library for TVs and programs using it may add and remove elements at any time. A powerful test was
1. Given an empty UI
2. And someone makes a series of API calls where each is [add_element(...), remove_element(...), user_left(), user_right(),...]
3. One and only one element is in focus
And another
1. Given an arbitrary UI created with those calls
2. If the user moves right and the focus changes
3. Then when the user moves left they will go back to the item they were previously on
It was very hard to write the library such that it didn't do what we wanted and passed these tests. I found a case where we had an explicit unit test and part of the spec that wouldn't pass while these tests were passing, because the spec was inconsistent.
These tests I think were actually clearer to write than a series of explicit cases too.
> Will it hit the database a hundred extra times to cover combinations that don't matter when I could have just enumerated ones that do?
Yes. The benefit is in trying things you think probably work but don't actually know they do, and cases you wouldn't have thought about but are totally possible. They're not a replacement for explicit tests, they're another tool.
I have had them failing showing there was a problem in some text extraction, which boiled down to the fact that lowercasing a Turkish I in python at least results in more than one character, so positions in the text after it had been lowercased were not the same as before. I had absolutely not considered that.
A finishing thought - if you can't say a general property that should remain true given some arbitrary input, how hard is it to understand what your function will do when you see a call to it?
> I have had them failing showing there was a problem in some text extraction, which boiled down to the fact that lowercasing a Turkish I in python at least results in more than one character, so positions in the text after it had been lowercased were not the same as before. I had absolutely not considered that.
Oof, this reminds me of an obscure problem caught by a property test (ScalaTest + ScalaCheck) a few years ago: we had to parse files containing fixed-width fields, with optional parts, padded with whitespace, etc. The customer couldn't provide any documentation for the format, so we had to figure it out based on a thousand files captured off their system.
One day a colleague ran a build on their machine, and hit a test failure. It turned out that they were using a different JVM (it was written for JDK11, but they were using JDK14); we were generating strings which include/exclude whitespace based on the 'Char::isWhitespace' method; but our parser combinators were matching whitespace using regular expressions.
This discrepancy caused the test failure, since ScalaCheck had generated a string containing a "mongolian vowel separator". That character was considered as whitespace in older Unicode standards, but not newer ones. The Char::isWhitespace method seemed to take that into account, but the regexp matcher wasn't (e.g. see https://unicode-explorer.com/c/180E )
I often find it easier in tests to start with the default strategies, then in the test construct the world. An example would be instead of creating strategies for user accounts and monetary transfers I made list of (integer, integer, integer) and interpret that as (from_account, to_account, amount).
You could have some assertions like (I'm afraid I don't know what benefits adjustment means but I know it's just an example)
* This benefits adjustment will always be 0 or positive
* It will only ever return a value or raise one of the following explicit exceptions <- "my code doesn't crash" is a good base assertion
* Benefits adjustments over longer periods are never less (e.g. adjustment for the time range 2020-2023 >= 2020-2021)
* If I create a second organisation with employees, the result of checking the first one never changes.
Your tests don't have to check exact results, otherwise you end up rebuilding the logic you're trying to test just to work out what the result should be. You can check general properties that should remain true.
You can extend this out as well making it more stateful in a sense. I built a UI library for TVs and programs using it may add and remove elements at any time. A powerful test was
1. Given an empty UI
2. And someone makes a series of API calls where each is [add_element(...), remove_element(...), user_left(), user_right(),...]
3. One and only one element is in focus
And another
1. Given an arbitrary UI created with those calls
2. If the user moves right and the focus changes
3. Then when the user moves left they will go back to the item they were previously on
It was very hard to write the library such that it didn't do what we wanted and passed these tests. I found a case where we had an explicit unit test and part of the spec that wouldn't pass while these tests were passing, because the spec was inconsistent.
These tests I think were actually clearer to write than a series of explicit cases too.
> Will it hit the database a hundred extra times to cover combinations that don't matter when I could have just enumerated ones that do?
Yes. The benefit is in trying things you think probably work but don't actually know they do, and cases you wouldn't have thought about but are totally possible. They're not a replacement for explicit tests, they're another tool.
I have had them failing showing there was a problem in some text extraction, which boiled down to the fact that lowercasing a Turkish I in python at least results in more than one character, so positions in the text after it had been lowercased were not the same as before. I had absolutely not considered that.
A finishing thought - if you can't say a general property that should remain true given some arbitrary input, how hard is it to understand what your function will do when you see a call to it?