Program testing can be used to show the presence of bugs, but never to show their absence!
– Edsger Wybe Dijkstra
DMS employs a variety of unit and integration tests. All tests and associated configuration is stored in the
test/ directory. If you want to change existing functionality or integrate a new feature into DMS, you will probably need to work with our test suite.
Can I use macOS?
We do not support running linting, tests, etc. on macOS at this time. Please use a Linux VM, Debian/Ubuntu is recommended.
We use BATS (Bash Automated Testing System) and additional support libraries. BATS is very similar to Bash, and one can easily and quickly get an understanding of how tests in a single file are run. A test file template provides a minimal working example for newcomers to look at.
test/ directory contains multiple directories. Among them is the
bats/ directory (which is the BATS git submodule) and the
helper/ directory. The latter is especially interesting because it contains common support functionality used in almost every test. Actual tests are located in
Test suite undergoing refactoring
We are currently in the process of restructuring all of our tests. Tests will be moved into
test/tests/parallel/ and new tests should be placed there as well.
There are many functions that aid in writing tests. We urge you to use them! They will not only ease writing a test but they will also do their best to ensure there are no race conditions or other unwanted side effects. To learn about the functions we provide, you can:
- look into existing tests for helper functions we already used
- look into the
test/helper/directory which contains all files that can (and will) be loaded in test files
We encourage you to try both of the approaches mentioned above. To make understanding and using the helper functions easy, every function contains detailed documentation comments. Read them carefully!
Tests are split into two categories:
test/tests/parallel/: Multiple test files are run concurrently to reduce the required time to complete the test suite. A test file will presently run it's own defined test-cases in a sequential order.
test/tests/serial/: Each test file is queued up to run sequentially. Tests that are unable to support running concurrently belong here.
Parallel tests are further partitioned into smaller sets. If your system has the resources to support running more than one of those sets at a time this is perfectly ok (our CI runs tests by distributing the sets across multiple test runners). Care must be taken not to mix running the serial tests while a parallel set is also running; this is handled for you when using
To run the test suite, you will need to:
- Install Docker
parallel(under Ubuntu, use
sudo apt-get -y install jq parallel)
git submodule update --init --recursiveif you haven't already initialized the git submodules
make to run commands.
make buildto create or update the local
mailserver-testing:ciDocker image (using the projects
- Run all tests:
make clean tests
- Run a single test:
make clean generate-accounts test/<TEST NAME WITHOUT .bats SUFFIX>
- Run multiple unrelated tests:
make clean generate-accounts test/<TEST NAME WITHOUT .bats SUFFIX>,<TEST NAME WITHOUT .bats SUFFIX>(just add a
,and then immediately write the new test name)
- Run a whole set or all serial tests:
make clean generate-accounts tests/parallel/setXwhere
Xis the number of the set or
make clean generate-accounts tests/serial
Setting the Degree of Parallelization for Tests
If your machine is capable, you can increase the amount of tests that are run simultaneously by prepending the
make clean all command with
BATS_PARALLEL_JOBS=X make clean all). This wil speed up the test procedure. You can also run all tests in serial by setting
BATS_PARALLEL_JOBS=1 this way.
The default value of
BATS_PARALLEL_JOBS is 2. This can be increased if your system has the resources spare to support running more jobs at once to complete the test suite sooner.
Test Output when Running in Parallel
When running tests in parallel (with
make clean generate-accounts tests/parallel/setX), BATS will delay outputting the results until completing all test cases within a file.
This likewise delays the reporting of test-case failures. When troubleshooting parallel set tests, you may prefer to run specific tests you're working on serially (as demonstrated in the example below).
When writing tests, ensure that parallel set tests still pass when run in parallel. You need to account for other tests running in parallel that may interfere with your own tests logic.
In this example, you've made a change to the Rspamd feature support (or adjusted it's tests). First verify no regressions have been introduced by running it's specific test file:
$ make clean generate-accounts test/rspamd rspamd.bats ✓ [Rspamd] Postfix's main.cf was adjusted  ✓ [Rspamd] normal mail passes fine  ✓ [Rspamd] detects and rejects spam  ✓ [Rspamd] detects and rejects virus 
As your feature work progresses your change for Rspamd also affects ClamAV. As your change now spans more than just the Rspamd test file, you could run multiple test files serially:
$ make clean generate-accounts test/rspamd,clamav rspamd.bats ✓ [Rspamd] Postfix's main.cf was adjusted  ✓ [Rspamd] normal mail passes fine  ✓ [Rspamd] detects and rejects spam  ✓ [Rspamd] detects and rejects virus  clamav.bats ✓ [ClamAV] log files exist at /var/log/mail directory  ✓ [ClamAV] should be identified by Amavis  ✓ [ClamAV] freshclam cron is enabled  ✓ [ClamAV] env CLAMAV_MESSAGE_SIZE_LIMIT is set correctly  ✓ [ClamAV] rejects virus 
You're almost finished with your change before submitting it as a PR. It's a good idea to run the full parallel set those individual tests belong to (especially if you've modified any tests):
$ make clean generate-accounts tests/parallel/set1 default_relay_host.bats ✓ [Relay] (ENV) 'DEFAULT_RELAY_HOST' should configure 'main.cf:relayhost'  spam_virus/amavis.bats ✓ [Amavis] SpamAssassin integration should be active  spam_virus/clamav.bats ✓ [ClamAV] log files exist at /var/log/mail directory  ✓ [ClamAV] should be identified by Amavis  ✓ [ClamAV] freshclam cron is enabled  ...
Even better, before opening a PR run the full test suite:
$ make clean tests