Unit testing is great way to build confidence that your code works. Usually you’d write unit tests for libraries; but if you have a particularly long or complicated shell script, it might be worth converting it to a modulino just so you can write tests for it. And if you are writing a library in shell code, you need all the help you can get anyway.
So let’s assume you have some shell code to test, now what? Well most shells have no built-in unit testing capabilities, so you need to write a script to load your code, and check it does the right thing and report the results.
This could be as simple as using C-style assert statements, but with interpreted code like shell, you don’t get the same guarantees about the correctness of the code; so if your test program only fails on asserts, it will report success even if you didn’t run all the tests you intended to. The other issue with some assert-style test code is when a test failure causes the test program to exit early, and report only one error, when it could have reported multiple test failures if only the test script had run to completion.
For those reasons I like the Test Anything Protocol. TAP-compliant test code must emit a test plan with the number of tests to run, and for every test case either “ok” for passes or “not ok” for failures. So TAP parsers can verify that the right number of tests ran, and when tests do fail, programmers can inspect the output and see the context of the failure. To that end I wrote a minimalist TAP library for POSIX-compatible shells.
Tutorial
Imagine we have written a shell library called examples/hello/hello.sh. It has one function, which by default prints “Hello, World!":
hello() {
subject="$1"
[ -z "$subject" ] && subject="World!"
echo "Hello, $subject"
}
To test it, we want to call hello and check it prints the expected output. Here’s our test-script, examples/hello/hello-test.sh:
#!/bin/sh
# import our test functions and our hello function
. "$PWD/tap.sh"
. "$PWD/examples/hello/hello.sh"
# test #1 does hello() print the expected output?
hello_out=$(hello)
if [ "$hello_out" = "Hello, World!" ];then
tap_pass "hello"
else
tap_fail "hello"
fi
# print our test plan to ensure we got here
tap_end
We can run the test script from the command line:
./examples/hello/hello-test.sh
ok 1 hello
1..1
This prints “ok” as our test passed! It also prints the number of tests run, so we can be sure all of our test script code was executed. However all this script does is emit TAP output. We can run it with a test harness which will interpret the output and tell us if the test passed or not. Perl’s prove is an easy harness to run and usually comes with Perl. You might already have it installed.
$ prove ./examples/hello/hello-test.sh
./examples/hello/hello-test.sh .. ok
All tests successful.
Files=1, Tests=1, 0 wallclock secs ( 0.01 usr + 0.00 sys = 0.01 CPU)
Result: PASS
Our hello function actually has two code paths: the default is to print “Hello, World!” but it also accepts an optional subject to greet instead. Let’s expand our test script to check that path:
#!/bin/sh
# import our test functions and our hello function
. "$PWD/tap.sh"
. "$PWD/examples/hello/hello.sh"
# test #1 does hello() print the expected output?
hello_out=$(hello)
if [ "$hello_out" = "Hello, World!" ];then
tap_pass "hello"
else
tap_fail "hello"
fi
# test #2 does hello "you" print the expected output?
hello_out=$(hello "you")
pass=0
[ "$hello_out" = "Hello, you!" ] && pass=1
tap_ok "$pass" "hello \"you\""
# print our test plan to ensure we ran 2 tests
tap_end "2"
Test #2 checks that calling hello with “you” emits “Hello, you!” instead of the default. Instead of an if/else block, it calls tap_ok with a success flag to pass or fail the test. It also calls tap_end with the number of tests to ensure we ran both tests. Running it with prove -v shows us the individual tests run, as well as a summary:
$ prove -v examples/hello/hello-test.sh
examples/hello/hello-test.sh ..
ok 1 hello
not ok 2 hello "you"
1..2
Failed 1/2 subtests
Test Summary Report
-------------------
examples/hello/hello-test.sh (Wstat: 0 Tests: 2 Failed: 1)
Failed test: 2
Files=1, Tests=2, 0 wallclock secs ( 0.01 usr + 0.00 sys = 0.01 CPU)
Result: FAIL
Uh oh, the second test case failed! It would be helpful if the test printed the mismatched variables to help us debug the issue. We can simplify our test cases to use the tap_cmp_str function to compare two strings and print them if they don’t match:
#!/bin/sh
# import our test functions and our hello function
. "$PWD/tap.sh"
. "$PWD/examples/hello/hello.sh"
# test #1 does hello print the expected output?
hello_out=$(hello)
tap_cmp_str "$hello_out" "Hello, World!" "hello"
# test #2 does hello "you" print the expected output?
hello_out=$(hello "you")
tap_cmp_str "$hello_out" "Hello, you!" "hello \"you\""
# print our test plan to ensure we ran 2 tests
tap_end "2"
Re-running the tests, now we get some actionable output:
$ prove -v examples/hello/hello-test.sh
examples/hello/hello-test.sh ..
ok 1 hello
not ok 2 hello "you" - expected 'Hello, you!' but got 'Hello, you'
1..2
Failed 1/2 subtests
Test Summary Report
-------------------
examples/hello/hello-test.sh (Wstat: 0 Tests: 2 Failed: 1)
Failed test: 2
Files=1, Tests=2, 0 wallclock secs ( 0.01 usr + 0.00 sys = 0.01 CPU)
Result: FAIL
Our hello function is appending the “!” in the wrong place; we want to greet everybody with enthusiasm, not just the “World”!. Here’s the fixed-up version, with the “!” moved to the echo argument:
hello() {
subject="$1"
[ -z "$subject" ] && subject="World"
echo "Hello, $subject!"
}
And now the tests pass:
$ prove -v examples/hello/hello-test.sh
examples/hello/hello-test.sh ..
ok 1 hello
ok 2 hello "you"
1..2
ok
All tests successful.
Files=1, Tests=2, 0 wallclock secs ( 0.01 usr + 0.00 sys = 0.01 CPU)
Result: PASS
If you download this repo you can run this code for yourself from the root project directory.
Running tests with a test harness is useful as it can run multiple test files and tell us if the test suite passed or failed overall. It can execute tests concurrently so the test suite runs faster. And by default it limits output to only the summary, so the terminal isn’t filled with noise. A few years ago I wrote an introduction to prove which describes its main features.
If prove isn’t your jam, The TAP website has a list of other TAP parsers that can be used in conjunction with a test harness. A test harness can be as simple as a one liner, here using tapview:
$ res="PASS";for t in examples/hello/*test.sh;do echo "$t"; "./$t" | tapview || res="FAIL";done;echo "$res"
examples/hello/hello-test.sh
..
2 tests, 0 failures.
PASS
Alternatives
- shunit2 is an xUnit-style unit test framework for Bourne-like shell code
- sharness is a TAP unit test shell library like
tap.shbut with more features - The TAP website has a list of shell test libraries (“producers”)