A new "pio tdd" workflow?

[jcw]

I’ve been experimenting a bit with a new development workflow. I’d like to describe some aspets of this, because there might be something of use for PIO. Deep breath. Where to start …

For me personally, fast edit/run/test dev cycles are the secret sauce which leads to more productivity. I don’t mean seconds, but sub-second cycles, because then you can just edit and tweak as you go, to quickly zoom in on various problems. A fast laptop and a fast upload (e.g. ST-Link is faster than DFU) goes a long way. With some tweaking, I can easily get to a 3…5 second cycle: making changes, compile & upload, reset, see the result. It helps of course that much of what I’m working on is low-level and uses barely any external libraries.

But since a few days now, I’ve been exploring a completely different dev mode: TDD with native builds, using make. No embedded µC in sight. I’m using DocTest, which is amazing (more on that later). With a little tdd.py script I hacked together, the result is that any source file save triggers a rebuild and shows the test results - within a fraction of a second. Combined with the editor saving on focus-change (I use vim on Mac), that’s not just “neat”, it’s … a game changer, IMNSHO.

Here’s the workflow: you edit a few lines, you save, you glance briefly at the partially visible shell window, see that all is green, and move on. Or if it’s a message in red: a quick look at the code under the cursor is often enough to see the problem and fix it, no need to even interpret the error message in many cases. It’s magical, and of course part of it is simply due to the TDD methodology.

How is this relevant for embedded development?

Well, there’s no doubt about it: native development will always be a lot faster than embedded, even with the fastest upload and reset cycles. I for one, will try even more than before to segment my code into pieces which can be developed natively and the rest. Some hardware can be simulated, and there’s TDD’s whole arsenal of tricks: mocks, spies, doubles, etc - but frankly, I don’t think that’ll be terribly practical for hobby-level or other open-source embedded development.

But perhaps there are some parts which might be useful to carry over into PlatformIO:

• currently, PIO is a one-shot command: you run it (or the IDE does for you), it does its thing, and then it exits - rinse and repeat
• part of the overhead comes from startup, i.e. Python finding and loading all the modules it needs, and then scanning source code, library dependencies, etc
• what it PlatformIO had a mode (pio tdd ...?) which keeps running in the background, watching for file changes, and performing tasks as soon as a trigger condition of some sort has been detected?
• if based on a tool such as fswatch (there are many alternatives), then watching for changes on the disk can easily be implemented, and with the right design this could stay quick even for larger projects
• PIO currently has Unity as test framework - while Unity works fine, it’s not really taking advantage of what’s possible today, I think
• would it make sense for PIO to generate a Makefile for the specific task of fast-turnaround TDD development? maybe not, maybe that’s considered heresy - the fact remains that I was blown away when I noticed the performance difference (on native builds of a small project)

The tdd.py script mentioned above is still very rough. It does handle transitive header dependencies, but it doesn’t know anything about files until they change, so on startup there’s no dependency info to work from. This clearly is not good enough - for this workflow to be truly effective, it needs accurate dependency management (just #include, no conditionals). Maybe there’s a way to query PIO and obtain these dependencies, and then run tdd.py while using this approach? I’ll probably explore a few avenues in the near future.

The main point here is that the proof-of-concept tdd.py script runs continuously in the background, which explains why it’s more responsive than pio / scons (it’s also trivially small, so this is not really a meaningful comparison).

DocTest

As mentioned before, I started using DocTest as unit testing framework. It’s based on ideas from Catch2 - but it’s considerably faster. It’s also very easy to use - being a single header file, there is no installation or setup to speak of (note that you’ll only see the real speed boost when keeping the main() it generates in a separate file, as that build takes a bit longer).

I’m not going to describe DocTest in detail - it’s not meant for embedded use (AFAIK), so it may not even be that useful for PIO, other than for native builds. But here is an example of a test I just wrote:

SUBCASE("allocate all") {
    int i;
    for (i = 0; pool.hasFree(); ++i)
        CHECK(pool.allocate() != nullptr);
    CAPTURE(i);
    CAPTURE(i+1);
    CHECK(i > 20);
}

And here’s the output of that (slightly contrived) failing test:

hallTest.cpp:37:
TEST CASE:  pool
  allocate all

hallTest.cpp:64: ERROR: CHECK( i > 20 ) is NOT correct!
  values: CHECK( 11 >  20 )
  logged: i := 11
          i+1 := 12

Three notes:

1. the report includes the expression as source code as well as both values of the comparison expression - there is some deep C++ template and operator overloading magic going on to make this possible (I think …) - this is astonishingly useful
2. you can capture any other info you want, which will only be evaluated and shown when a failure is actually being reported - it does not slow down the tests (in the above example, I could have captured all the pointers returned from allocate(), for example)
3. these tests run really really fast, thousands of assertions are no problem at all (on native)

In comparison to Unity, what also stands out is the much simpler setup / teardown logic, the fact that you can nest test cases as needed, and the fact that all test cases self-register, i.e. there’s no risk of forgetting to call a test and concluding that “it works” when in fact the test never ran.

As I said, DocTest is not for embedded use (its stringified overhead and heavy reliance on the C++ Standard Template Library might quickly exceed what fits in an embedded target).

But at the same time, these development mode differences are so substantial, that IMO it’s worth trying as much as possible to bring the best of native to embedded dev, wherever possible.

I’m going to spend a lot more time in this “native + make” mode, while improving the tdd.py script further. Trying to move even more of my development over to native mode, because it’s so darn effective. And fun!

-jcw

PS. The book “Test Driven Development for Embedded C” by James Grenning is a classic. Lots of ideas how you can develop software for embedded µCs without running on embedded µCs (up to a point, evidently). Maybe there’s no point in the context of hobby-projects … or maybe there’s an opportunity for PIO to help bring this approach more into the mainstream?

[jcw]

First off, a big hurray for Ivan and his team: doctest has been added to PIO 6.0.2 !

I’ve been getting good mileage out of an automated CLI-based “continuous” test approach, especially on native tests. The basic idea is to watch for file changes in the source code, and automatically re-run pio test when that happens. The following ctdd.sh shell script is an example (for MacOS in this case), it relies on a tool called fswatch:

DIRS="lib test"

echo Watching: $DIRS
fswatch -l 0.1 -r $DIRS |
    while read; do
        clear
        pio test "$@"
        read -t 1
    done

With my editor set to auto-save on focus-change, this re-runs the tests and makes TDD a breeze. Perhaps something like pio watch ... or pio tdd ... could be made to incorporate this approach? It would remove the need for a separate script, and could in fact be smarter about which directories to watch.

[maxgerhardt]

That’s a really neat idea and way of developing.

[jcw]

Yup, works well. Here’s a more elaborate version which also works on Linux:

#!/bin/sh

# ctdd.sh - Continuous TDD: runs "pio test" on each file change
#
# Usage: ./ctdd.sh ?...?
#
#   (all cmdline args are passed on to "pio test")

DIRS="lib test"

case `uname` in
    Darwin) FLAGS='' ;;
    Linux) FLAGS='--event Created --event Removed --event Updated' ;;
esac

echo Watching: $DIRS
while fswatch -l 0.1 -1 -r $FLAGS $DIRS; do
    clear
    pio test "$@"
done

[dvdnwk]

Awesome idea!

Here is my implementation with filtering to specified test(s), detecting env, output coloring:

#!/bin/bash
#
# Runs test(s) on file changes.
#
# Requires `fswatch`:
# https://github.com/emcrisostomo/fswatch
#
# Pass path to test dir as the first argument.
# For CLion's External Tools you can use `$FileRelativeDir$` macro var.
#
# Env is decided automatically from first dir after test root dir.
#
# Example usage:
#   * `./tdd.sh "test/native/*"`
#   * `./tdd.sh "test/native/Foo/test_foo/"`
#
# Thanks to `jcw` for inspiration:
# https://community.platformio.org/t/a-new-pio-tdd-workflow/21431

# without it colors don't work in CLion's output window
export PLATFORMIO_FORCE_ANSI=true

# first argument without test dir name
test_filter=${1#"test/"}
test_filter=${test_filter#"/test/"}

if [ -n "${test_filter}" ]
then
  test_filter_arg=" --filter=${test_filter}"
else
  test_filter_arg=""
fi

# extract the env from the first dir
test_env=$(echo "${test_filter}" | cut -d "/" -f 1)

if [ -n "${test_env}" ]
then
  test_env_arg=" -e ${test_env}"
else
  test_env_arg=""
fi

test_command="pio test${test_env_arg}${test_filter_arg}"

watch_dirs="config lib src test"

font_escape="\033["
font_reset="${font_escape}0m"

font_regular_blue="${font_escape}0;34m"
font_bold_blue="${font_escape}1;34m"
font_light_gray="${font_escape}0;37m"
font_italic="${font_escape}3m"

script_name=$(basename "${0}")

waiting_message="
${font_bold_blue}\
---
${script_name}: \
waiting for changes to run...
  \`\
${font_regular_blue}\
${font_italic}\
${test_command}\
${font_bold_blue}\`
---\
${font_reset}\
"

running_message="
${font_bold_blue}\
---
${script_name}: \
running...
  \`\
${font_regular_blue}\
${font_italic}\
${test_command}\
${font_reset}\
${font_bold_blue}\
\`
---\
${font_reset}
"

echo -e "${running_message}"
$test_command
echo -e "${waiting_message}"

is_not_temporary_file () {
  # checking the last char of the first argument
  [[ "${1: -1}" != "~" ]]
}

fswatch --latency 0.1 --recursive ${watch_dirs} |
  while read -r changed_file; do
    if is_not_temporary_file "${changed_file}"; then
      echo -e "${font_light_gray}File changed: \`${changed_file}\`${font_reset}"
      echo -e "${running_message}"
      $test_command
      echo -e "${waiting_message}"
    fi
  done

As a bonus, it can be added to External Tools in CLion:

That way we can easily activate it using a shortcut, while editing the test file:

[strooom]

one of the advantages of Unity is that it can run both native/desktop AND on the target.
Doing unitTests native yields indeed a faster test-cycle, but you should also test those hardware independent libraries on the target : it’s a different environment and so your code could behave different. (data-types may have different width, stack-size is different, etc…)

So I like that I can write a single test, and run it on all targets.

Also for simple things like syntax errors etc, IntelliSense is kicking in (at least on VSCode) and this is basically doing what you are describing : compiling in the background while you are writing the code. Even on larger projects this works amazingly well for me.

That being said, I like the idea of more support for TDD in PIO, and this support could make it stand out from other development setups.

[dvdnwk]

@jcw

pio tdd

I could try to implement pio tdd or pio test --tdd or something like that and make a PR, if @ivankravets approved the idea.

The questions would be:

1. The API? I think best would be pio test --tdd. It would just do the the same as pio test, with --filter, -e flags, etc.; but instead of finishing work, await for the changes to refresh the test(s) status.
2. How to watch for changes? fswatch is working great for me, but I don’t know if this can/should be included with the pio installation. Maybe as an optional requirement for --tdd to work?