PlatformIO Community

A new "pio tdd" workflow?

I’ve been experimenting a bit with a new development workflow. I’d like to describe some aspets of this, because there might be something of use for PIO. Deep breath. Where to start …

For me personally, fast edit/run/test dev cycles are the secret sauce which leads to more productivity. I don’t mean seconds, but sub-second cycles, because then you can just edit and tweak as you go, to quickly zoom in on various problems. A fast laptop and a fast upload (e.g. ST-Link is faster than DFU) goes a long way. With some tweaking, I can easily get to a 3…5 second cycle: making changes, compile & upload, reset, see the result. It helps of course that much of what I’m working on is low-level and uses barely any external libraries.

But since a few days now, I’ve been exploring a completely different dev mode: TDD with native builds, using make. No embedded µC in sight. I’m using DocTest, which is amazing (more on that later). With a little tdd.py script I hacked together, the result is that any source file save triggers a rebuild and shows the test results - within a fraction of a second. Combined with the editor saving on focus-change (I use vim on Mac), that’s not just “neat”, it’s … a game changer, IMNSHO.

Here’s the workflow: you edit a few lines, you save, you glance briefly at the partially visible shell window, see that all is green, and move on. Or if it’s a message in red: a quick look at the code under the cursor is often enough to see the problem and fix it, no need to even interpret the error message in many cases. It’s magical, and of course part of it is simply due to the TDD methodology.

How is this relevant for embedded development?

Well, there’s no doubt about it: native development will always be a lot faster than embedded, even with the fastest upload and reset cycles. I for one, will try even more than before to segment my code into pieces which can be developed natively and the rest. Some hardware can be simulated, and there’s TDD’s whole arsenal of tricks: mocks, spies, doubles, etc - but frankly, I don’t think that’ll be terribly practical for hobby-level or other open-source embedded development.

But perhaps there are some parts which might be useful to carry over into PlatformIO:

  • currently, PIO is a one-shot command: you run it (or the IDE does for you), it does its thing, and then it exits - rinse and repeat
  • part of the overhead comes from startup, i.e. Python finding and loading all the modules it needs, and then scanning source code, library dependencies, etc
  • what it PlatformIO had a mode (pio tdd ...?) which keeps running in the background, watching for file changes, and performing tasks as soon as a trigger condition of some sort has been detected?
  • if based on a tool such as fswatch (there are many alternatives), then watching for changes on the disk can easily be implemented, and with the right design this could stay quick even for larger projects
  • PIO currently has Unity as test framework - while Unity works fine, it’s not really taking advantage of what’s possible today, I think
  • would it make sense for PIO to generate a Makefile for the specific task of fast-turnaround TDD development? maybe not, maybe that’s considered heresy - the fact remains that I was blown away when I noticed the performance difference (on native builds of a small project)

The tdd.py script mentioned above is still very rough. It does handle transitive header dependencies, but it doesn’t know anything about files until they change, so on startup there’s no dependency info to work from. This clearly is not good enough - for this workflow to be truly effective, it needs accurate dependency management (just #include, no conditionals). Maybe there’s a way to query PIO and obtain these dependencies, and then run tdd.py while using this approach? I’ll probably explore a few avenues in the near future.

The main point here is that the proof-of-concept tdd.py script runs continuously in the background, which explains why it’s more responsive than pio / scons (it’s also trivially small, so this is not really a meaningful comparison).

DocTest

As mentioned before, I started using DocTest as unit testing framework. It’s based on ideas from Catch2 - but it’s considerably faster. It’s also very easy to use - being a single header file, there is no installation or setup to speak of (note that you’ll only see the real speed boost when keeping the main() it generates in a separate file, as that build takes a bit longer).

I’m not going to describe DocTest in detail - it’s not meant for embedded use (AFAIK), so it may not even be that useful for PIO, other than for native builds. But here is an example of a test I just wrote:

SUBCASE("allocate all") {
    int i;
    for (i = 0; pool.hasFree(); ++i)
        CHECK(pool.allocate() != nullptr);
    CAPTURE(i);
    CAPTURE(i+1);
    CHECK(i > 20);
}

And here’s the output of that (slightly contrived) failing test:

hallTest.cpp:37:
TEST CASE:  pool
  allocate all

hallTest.cpp:64: ERROR: CHECK( i > 20 ) is NOT correct!
  values: CHECK( 11 >  20 )
  logged: i := 11
          i+1 := 12

Three notes:

  1. the report includes the expression as source code as well as both values of the comparison expression - there is some deep C++ template and operator overloading magic going on to make this possible (I think …) - this is astonishingly useful
  2. you can capture any other info you want, which will only be evaluated and shown when a failure is actually being reported - it does not slow down the tests (in the above example, I could have captured all the pointers returned from allocate(), for example)
  3. these tests run really really fast, thousands of assertions are no problem at all (on native)

In comparison to Unity, what also stands out is the much simpler setup / teardown logic, the fact that you can nest test cases as needed, and the fact that all test cases self-register, i.e. there’s no risk of forgetting to call a test and concluding that “it works” when in fact the test never ran.

As I said, DocTest is not for embedded use (its stringified overhead and heavy reliance on the C++ Standard Template Library might quickly exceed what fits in an embedded target).

But at the same time, these development mode differences are so substantial, that IMO it’s worth trying as much as possible to bring the best of native to embedded dev, wherever possible.

I’m going to spend a lot more time in this “native + make” mode, while improving the tdd.py script further. Trying to move even more of my development over to native mode, because it’s so darn effective. And fun!

-jcw

PS. The book “Test Driven Development for Embedded C” by James Grenning is a classic. Lots of ideas how you can develop software for embedded µCs without running on embedded µCs (up to a point, evidently). Maybe there’s no point in the context of hobby-projects … or maybe there’s an opportunity for PIO to help bring this approach more into the mainstream?