Please help me understand how I can do TDD in PlatformIO

Advanced Solutions Unit Testing
1

Edit: Wow, wouldn’t you know I begin understanding this after I gave in and wrote a post asking for help. See my reply(or replies) for what I’ve found out on my own. I decided rather than removing my questions I will keep them for posterity.

Hello everyone. I feel some context is valuable here so I’ll share some of that before I get into my questions.

Context
I am a test architect born out of a software engineer in my day job. I’ve always enjoyed building DIY electronics kits, so I figured it was high time I check out the microcontroller and embedded device scene. Somewhat unusually, I have no interest in using this to learn programming as seems to be the primary use-case for it. Instead, I have been trying to find the best way to do serious microcontroller development work - just to quench my own curiosity - which is how I found PlatformIO.

IMO, the Arduino IDE isn’t suited for serious development work for many reasons, but my primary gripes are lack of test tooling and poor intellisense/multi-file editing. Upon digging deeper, I found Atmel Studio 7. I decided against using it because it’s Windows only and seems to be an offender of platform lock-in behavior. PlatformIO seems to be a nice project that mixes the two well. It has good Unit Test support, ability to run in a CICD pipeline, and is of course open source and trying to break free from platform lock-in.

If some of the things I ask for don’t exist or haven’t been considered I certainly don’t mind contributing to the PlatformIO project, but I get the feeling my questions come from me not understanding the domain well enough yet. I will admit that C/C++ is not my area of expertise and when I’ve looked into testing capabilities in the past I have always seemed to find they aren’t great so maybe some of my issues are related to that cultural difference (Python is my preferred language actually, but I also do lots of Node and Java).

Questions

Note that all of my questions are intentionally worded generically. If you must know, I’m currently on a Windows PC using an ATTiny85 & Tiny AVR programmer. I also have an Arduino Uno & Arduino Mega 2560. I also have Linux and Mac dev environments.

  1. Do you think PlatformIO is what I’m looking for? Perhaps it’s really not meant for “serious” work as I described it?
  2. Is there a guide/docs of any kind that may be helpful to get one started using TDD (Test Driven Development) on microcontrollers/does the TDD concept even exist in the microcontroller space yet?
  3. Is there a way to Unit Test locally without the need of a microcontroller programmer being plugged in? Ideally what I’m hoping for is to use mocking/stubbing and running locally as x86/64 (then having the ability to also run them on the real hardware ideally).
  4. If that’s not possible, perhaps I could use a simulator of some kind? (I’ve found the docs on simavr and an even more obscure WizIO but couldn’t work out how to use it without still involving a physical microcontroller)
  5. If running tests requires uploading to a microcontroller, how does the CICD pipeline integration work? Are Unit Tests ran as part of the pipeline?
  6. Are Unit Tests included directly within the binary sent to the device or are they built locally then ran over a debugging protocol on the device?
  7. If Unit Tests are included directly within the binary, it seems like this could have not inconsequential impacts to processing speed and available memory, invalidating test results? Is there a way to build the project without tests/run the tests externally?
2

Seems I have figured out some of this on my own…

  1. Seems to be the answer is yes - by specifying native as the platform.
  2. Short answer seems to be no (and somewhat invalidates question 7). See here. After I continue learning for myself I will update with better instructions
3

I read through your whole post really excited to see what kind of response you were going to get because I’ve had many of the same questions without finding good answers. Then I got to the bottom and there was nothing. It saddens me that the Arduino/Embedded community doesn’t have the same great toolchain options that higher level programming does. They make development so much easier!

4

PlatformIO has put out many blog posts about Unit testing with PlatformIO, as well as having a vast documentation on the topic. Please refer to

With this information one can definitely use PlatformIO in test-driven-development style way.

5

Yes, PlatformIO can do TDD.

TDD is not restricted to desktop applications. See the PlatformIO-provided reasources I’ve linked to above.

Yes. You can always use the native platform to compile for your local (e.g., x64) machine and test hardware–independent logic that way. The calculator example I’ve linked to above is one such example. Also, emulation is possible, depending on the microcontroller. You’ve already talked about simavr as a viable AVR emulator, but PlatformIO also supports “Renode”, a much more powerful emulator for all sorts of boards and architectures (ARM, RISC-V, some FPGAs,…). See supported Renode boards and PlatformIO’s Renode documentation.

You choose what happens in the CI/CD pipeline. If you setup CI/CD to run unit tests on the microcontroller environments (and not the native environment which it can execute natively), and no emulator is used, you need real hardware to perform the unit tests on. PlatformIO offers something called PlatformIO Remote, and with it the pio remote test command. Basically you can setup a PlatformIO account (free), then connect a computer of your choice to the PlatformIO cloud (pio remote agent start). Other computers can then log into the PlatformIO account (pio account login) and see all connected remote agent for that account ( pio remote agent list) and with it, the microcontroller boards connected to that computer. You can literally take a Raspberry Pi, connect your microcontroller board to that you want to run unit tests on, set it up with your PlatformIO account, start the remote agent listener, and then on the CI/CD side, log into that same PlatformIO account and execute remote unit tests. The documentation I’ve linked to talks about this in more detail.

With the standard PlatformIO way of doing unit tests, the Unity test framework is built into the firmware, and with it all the unit tests. Upon executing the firmware, the firmware autonomously executed all the unit tests and sends back the results via serial (or a custom-defined transport protocol). This is apparent when looking at the unit testing examples I’ve linked to.

One can of course engineer much more custom complicated stuff, but that’s the standard of how it works.

I don’t see how that adds so much memory and processing overhead. You have the logic you want to test, you compile that in, plus some test functions that call that logic and test its output for correctness. If you don’t test your input functions with gigabytes of data that is to be contained in the firmware, it’ll be fine. External fuzzing is a different topic.

6

Wow thank you for the quick and thorough response!

I’ll have to look into “Renode” because emulation would be pretty valuable for my use case.

Regarding unit testing taking up memory, isn’t it true that the unit test firmware is actually a different binary than the main firmware? I got the feeling that when you click the “Test” task, PIO creates a test harness that has its own set of serial commands that run tests and print to the PIO terminal. Then when you do a normal build/upload, none of that would get carried along.

7

Correct, building the test firmware takes a different path than building the regular firmware.

When running “PIO Test”, it will include the Unity code (<home path>/.platformio/pacakges/tool-unity) along with an auto-generated cpp file which Unity needs to able to ouput anything. E.g., with

in conjunction with this template, the default implementation when the project has framework = arduino is that output_char() maps to Serial.print(), etc.

Next to Unity, all the selected unit tests files (see test_filter) are compiled in, plus any of the libraries (in the lib/ folder of the project or referenced in lib_deps) that the unit test functions have dependencies to. The src/ folder of the firmware is by default not compiled, since the recommended way is to have the logic that you want to unit-test separated as free-standing components in lib/. The option test_build_project_src can however override this behavior.

But thinking back about sizes again: Given the standard way of doing things, where you have a main firmware that uses some components, the regular firmware will be much bigger than the firmware that just uses the components and tests them, but doesn’t have the entire firmware logic in it. Of course when the amount of “firmware logic in src/” and “tested components in lib/ and size of unit test functions in test/” shifts, so do these firmware sizes shift as well.