Unit-Testing Arduino Libraries

I’m trying to set up unit tests for an Arduino library which kind of works. However, I’d like to test if the library examples compile for a couple of boards.
To achieve this, I set up a unit test for each of the examples like this:

#include <unity.h>
#include "../examples/example1/example1.ino"

int main(int argc, char** argv)
    RUN_TEST([] { TEST_PASS(); });

A pio test walks through all the examples, compiles and uploads the example. So far, so good. To save time in this step I’d prefer to only compile and check for compilation errors. So, I tried

'pio test --without-uploading --without-testing ’

which is only compiling. The problem is that it ignores tests which compile without error and only lists the ones with errors which is confusing.
Here the output of such a run:

C:\Users\lutz\source\teensy\01_Scratch\unit_test_testing\lib\myLib>pio test  --without-uploading --without-testing --filter */test_example1
Verbosity level can be increased via `-v, -vv, or -vvv` option
Collected 2 tests

Processing compileExamples/test_example1 in t41 environment
teensy_size: Memory Usage on Teensy 4.1:
teensy_size:   FLASH: code:11052, data:3992, headers:8504   free for files:8102916
teensy_size:    RAM1: variables:4672, code:8384, padding:24384   free for local variables:486848
teensy_size:    RAM2: variables:12384  free for malloc/new:511904
----------------------------------------------------------- t41:compileExamples/test_example1 [PASSED] Took 1.86 seconds -----------------------------------------------------------
Processing compileExamples/test_example1 in uno environment
src\myLib.cpp: In function 'void doBlink(unsigned int)':
src\myLib.cpp:9:9: error: 'digitalWriteFast' was not declared in this scope
         digitalWriteFast(LED_BUILTIN, LOW);
src\myLib.cpp:9:9: note: suggested alternative: 'digitalWrite'
         digitalWriteFast(LED_BUILTIN, LOW);
src\myLib.cpp:15:9: error: 'digitalToggleFast' was not declared in this scope
src\myLib.cpp:15:9: note: suggested alternative: 'digitalRead'
*** [.pio\build\uno\src\myLib.cpp.o] Error 1
Building stage has failed, see errors above. Use `pio test -vvv` option to enable verbose output.
---------------------------------------------------------- uno:compileExamples/test_example1 [ERRORED] Took 1.16 seconds ---------------------------------------------------------- 

===================================================================================== SUMMARY ===================================================================================== 
Environment    Test                           Status    Duration
-------------  -----------------------------  --------  ------------
uno            compileExamples/test_example1  ERRORED   00:00:01.156
==================================================================== 1 test cases: 0 succeeded in 00:00:03.018 ==================================================================== 


As expected the compilation passes for the Teensy and fails for the Uno. The summary at the end only prints the failed attempt for the uno and ignores the passed attempt for the Teensy. Don’t know if this is a bug or a feature. In any case it is confusing.
If needed, I can upload the full test project to GitHub

It might be a special case that if compilation of the unit tests errors out, you get that result.

All in all, it’s a very unusual way to test succesull compilation of a sketch by #include-ing the sketch. You would do this with a pio ci command as e.g. seen here, nicely pairable with e.g. Github.

Thanks for the pointer! Looks promising indeed, I’ll give it a try.