PlatformIO Community

Test options seem ignored in Core 6.0.1

It looks like test options to filter for certain tests or ignore certain tests are no longer working in Core 6.0.1

Here is an example : trying to ignore test.cpp but you see PIO runs the test anyway.

PS C:\Users\Pascal\Documents\GitHub\Moovr> pio test -e native --ignore test.cpp
Verbose mode can be enabled via `-v, --verbose` option
Collected 1 tests

Processing * in native environment
----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------Building...
Testing...
test\test.cpp:68: initialization        [PASSED]
test\test.cpp:69: ignoreEventNone       [PASSED]
test\test.cpp:70: pushPop       [PASSED]
test\test.cpp:71: boundariesUnderflow   [PASSED]
test\test.cpp:72: boundariesOverflow    [PASSED]
---------------------------------------------------------------------- native:* [PASSED] Took 1.47 seconds ---------------------------------------------------------------------- 

==================================================================================== SUMMARY ==================================================================================== 
Environment    Test    Status    Duration
-------------  ------  --------  ------------
native         *       PASSED    00:00:01.467
=================================================================== 5 test cases: 5 succeeded in 00:00:01.467 ===================================================================

Furthermore, it looks like tests in subfolders are no longer found. If I move the test.cpp in a subfolder eg. test/abc/test.cpp and then run it, PIO does not find the test…

PS C:\Users\Pascal\Documents\GitHub\Moovr> pio test -e native                  
Verbose mode can be enabled via `-v, --verbose` option
Collected 1 tests

Processing * in native environment
----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------Building...
C:/msys64/mingw64/bin/../lib/gcc/x86_64-w64-mingw32/12.1.0/../../../../x86_64-w64-mingw32/bin/ld.exe: C:/msys64/mingw64/bin/../lib/gcc/x86_64-w64-mingw32/12.1.0/../../../../lib/libmingw32.a(lib64_libmingw32_a-crt0_c.o): in function `main':
C:/M/mingw-w64-crt-git/src/mingw-w64/mingw-w64-crt/crt/crt0_c.c:18: undefined reference to `WinMain'
collect2.exe: error: ld returned 1 exit status
*** [.pio\build\native\program.exe] Error 1
Building stage has failed, see errors above. Use `pio test --verbose` option to enable verbose output.
---------------------------------------------------------------------- native:* [ERRORED] Took 1.57 seconds ----------------------------------------------------------------------
==================================================================================== SUMMARY ==================================================================================== 
Environment    Test    Status    Duration
-------------  ------  --------  ------------
native         *       ERRORED   00:00:01.566
=================================================================== 1 test cases: 0 succeeded in 00:00:01.566 ===================================================================

PIO core 6 has major refactorings and changed documentation in regards to unit testing (Debugging — PlatformIO latest documentation).

Code with test cases must have folder names starting with test_, according to the documentation.

I think the --ignore <t> switch ignores the test fodler test_<t>.

1 Like

Yes, I can confirm this is indeed how it works. I had read the docs, but I think they could be improved a bit. I will post a proposal later today.

Small correction :
the --ignore <t> option ignores the folder <t>, not the folder test_<t>

Suggestion for the documentation:

If you have a single test-application you can place the sourcecode files directly in the test_dir folder.

If you want several test-applications, you can organize them in subfolders under test_dir .

  • each test-application goes into a subfolder with a name starting with test_ .

  • additional levels of folders are allowed to properly organize your tests - see example Pizza Project below.

  • the test_filter option in platformio.ini or the --filter option on the CLI will restrict the tests to only those test applications inside a folder with a path and foldername matching the filter.

  • the test_ignore option in platformio.ini or the --ignore option on the CLI will ignore all test applications inside a folder with a path and foldername matching the filter.

Example of using the filter / ignore option in platformio.ini for the Pizza Project below:
test_filter = embedded/stove/*
will only execute the embedded/stove/test_humidity and embedded/stove/test_temperature tests.

2 Likes

I’ve read several other people struggling with the docs too (e.g. Pio test broke with esp32doit-devkit-v1 in pio core 6.1 - #3 by robson.o.d). Can you post your improvement proposa in https://github.com/platformio/platformio-docs/?

Thank you so much for the suggestion! I’ve just docs with your proposals. :pray:

I could contribute it to the documentation, but I’m not sure what process you guys prefer.
I could fork the platformio-docs repository, make suggestions and do a pull request.
Or any other way. Let me know what you prefer.

Your text was already added to the docs in the latest commit in Improve docs for Unit Testing -> Structure · platformio/platformio-docs@24524ca · GitHub, so all is good.

1 Like