Test options seem ignored in Core 6.0.1

[strooom]

It looks like test options to filter for certain tests or ignore certain tests are no longer working in Core 6.0.1

Here is an example : trying to ignore test.cpp but you see PIO runs the test anyway.

PS C:\Users\Pascal\Documents\GitHub\Moovr> pio test -e native --ignore test.cpp
Verbose mode can be enabled via `-v, --verbose` option
Collected 1 tests

Processing * in native environment
----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------Building...
Testing...
test\test.cpp:68: initialization        [PASSED]
test\test.cpp:69: ignoreEventNone       [PASSED]
test\test.cpp:70: pushPop       [PASSED]
test\test.cpp:71: boundariesUnderflow   [PASSED]
test\test.cpp:72: boundariesOverflow    [PASSED]
---------------------------------------------------------------------- native:* [PASSED] Took 1.47 seconds ---------------------------------------------------------------------- 

==================================================================================== SUMMARY ==================================================================================== 
Environment    Test    Status    Duration
-------------  ------  --------  ------------
native         *       PASSED    00:00:01.467
=================================================================== 5 test cases: 5 succeeded in 00:00:01.467 ===================================================================

Furthermore, it looks like tests in subfolders are no longer found. If I move the test.cpp in a subfolder eg. test/abc/test.cpp and then run it, PIO does not find the test…

PS C:\Users\Pascal\Documents\GitHub\Moovr> pio test -e native                  
Verbose mode can be enabled via `-v, --verbose` option
Collected 1 tests

Processing * in native environment
----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------Building...
C:/msys64/mingw64/bin/../lib/gcc/x86_64-w64-mingw32/12.1.0/../../../../x86_64-w64-mingw32/bin/ld.exe: C:/msys64/mingw64/bin/../lib/gcc/x86_64-w64-mingw32/12.1.0/../../../../lib/libmingw32.a(lib64_libmingw32_a-crt0_c.o): in function `main':
C:/M/mingw-w64-crt-git/src/mingw-w64/mingw-w64-crt/crt/crt0_c.c:18: undefined reference to `WinMain'
collect2.exe: error: ld returned 1 exit status
*** [.pio\build\native\program.exe] Error 1
Building stage has failed, see errors above. Use `pio test --verbose` option to enable verbose output.
---------------------------------------------------------------------- native:* [ERRORED] Took 1.57 seconds ----------------------------------------------------------------------
==================================================================================== SUMMARY ==================================================================================== 
Environment    Test    Status    Duration
-------------  ------  --------  ------------
native         *       ERRORED   00:00:01.566
=================================================================== 1 test cases: 0 succeeded in 00:00:01.566 ===================================================================

[maxgerhardt]

PIO core 6 has major refactorings and changed documentation in regards to unit testing (Debugging — PlatformIO latest documentation).

@strooom

If I move the test.cpp in a subfolder eg. test/abc/test.cpp and then run

Code with test cases must have folder names starting with test_, according to the documentation.

@strooom

pio test -e native --ignore test.cpp

I think the --ignore <t> switch ignores the test fodler test_<t>.

[strooom]

Yes, I can confirm this is indeed how it works. I had read the docs, but I think they could be improved a bit. I will post a proposal later today.

[strooom]

Small correction :
the --ignore <t> option ignores the folder <t>, not the folder test_<t>

[strooom]

Suggestion for the documentation:

If you have a single test-application you can place the sourcecode files directly in the test_dir folder.

If you want several test-applications, you can organize them in subfolders under test_dir .

• each test-application goes into a subfolder with a name starting with test_ .

• additional levels of folders are allowed to properly organize your tests - see example Pizza Project below.

• the test_filter option in platformio.ini or the --filter option on the CLI will restrict the tests to only those test applications inside a folder with a path and foldername matching the filter.

• the test_ignore option in platformio.ini or the --ignore option on the CLI will ignore all test applications inside a folder with a path and foldername matching the filter.

Example of using the filter / ignore option in platformio.ini for the Pizza Project below:
test_filter = embedded/stove/*
will only execute the embedded/stove/test_humidity and embedded/stove/test_temperature tests.

[maxgerhardt]

I’ve read several other people struggling with the docs too (e.g. Pio test broke with esp32doit-devkit-v1 in pio core 6.1 - #3 by robson.o.d). Can you post your improvement proposa in GitHub - platformio/platformio-docs: PlatformIO Documentation?

[ivankravets]

Thank you so much for the suggestion! I’ve just docs with your proposals.

[strooom]

I could contribute it to the documentation, but I’m not sure what process you guys prefer.
I could fork the platformio-docs repository, make suggestions and do a pull request.
Or any other way. Let me know what you prefer.

[maxgerhardt]

Your text was already added to the docs in the latest commit in Improve docs for Unit Testing -> Structure · platformio/platformio-docs@24524ca · GitHub, so all is good.

[daven006]

Thanks; I’ve tried moving my test program to a sub folder (ie test/test_temp) and it still fails in the same way. NB the code I’m testing is in the src folder.

Directory structure (partly truncated):

Folder PATH listing
Volume serial number is C497-4E30
C:.
±–.pio
| ±–build
| | -–wemos_d1_mini32
| -–libdeps
| -–wemos_d1_mini32
| ±–ArduinoJson
| | ±–examples
| | | ±–JsonConfigFile
.
.
.
| | | ±–ProgmemExample
| | | -–StringExample
| | -–src
| | -–ArduinoJson
| | ±–Array
.
.
.
| | ±–StringStorage
| | -–Variant
| -–Unity
| ±–.github
| | -–workflows
| ±–auto
| ±–examples
| | ±–example_1
| | | ±–src
| | | -–test
| | | -–test_runners
.
.
.
| -–src
±–.vscode
±–data
±–include
±–lib
±–src
±–TempMapping
| -–.pio
| -–build
| -–wemos_d1_mini32
| ±–lib843
| | -–FS
| ±–lib99b
| | -–.pio
| | -–libdeps
| | -–wemos_d1_mini32
| | -–Unity
| | -–examples
| | -–example_1
| | ±–src
| | -–test
| | -–test_runners
| ±–libc64
| | -–src
| ±–libdf6
| | -–SPIFFS
| ±–test
| | -–test_temp
| ±–unity_config
| -–unity_config_build
-–test
-–test_temp