Code coverage issue on native

Advanced Solutions Unit Testing
1

Hi everyone

I ran into a weird issue since pio v6, and upgrade of googletest to v1.11, still working for the respiraworks ventilator project.
I am currently working on a specific branch with restructured tests for the new versions (thanks again for the quick response on my previous topic!), and our CI fails to report proper coverage from those tests.
After a little investigation, it appears that our unit tests (on native, using googletest and compiled with --coverage and -lgcov) sometimes do not create any gcda files during tests execution (even though the tests succeed). Though other times they create them as expected.

Any ideas as to what might cause that?

2

Hi @asmodai27, thanks for contacting us! Also, thanks for using PlatformIO. We added your project to our marketing content and will promote it soon.

Yes, I saw your project personally and had plans to contact you. I didn’t like the folder structure, especially how do you use test_ folders. The test_ folder is intended for isolated test cases. For example, the files from this folder could be moved to the test_core folder.

Could you run your test with pio test -vvv? It will show all flags that are passed to the compiler.

3

Yes, that structure is very much a work in progress, which I tried to see if it would help get the coverage working again: the common libraries test was structured like that (by mistake) and worked (which I now realize was by chance).

I get the following options when running pio test -e native -vvv for building libraries (omitting duplicate lines):

g++ -o .pio/build/native/test/test_sensor/sensors_test.o -c -std=gnu++17 -fno-exceptions -Wall -Werror -Wconversion -Wno-sign-conversion -Wno-sign-compare -pthread -g --coverage -pthread -DPLATFORMIO=60002 -DUNIT_TEST -DPIO_UNIT_TESTING -I. -I/data/RespiraWorks/VentilatorSoftware/software/common/test_libs/custom_google -I.pio/libdeps/native/googletest -I.pio/libdeps/native/googletest/googlemock -I.pio/libdeps/native/googletest/googlemock/include -I.pio/libdeps/native/googletest/googletest -I.pio/libdeps/native/googletest/googletest/include -Ilib/debug -I/data/RespiraWorks/VentilatorSoftware/software/common/libs/binary_utils -Ilib/core -Ilib/sensors -Ilib/pid -Ilib/actuators -Ilib/non_volatile -I/data/RespiraWorks/VentilatorSoftware/software/common/libs/checksum -Ilib/debugvars -I/data/RespiraWorks/VentilatorSoftware/software/common/generated_libs/network_protocol -I/data/RespiraWorks/VentilatorSoftware/software/common/third_party/nanopb -Ilib/hal -I/data/RespiraWorks/VentilatorSoftware/software/common/libs/framing -I/data/RespiraWorks/VentilatorSoftware/software/common/libs/units -Itest/test_sensor -Itest -I/home/asmodai/.platformio/platforms/native/builder/common/include test/test_sensor/sensors_test.cpp
ar rc .pio/build/native/lib58d/libunits.a .pio/build/native/lib58d/units/units.o
ranlib .pio/build/native/lib58d/libunits.a
gcc -DPLATFORMIO=60002 -DUNIT_TEST -DPIO_UNIT_TESTING -Ilib/hal -I/data/RespiraWorks/VentilatorSoftware/software/common/libs/framing -I/data/RespiraWorks/VentilatorSoftware/software/common/libs/units -Itest/test_sensor -Itest -I/home/asmodai/.platformio/platforms/native/builder/common/include -c -o .pio/build/native/lib5ad/hal/startup_stm32l452xx.o lib/hal/startup_stm32l452xx.S

And for building the program itself:

g++ -o .pio/build/native/program -pthread -pthread -fsanitize=undefined -fsanitize=address .pio/build/native/test/test_sensor/sensors_test.o -L.pio/build/native -Wl,--start-group .pio/build/native/lib58d/libunits.a .pio/build/native/lib5ad/libhal.a .pio/build/native/lib87a/libnanopb.a .pio/build/native/lib383/libnetwork_protocol.a .pio/build/native/lib211/libdebugvars.a .pio/build/native/libfab/libchecksum.a .pio/build/native/lib388/libnon_volatile.a .pio/build/native/lib3f9/libactuators.a .pio/build/native/libafc/libpid.a .pio/build/native/libb1f/libsensors.a .pio/build/native/lib61e/libcore.a .pio/build/native/lib298/libbinary_utils.a .pio/build/native/lib511/libdebug.a .pio/build/native/lib823/libgoogletest.a .pio/build/native/lib50a/libcustom_google.a -lgcov -Wl,--end-group

Also, new information: when I run this several times in a row, though I may have only a few gcda files during the first run, the build process is not re-done, and some more gcda files may (or may not) generated, and (after a few runs) I sometimes have all the files I want.

Thank you for you interest, this is very welcome!

4

I see only duplicated flags. You don’t need to pass -pthread manually. Nevertheless, I don’t think this is a problem.

Please update this branch. So, we can test it.

5

Indeed, removing -pthread from build_flags did not change the results.

I have updated the test structure to be more sensible, removed -pthread from build_flags on the branch, and pushed it to the github repo.
As you expected, this did not change the coverage reports.

Thank you very much for your help!

6

Hi, any new regarding this issue?
We are hitting more and more errors on our CI due to the fact that we have kept platformio back to v5.2.5 in the interim, first it was an incompatibility with native@1.2.2, and now we hit a new error during build with platformio 5.2.5 (this is on CI but I have the same errors on my machine):

ModuleNotFoundError: No module named 'platformio.public':
  File "/opt/circleci/.pyenv/versions/3.10.3/lib/python3.10/site-packages/platformio/builder/main.py", line 181:
    env.SConscript("$BUILD_SCRIPT")
  File "/home/circleci/.platformio/packages/tool-scons/scons-local-4.3.0/SCons/Script/SConscript.py", line 597:
    return _SConscript(self.fs, *files, **subst_kw)
  File "/home/circleci/.platformio/packages/tool-scons/scons-local-4.3.0/SCons/Script/SConscript.py", line 285:
    exec(compile(scriptdata, scriptname, 'exec'), call_stack[-1].globals)
  File "/home/circleci/.platformio/platforms/ststm32/builder/main.py", line 23:
    from platformio.public import list_serial_ports

I guess we could make it work again by keeping more and more versions back, but migrating our project to use the latest versions would be so much better…

7

The error comes from using the latest STSTM32 platform version which uses the import platformio.public line that requires core version >= 6. Specifically, that changed in Add compatibility with PIO Core 6.0 · platformio/platform-ststm32@99f9bc0 · GitHub

So, try using

platform = ststm32@15.3.0

in the platformio.ini.

8

Could you help us to debug this issue? You can build a project in --verbose mode with PIO Core v5 and v6. Try to compare flags. Do you see any difference?

9

Problem is that it no longer builds properly even with platformio 5.2.5 due to other dependencies having been updated.
The ststm32 build is now fixed thanks to @maxgerhardt solution, but the native unit tests builds also hits a new error: Error: Unknown Unit Test transport 'None', and I can’t even see the build options.

10

Found a machine which still had the old libraries run the tests with platformio v5.2.5, and this resulted in the following build options (again I removed the ones that had the same options):

g++ -o .pio/build/native/test/sensor_tests/sensors_test.o -c -std=gnu++17 -fno-exceptions -Wall -Werror -Wconversion -Wno-sign-conversion -Wno-sign-compare -pthread -g --coverage -DPLATFORMIO=50205 -DUNIT_TEST -DUNITY_INCLUDE_CONFIG_H -I. -Ilib/debug -I/home/admin/ventilator/software/common/libs/binary_utils -Ilib/serial -I/home/admin/ventilator/software/common/libs/framing -I/home/admin/ventilator/software/common/libs/proto_traits -Ilib/sensors -Ilib/pid -Ilib/actuators -Ilib/core -Ilib/non_volatile -I/home/admin/ventilator/software/common/libs/checksum -Ilib/debugvars -I/home/admin/ventilator/software/common/generated_libs/network_protocol -I/home/admin/ventilator/software/common/third_party/nanopb -Ilib/hal -I/home/admin/ventilator/software/common/libs/units -I.pio/libdeps/native/googletest -I.pio/libdeps/native/googletest/googlemock/include -I.pio/libdeps/native/googletest/googlemock -I.pio/libdeps/native/googletest/googletest/include -I.pio/libdeps/native/googletest/googletest -I/home/admin/.platformio/platforms/native/builder/common/include -I.pio/build/native/UnityTestLib -I/home/admin/.platformio/packages/tool-unity test/sensor_tests/sensors_test.cpp
ar rc .pio/build/native/libUnityTestLib.a .pio/build/native/UnityTestLib/unity.o
ranlib .pio/build/native/libUnityTestLib.a
gcc -DPLATFORMIO=50205 -DUNIT_TEST -DUNITY_INCLUDE_CONFIG_H -Ilib/hal -I/home/admin/ventilator/software/common/libs/units -I/home/admin/.platformio/platforms/native/builder/common/include -I.pio/build/native/UnityTestLib -I/home/admin/.platformio/packages/tool-unity -c -o .pio/build/native/lib29f/hal/startup_stm32l452xx.o lib/hal/startup_stm32l452xx.S
g++ -o .pio/build/native/program -pthread -fsanitize=undefined -fsanitize=address .pio/build/native/test/sensor_tests/sensors_test.o .pio/build/native/test/tmp_pio_test_transport.o -L.pio/build/native -Wl,--start-group .pio/build/native/lib540/libgoogletest.a .pio/build/native/lib39c/libunits.a .pio/build/native/lib29f/libhal.a .pio/build/native/libee7/libnanopb.a .pio/build/native/lib290/libnetwork_protocol.a .pio/build/native/lib2b4/libdebugvars.a .pio/build/native/lib6dd/libchecksum.a .pio/build/native/lib548/libnon_volatile.a .pio/build/native/lib9d6/libactuators.a .pio/build/native/lib689/libpid.a .pio/build/native/lib208/libsensors.a .pio/build/native/lib983/libproto_traits.a .pio/build/native/libeeb/libserial.a .pio/build/native/lib848/libcore.a .pio/build/native/libbb4/libbinary_utils.a .pio/build/native/lib4e3/libdebug.a .pio/build/native/libUnityTestLib.a -lgcov -Wl,--end-group

Biggest difference I notice is that v5.2.5 builds .pio/build/native/test/tmp_pio_test_transport.o and includes -I.pio/build/native/UnityTestLib -I/home/admin/.platformio/packages/tool-unity

On that machine, I have:

$ pio platform list
native ~ Native
===============
Native development platform is intended to be used for desktop OS. This platform uses built-in toolchains (preferable based on GCC), frameworks, libs from particular OS where it will be run.

Home: http://platformio.org/platforms/native
Packages: 
Version: 1.1.4

ststm32 ~ ST STM32
==================
The STM32 family of 32-bit Flash MCUs based on the ARM Cortex-M processor is designed to offer new degrees of freedom to MCU users. It offers a 32-bit product range that combines very high performance, real-time capabilities, digital signal processing, and low-power, low-voltage operation, while maintaining full integration and ease of development.

Home: http://www.st.com/web/en/catalog/mmc/FM141/SC1169?sc=stm32
Frameworks: arduino, cmsis, libopencm3, mbed, spl, stm32cube, zephyr
Packages: toolchain-gccarmnoneeabi, framework-mbed, framework-cmsis, framework-cmsis-stm32f0, framework-cmsis-stm32f1, framework-cmsis-stm32f2, framework-cmsis-stm32f3, framework-cmsis-stm32f4, framework-cmsis-stm32f7, framework-cmsis-stm32g0, framework-cmsis-stm32g4, framework-cmsis-stm32h7, framework-cmsis-stm32l0, framework-cmsis-stm32l1, framework-cmsis-stm32l4, framework-cmsis-stm32l5, framework-spl, framework-libopencm3, framework-arduinoststm32, framework-arduinoststm32-maple, framework-arduinostm32mxchip, framework-arduinoststm32l0, framework-arduino-mbed, framework-stm32cubef0, framework-stm32cubef1, framework-stm32cubef2, framework-stm32cubef3, framework-stm32cubef4, framework-stm32cubef7, framework-stm32cubeg0, framework-stm32cubeg4, framework-stm32cubeh7, framework-stm32cubel0, framework-stm32cubel1, framework-stm32cubel4, framework-stm32cubel5, framework-zephyr, tool-stm32duino, tool-openocd, tool-jlink, tool-dfuutil, tool-cmake, tool-dtc, tool-ninja, tool-gperf, tool-ldscripts-ststm32
Version: 15.3.0

ststm32 ~ ST STM32
==================
The STM32 family of 32-bit Flash MCUs based on the ARM Cortex-M processor is designed to offer new degrees of freedom to MCU users. It offers a 32-bit product range that combines very high performance, real-time capabilities, digital signal processing, and low-power, low-voltage operation, while maintaining full integration and ease of development.

Home: http://www.st.com/web/en/catalog/mmc/FM141/SC1169?sc=stm32
Frameworks: arduino, cmsis, libopencm3, mbed, spl, stm32cube, zephyr
Packages: toolchain-gccarmnoneeabi, framework-mbed, framework-cmsis, framework-cmsis-stm32f0, framework-cmsis-stm32f1, framework-cmsis-stm32f2, framework-cmsis-stm32f3, framework-cmsis-stm32f4, framework-cmsis-stm32f7, framework-cmsis-stm32g0, framework-cmsis-stm32g4, framework-cmsis-stm32h7, framework-cmsis-stm32l0, framework-cmsis-stm32l1, framework-cmsis-stm32l4, framework-cmsis-stm32l5, framework-spl, framework-libopencm3, framework-arduinoststm32, framework-arduinoststm32-maple, framework-arduinostm32mxchip, framework-arduinoststm32l0, framework-arduino-mbed, framework-stm32cubef0, framework-stm32cubef1, framework-stm32cubef2, framework-stm32cubef3, framework-stm32cubef4, framework-stm32cubef7, framework-stm32cubeg0, framework-stm32cubeg4, framework-stm32cubeh7, framework-stm32cubel0, framework-stm32cubel1, framework-stm32cubel4, framework-stm32cubel5, framework-zephyr, tool-stm32duino, tool-openocd, tool-jlink, tool-dfuutil, tool-cmake, tool-dtc, tool-ninja, tool-gperf, tool-ldscripts-ststm32
Version: 15.2.0

$ pio lib list
Library Storage: /home/admin/ventilator/software/controller/.pio/libdeps/stm32
No items found

Library Storage: /home/admin/ventilator/software/controller/.pio/libdeps/stm32-uart
No items found

Library Storage: /home/admin/ventilator/software/controller/.pio/libdeps/integration-test
No items found

Library Storage: /home/admin/ventilator/software/controller/.pio/libdeps/native
googletest
==========
googletest is a testing framework developed by the Testing Technology team with Google's specific requirements and constraints in mind. No matter whether you work on Linux, Windows, or a Mac, if you write C++ code, googletest can help you. And it supports any kind of tests, not just unit tests.

Version: 1.10.0
Homepage: https://github.com/google/googletest/blob/master/README.md
License: BSD-3-Clause
Keywords: unittest, unit, test, gtest, gmock
Compatible frameworks: arduino
Compatible platforms: espressif32, espressif8266
11

That’s an older version, the same thing goes for Releases · platformio/platform-native · GitHub as for STSTM32: Try using platform = native@1.1.4.

12

What should I do on my local machine to reproduce this bug? Could you provide simple steps? I’m not familiar with your project, please help.

13

For the initial one (which would make fixing the second one unnecessary), starting from a clean clone of the repo:

$ git checkout jtravert/upgrade_to_pio_6.0
$ cd software/controller

$ ./controller.sh install #will install some dependencies most notably our toolchain, platformio and other python modules
$ # alternately, since you probably already have platformio installed, and you will only be running unit tests, you can probably only install the toolchain: sudo apt install -y build-essential gcovr lcov
$ pio test -e native

Check the creation of *.gcda files in .pio/build/native. we expect over 90 files, most of the times we only get a handful (10 to 30) of those.
Run pio test -e native again, a few more files are created.

If you want to restart from zero of those files, just clean the build folder: rm -rf .pio/build.

14

Sorry for the delay. We were busy with the upcoming PlatformIO Core 6.0.3. I see you already removed that branch. Did you solve the migration task?

15

Hi, no, we didn’t solve the problem, but we merged it to master, as the other issue (unit tests failing with v5.2.5) made it even worse than coverage reports being off, and blocked any other progress, and I forgot that our repo is setup to automatically delete merged branches…

16

Could you provide updated steps on how to reproduce this issue? Thanks.

17

Same as before but without the branch change:

# if you don't already have it, clone the repo
$ git clone https://github.com/RespiraWorks/Ventilator.git ventilator
$ cd ventilator/software/controller

$ ./controller.sh install #will install some dependencies most notably our toolchain, platformio and other python modules
# alternately, since you probably already have platformio installed, and you will only be running unit tests, you can probably only install the toolchain: sudo apt install -y build-essential gcovr lcov
$ pio test -e native

Check the creation of *.gcda files in .pio/build/native. we expect over 90 files, most of the times we only get a handful (10 to 30) of those.
Run pio test -e native again, a few more files are created.

18

Built a minimal example to make it easier to reproduce:

I built a small script as well to facilitate the interpretation of results, running ./regenerate_coverage.sh will clean the directories, run the tests and then generate html files from the *.gcda (and *.gcno) files (using lcov), and while it most often yields proper results (see expected result) but sometimes (one in 10 maybe?) yields partial results (see occasional result).

Seeing as our main project has many more unit tests, it follows that if one in ten unit test runs does not generate coverage reports, our coverage reports are plain wrong.

As before, make sure you have platformio, build-essential, gcovr and lcov installed.

1 Like
19

I’ve just used the latest PlatformIO Core 6.1-dev (pio upgrade --dev), it comes with the latest GoogleTest 1.12.1, and got this result:

Screen Shot 2022-07-01 at 21.20.20
Screen Shot 2022-07-01 at 21.20.203306Ă—634 220 KB

Is it good or bad? I used my virtual machine with the latest Ubuntu.

pio system info
--------------------------  ---------------------------------------------
PlatformIO Core             6.1.0b1
Python                      3.10.4-final.0
System Type                 linux_x86_64
Platform                    Linux-5.15.0-35-generic-x86_64-with-glibc2.35
File System Encoding        utf-8
Locale Encoding             UTF-8
PlatformIO Core Directory   /root/.platformio
PlatformIO Core Executable  /tmp/piodev/bin/platformio
Python Executable           /tmp/piodev/bin/python3
Global Libraries            0
Development Platforms       3
Tools & Toolchains          7
--------------------------  ---------------------------------------------

gcc --version
gcc (Ubuntu 11.2.0-19ubuntu1) 11.2.0
Copyright (C) 2021 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.  There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
20

This is what we expect, though as I said, the problem is somewhat random, and we get correct results on this example more often than incorrect ones.
I am trying to refine it so the problem is more systematic.

I just tested platformio core 6.1.0b1 on the full project (though with an older version of gcc), and I still have an average of half of the tests that properly generate gcda files.

Note that it happens on CI as well as my machine. On my machine:

$ pio system info
--------------------------  ---------------------------------------------
PlatformIO Core             6.1.0b1
Python                      3.8.10-final.0
System Type                 linux_x86_64
Platform                    Linux-5.4.0-120-generic-x86_64-with-glibc2.29
File System Encoding        utf-8
Locale Encoding             UTF-8
PlatformIO Core Directory   /home/asmodai/.platformio
PlatformIO Core Executable  /home/asmodai/.platformio/penv/bin/platformio
Python Executable           /home/asmodai/.platformio/penv/bin/python
Global Libraries            0
Development Platforms       2
Tools & Toolchains          9
--------------------------  ---------------------------------------------

$ gcc --version
gcc (Ubuntu 9.4.0-1ubuntu1~20.04.1) 9.4.0
Copyright (C) 2019 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.  There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.