Code coverage issue on native

Advanced Solutions Unit Testing
10

Found a machine which still had the old libraries run the tests with platformio v5.2.5, and this resulted in the following build options (again I removed the ones that had the same options):

g++ -o .pio/build/native/test/sensor_tests/sensors_test.o -c -std=gnu++17 -fno-exceptions -Wall -Werror -Wconversion -Wno-sign-conversion -Wno-sign-compare -pthread -g --coverage -DPLATFORMIO=50205 -DUNIT_TEST -DUNITY_INCLUDE_CONFIG_H -I. -Ilib/debug -I/home/admin/ventilator/software/common/libs/binary_utils -Ilib/serial -I/home/admin/ventilator/software/common/libs/framing -I/home/admin/ventilator/software/common/libs/proto_traits -Ilib/sensors -Ilib/pid -Ilib/actuators -Ilib/core -Ilib/non_volatile -I/home/admin/ventilator/software/common/libs/checksum -Ilib/debugvars -I/home/admin/ventilator/software/common/generated_libs/network_protocol -I/home/admin/ventilator/software/common/third_party/nanopb -Ilib/hal -I/home/admin/ventilator/software/common/libs/units -I.pio/libdeps/native/googletest -I.pio/libdeps/native/googletest/googlemock/include -I.pio/libdeps/native/googletest/googlemock -I.pio/libdeps/native/googletest/googletest/include -I.pio/libdeps/native/googletest/googletest -I/home/admin/.platformio/platforms/native/builder/common/include -I.pio/build/native/UnityTestLib -I/home/admin/.platformio/packages/tool-unity test/sensor_tests/sensors_test.cpp
ar rc .pio/build/native/libUnityTestLib.a .pio/build/native/UnityTestLib/unity.o
ranlib .pio/build/native/libUnityTestLib.a
gcc -DPLATFORMIO=50205 -DUNIT_TEST -DUNITY_INCLUDE_CONFIG_H -Ilib/hal -I/home/admin/ventilator/software/common/libs/units -I/home/admin/.platformio/platforms/native/builder/common/include -I.pio/build/native/UnityTestLib -I/home/admin/.platformio/packages/tool-unity -c -o .pio/build/native/lib29f/hal/startup_stm32l452xx.o lib/hal/startup_stm32l452xx.S
g++ -o .pio/build/native/program -pthread -fsanitize=undefined -fsanitize=address .pio/build/native/test/sensor_tests/sensors_test.o .pio/build/native/test/tmp_pio_test_transport.o -L.pio/build/native -Wl,--start-group .pio/build/native/lib540/libgoogletest.a .pio/build/native/lib39c/libunits.a .pio/build/native/lib29f/libhal.a .pio/build/native/libee7/libnanopb.a .pio/build/native/lib290/libnetwork_protocol.a .pio/build/native/lib2b4/libdebugvars.a .pio/build/native/lib6dd/libchecksum.a .pio/build/native/lib548/libnon_volatile.a .pio/build/native/lib9d6/libactuators.a .pio/build/native/lib689/libpid.a .pio/build/native/lib208/libsensors.a .pio/build/native/lib983/libproto_traits.a .pio/build/native/libeeb/libserial.a .pio/build/native/lib848/libcore.a .pio/build/native/libbb4/libbinary_utils.a .pio/build/native/lib4e3/libdebug.a .pio/build/native/libUnityTestLib.a -lgcov -Wl,--end-group

Biggest difference I notice is that v5.2.5 builds .pio/build/native/test/tmp_pio_test_transport.o and includes -I.pio/build/native/UnityTestLib -I/home/admin/.platformio/packages/tool-unity

On that machine, I have:

$ pio platform list
native ~ Native
===============
Native development platform is intended to be used for desktop OS. This platform uses built-in toolchains (preferable based on GCC), frameworks, libs from particular OS where it will be run.

Home: http://platformio.org/platforms/native
Packages: 
Version: 1.1.4

ststm32 ~ ST STM32
==================
The STM32 family of 32-bit Flash MCUs based on the ARM Cortex-M processor is designed to offer new degrees of freedom to MCU users. It offers a 32-bit product range that combines very high performance, real-time capabilities, digital signal processing, and low-power, low-voltage operation, while maintaining full integration and ease of development.

Home: http://www.st.com/web/en/catalog/mmc/FM141/SC1169?sc=stm32
Frameworks: arduino, cmsis, libopencm3, mbed, spl, stm32cube, zephyr
Packages: toolchain-gccarmnoneeabi, framework-mbed, framework-cmsis, framework-cmsis-stm32f0, framework-cmsis-stm32f1, framework-cmsis-stm32f2, framework-cmsis-stm32f3, framework-cmsis-stm32f4, framework-cmsis-stm32f7, framework-cmsis-stm32g0, framework-cmsis-stm32g4, framework-cmsis-stm32h7, framework-cmsis-stm32l0, framework-cmsis-stm32l1, framework-cmsis-stm32l4, framework-cmsis-stm32l5, framework-spl, framework-libopencm3, framework-arduinoststm32, framework-arduinoststm32-maple, framework-arduinostm32mxchip, framework-arduinoststm32l0, framework-arduino-mbed, framework-stm32cubef0, framework-stm32cubef1, framework-stm32cubef2, framework-stm32cubef3, framework-stm32cubef4, framework-stm32cubef7, framework-stm32cubeg0, framework-stm32cubeg4, framework-stm32cubeh7, framework-stm32cubel0, framework-stm32cubel1, framework-stm32cubel4, framework-stm32cubel5, framework-zephyr, tool-stm32duino, tool-openocd, tool-jlink, tool-dfuutil, tool-cmake, tool-dtc, tool-ninja, tool-gperf, tool-ldscripts-ststm32
Version: 15.3.0

ststm32 ~ ST STM32
==================
The STM32 family of 32-bit Flash MCUs based on the ARM Cortex-M processor is designed to offer new degrees of freedom to MCU users. It offers a 32-bit product range that combines very high performance, real-time capabilities, digital signal processing, and low-power, low-voltage operation, while maintaining full integration and ease of development.

Home: http://www.st.com/web/en/catalog/mmc/FM141/SC1169?sc=stm32
Frameworks: arduino, cmsis, libopencm3, mbed, spl, stm32cube, zephyr
Packages: toolchain-gccarmnoneeabi, framework-mbed, framework-cmsis, framework-cmsis-stm32f0, framework-cmsis-stm32f1, framework-cmsis-stm32f2, framework-cmsis-stm32f3, framework-cmsis-stm32f4, framework-cmsis-stm32f7, framework-cmsis-stm32g0, framework-cmsis-stm32g4, framework-cmsis-stm32h7, framework-cmsis-stm32l0, framework-cmsis-stm32l1, framework-cmsis-stm32l4, framework-cmsis-stm32l5, framework-spl, framework-libopencm3, framework-arduinoststm32, framework-arduinoststm32-maple, framework-arduinostm32mxchip, framework-arduinoststm32l0, framework-arduino-mbed, framework-stm32cubef0, framework-stm32cubef1, framework-stm32cubef2, framework-stm32cubef3, framework-stm32cubef4, framework-stm32cubef7, framework-stm32cubeg0, framework-stm32cubeg4, framework-stm32cubeh7, framework-stm32cubel0, framework-stm32cubel1, framework-stm32cubel4, framework-stm32cubel5, framework-zephyr, tool-stm32duino, tool-openocd, tool-jlink, tool-dfuutil, tool-cmake, tool-dtc, tool-ninja, tool-gperf, tool-ldscripts-ststm32
Version: 15.2.0

$ pio lib list
Library Storage: /home/admin/ventilator/software/controller/.pio/libdeps/stm32
No items found

Library Storage: /home/admin/ventilator/software/controller/.pio/libdeps/stm32-uart
No items found

Library Storage: /home/admin/ventilator/software/controller/.pio/libdeps/integration-test
No items found

Library Storage: /home/admin/ventilator/software/controller/.pio/libdeps/native
googletest
==========
googletest is a testing framework developed by the Testing Technology team with Google's specific requirements and constraints in mind. No matter whether you work on Linux, Windows, or a Mac, if you write C++ code, googletest can help you. And it supports any kind of tests, not just unit tests.

Version: 1.10.0
Homepage: https://github.com/google/googletest/blob/master/README.md
License: BSD-3-Clause
Keywords: unittest, unit, test, gtest, gmock
Compatible frameworks: arduino
Compatible platforms: espressif32, espressif8266
11

That’s an older version, the same thing goes for Releases · platformio/platform-native · GitHub as for STSTM32: Try using platform = native@1.1.4.

12

What should I do on my local machine to reproduce this bug? Could you provide simple steps? I’m not familiar with your project, please help.

13

For the initial one (which would make fixing the second one unnecessary), starting from a clean clone of the repo:

$ git checkout jtravert/upgrade_to_pio_6.0
$ cd software/controller

$ ./controller.sh install #will install some dependencies most notably our toolchain, platformio and other python modules
$ # alternately, since you probably already have platformio installed, and you will only be running unit tests, you can probably only install the toolchain: sudo apt install -y build-essential gcovr lcov
$ pio test -e native

Check the creation of *.gcda files in .pio/build/native. we expect over 90 files, most of the times we only get a handful (10 to 30) of those.
Run pio test -e native again, a few more files are created.

If you want to restart from zero of those files, just clean the build folder: rm -rf .pio/build.

14

Sorry for the delay. We were busy with the upcoming PlatformIO Core 6.0.3. I see you already removed that branch. Did you solve the migration task?

15

Hi, no, we didn’t solve the problem, but we merged it to master, as the other issue (unit tests failing with v5.2.5) made it even worse than coverage reports being off, and blocked any other progress, and I forgot that our repo is setup to automatically delete merged branches…

16

Could you provide updated steps on how to reproduce this issue? Thanks.

17

Same as before but without the branch change:

# if you don't already have it, clone the repo
$ git clone https://github.com/RespiraWorks/Ventilator.git ventilator
$ cd ventilator/software/controller

$ ./controller.sh install #will install some dependencies most notably our toolchain, platformio and other python modules
# alternately, since you probably already have platformio installed, and you will only be running unit tests, you can probably only install the toolchain: sudo apt install -y build-essential gcovr lcov
$ pio test -e native

Check the creation of *.gcda files in .pio/build/native. we expect over 90 files, most of the times we only get a handful (10 to 30) of those.
Run pio test -e native again, a few more files are created.

18

Built a minimal example to make it easier to reproduce:

I built a small script as well to facilitate the interpretation of results, running ./regenerate_coverage.sh will clean the directories, run the tests and then generate html files from the *.gcda (and *.gcno) files (using lcov), and while it most often yields proper results (see expected result) but sometimes (one in 10 maybe?) yields partial results (see occasional result).

Seeing as our main project has many more unit tests, it follows that if one in ten unit test runs does not generate coverage reports, our coverage reports are plain wrong.

As before, make sure you have platformio, build-essential, gcovr and lcov installed.

1 Like
19

I’ve just used the latest PlatformIO Core 6.1-dev (pio upgrade --dev), it comes with the latest GoogleTest 1.12.1, and got this result:

Screen Shot 2022-07-01 at 21.20.20
Screen Shot 2022-07-01 at 21.20.203306×634 220 KB

Is it good or bad? I used my virtual machine with the latest Ubuntu.

pio system info
--------------------------  ---------------------------------------------
PlatformIO Core             6.1.0b1
Python                      3.10.4-final.0
System Type                 linux_x86_64
Platform                    Linux-5.15.0-35-generic-x86_64-with-glibc2.35
File System Encoding        utf-8
Locale Encoding             UTF-8
PlatformIO Core Directory   /root/.platformio
PlatformIO Core Executable  /tmp/piodev/bin/platformio
Python Executable           /tmp/piodev/bin/python3
Global Libraries            0
Development Platforms       3
Tools & Toolchains          7
--------------------------  ---------------------------------------------

gcc --version
gcc (Ubuntu 11.2.0-19ubuntu1) 11.2.0
Copyright (C) 2021 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.  There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
20

This is what we expect, though as I said, the problem is somewhat random, and we get correct results on this example more often than incorrect ones.
I am trying to refine it so the problem is more systematic.

I just tested platformio core 6.1.0b1 on the full project (though with an older version of gcc), and I still have an average of half of the tests that properly generate gcda files.

Note that it happens on CI as well as my machine. On my machine:

$ pio system info
--------------------------  ---------------------------------------------
PlatformIO Core             6.1.0b1
Python                      3.8.10-final.0
System Type                 linux_x86_64
Platform                    Linux-5.4.0-120-generic-x86_64-with-glibc2.29
File System Encoding        utf-8
Locale Encoding             UTF-8
PlatformIO Core Directory   /home/asmodai/.platformio
PlatformIO Core Executable  /home/asmodai/.platformio/penv/bin/platformio
Python Executable           /home/asmodai/.platformio/penv/bin/python
Global Libraries            0
Development Platforms       2
Tools & Toolchains          9
--------------------------  ---------------------------------------------

$ gcc --version
gcc (Ubuntu 9.4.0-1ubuntu1~20.04.1) 9.4.0
Copyright (C) 2019 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.  There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
22

I updated the regenerate_coverage.sh script in that minimal example so it loops until the problem happens.
This probably won’t make it much easier to investigate, but should make it easier to see.

EDIT: I also just added a hard limit to 50 iterations and a report of the number of iterations to make the bug appear (or not), which averages around 4 runs (but can be anything between 1 and 15) on my machine.

23

We don’t provide any random build flags or commands. You can even compare the verbose output from the multiple “run -v” iterations. Could be an issue with OS/compiler version?

24

Well, the issue appeared with platformio v6.0, and could be resolved by going back to v5.2.5 (up until the point that libraries started loosing backwards-compatibility and it became hard to maintain).
It doesn’t seem to depend on the gcc/OS versions as it appears on several combinations (my machine, CI and other people on the project), which all use different OS, and with several different gcc versions.

I understand that this is not a straight up platformio issue, but something changed in the way platformio does things which made it happen. I am also still investigating configration solutions on my end.

25

Just found some interesting piece of information regarding the way gcda files are generated:

Specifically this part caught my eyes:

At running time the statistic data is gathered and stored in memory. Some exit callback is registered and is called to write the data to the .gcda file when the program terminates. This means if you call abort() instead of exit() in your program, no .gcda file would be generated.

Is it possible that the fact we build a custom main() for each test instead of using the one from googletest changes the way the program terminates?

26

Thanks, an interesting note. This is a GoogleTest’s default main handler googletest/gtest_main.cc at main · google/googletest · GitHub

27

Sorry, I finally found the issue. Let me think about how to fix it.

1 Like
28

Fixed in

Could you re-run pio upgrade --dev. Does it produce all coverage files now?

Please note that you will need to update your “main” function of the google tests and return the 0 code instead of GoogleTest’s return code (which always is negative when tests failed). We updated “Get Started” guide for GoogleTest GoogleTest — PlatformIO latest documentation

29

Hello.

The coverage reports are generated properly with pio 6.1.0rc1. I still have some cleanup to do to get this to production, but that is awesome.

Thank you very much!

1 Like
30

Wow! What great news! Sorry again for the delay in fixing this problem. Glad that we finally solved all issues that turn away people from upgrading to PlatformIO Core 6.0

Happy coding with PlatformIO! :rocket: