Unit Testing compile errors when using lib_extra_dirs

_joku · April 15, 2022, 3:18pm

Hello,
While working on a project (avaanchor directory) I have written a library (protocol directory) for de/serializing network packets, that I now also want to use in a different project (avanode directory). Moving the library a few levels up and including it using it with lib_extra_dirs = ${PROJECT_DIR}/../../common_libs worked fine and builds. However, there are unit tests in the test/ subdirectory of the first project that now won’t compile, because a header inside the moved library can’t be found.
This is my folder structure:

├── avaanchor
...
│   ├── feather-firmware
│   │   ├── include
│   │   │   └── README
│   │   ├── lib
│   │   │   └── README
│   │   ├── platformio.ini
│   │   ├── src
│   │   │   ├── main.cpp
│   │   │   └── send_lora_command.py
│   │   └── test
│   │       ├── README
│   │       └── test_protocol.cpp
...
├── avanode
│   └── README.md
├── common_libs
│   ├── config
│   │   └── config.h
│   ├── id_manager
│   ├── logging
│   │   ├── macros.h
│   │   ├── PriUint64.h
│   │   └── Streaming.h
│   └── protocol
│       ├── fields
│       │   ├── BinaryData.h
│       │   ├── GenericField.h
│       │   ├── NetworkNumber.h
│       │   ├── PacketType.cpp
│       │   ├── PacketType.h
│       │   ├── UnitID.cpp
│       │   └── UnitID.h
│       └── packets
│           ├── AddressablePacket.cpp
│           ├── AddressablePacket.h
│           ├── Command.h
│           ├── GenericPacket.cpp
│           ├── GenericPacket.h
│           ├── RangeRequest.h
│           └── StatusReport.h

This is the platformio.ini for the first project:

[common]
lib_extra_dirs = ${PROJECT_DIR}/../../common_libs

[common_embedded]
extends = common
platform = atmelsam
board = adafruit_feather_m0
framework = arduino
lib_deps = mikem/RadioHead@^1.113, adafruit/Adafruit INA219 @ ^1.1.1, adafruit/Adafruit SHTC3 Library @ ^1.0.0

[env:debug]
extends = common_embedded
build_flags = -D DEBUG_ON

[env:release]
extends = common_embedded

[env:test_native]
extends = common
platform = native
test_ignore = test_embedded
build_flags = -g -O0

And finally the error log when trying to run test_native > Advanced > Test in the sidebar:

Verbose mode can be enabled via `-v, --verbose` option
Collected 1 items

Processing * in test_native environment
--------------------------------------------------------------------------------------------------------------------------------------------------------------
Building...
/home/joku/SHK/avarange/common_libs/protocol/fields/PacketType.cpp:2:10: fatal error: macros.h: No such file or directory

****************************************************************
* Looking for macros.h dependency? Check our library registry!
*
* CLI  > platformio lib search "header:macros.h"
* Web  > https://registry.platformio.org/search?q=header:macros.h
*
****************************************************************

    2 | #include <macros.h>
      |          ^~~~~~~~~~
/home/joku/SHK/avarange/common_libs/protocol/fields/UnitID.cpp:2:10: fatal error: macros.h: No such file or directory

****************************************************************
* Looking for macros.h dependency? Check our library registry!
*
* CLI  > platformio lib search "header:macros.h"
* Web  > https://registry.platformio.org/search?q=header:macros.h
*
****************************************************************

    2 | #include <macros.h>
      |          ^~~~~~~~~~
compilation terminated.
compilation terminated.
*** [.pio/build/test_native/lib18c/protocol/fields/UnitID.o] Error 1
*** [.pio/build/test_native/lib18c/protocol/fields/PacketType.o] Error 1
In file included from test/test_protocol.cpp:1:
/home/joku/SHK/avarange/common_libs/protocol/fields/BinaryData.h:6:10: fatal error: macros.h: No such file or directory

****************************************************************
* Looking for macros.h dependency? Check our library registry!
*
* CLI  > platformio lib search "header:macros.h"
* Web  > https://registry.platformio.org/search?q=header:macros.h
*
****************************************************************

    6 | #include <macros.h>
      |          ^~~~~~~~~~
compilation terminated.
*** [.pio/build/test_native/test/test_protocol.o] Error 1
================================================================= [FAILED] Took 4.89 seconds =================================================================

Test    Environment    Status    Duration
------  -------------  --------  ------------
*       test_native    FAILED    00:00:04.889
=========================================================== 1 failed, 0 succeeded in 00:00:04.889 ===========================================================
The terminal process "platformio 'test', '--environment', 'test_native'" terminated with exit code: 1.

Terminal will be reused by tasks, press any key to close it.

I really don’t know what is wrong, since both debug and release targets build and there are no changes to options for the test_native target that mess with the library dependency finder. Does anyone have experience with this and have an idea what I am doing wrong? Also, ideally, I want to have the unit tests together with the library code in a single directory. Is there any way to do this?

Thank You!

maxgerhardt · April 15, 2022, 3:34pm

Please add lib_ldf_mode = chain+ and on the CLI, do pio test -e test_native --without-upload --without-testing -v and post the output.

_joku · April 15, 2022, 6:38pm

Collected 1 items

Processing * in test_native environment
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Building...
LDF: Library Dependency Finder -> https://bit.ly/configure-pio-ldf
LDF Modes: Finder ~ chain+, Compatibility ~ soft
Framework incompatible library /home/joku/SHK/avarange/common_libs/logging
More details about "Library Compatibility Mode": https://docs.platformio.org/page/librarymanager/ldf.html#ldf-compat-mode
Found 3 compatible libraries
Scanning dependencies...
Dependency Graph
|-- <protocol> (/home/joku/SHK/avarange/common_libs/protocol)
|   |-- <config> (/home/joku/SHK/avarange/common_libs/config)
|-- <config> (/home/joku/SHK/avarange/common_libs/config)
Building in release mode
g++ -o .pio/build/test_native/test/test_protocol.o -c -g -O0 -DPLATFORMIO=50205 -DUNIT_TEST -DUNITY_INCLUDE_CONFIG_H -Iinclude -Isrc -I/home/joku/SHK/avarange/common_libs/protocol -I/home/joku/SHK/avarange/common_libs/config -I.pio/build/test_native/UnityTestLib -I/home/joku/.platformio/packages/tool-unity test/test_protocol.cpp
gcc -o .pio/build/test_native/test/tmp_pio_test_transport.o -c -g -O0 -DPLATFORMIO=50205 -DUNIT_TEST -DUNITY_INCLUDE_CONFIG_H -Iinclude -Isrc -I/home/joku/SHK/avarange/common_libs/protocol -I/home/joku/SHK/avarange/common_libs/config -I.pio/build/test_native/UnityTestLib -I/home/joku/.platformio/packages/tool-unity test/tmp_pio_test_transport.c
g++ -o .pio/build/test_native/lib18c/protocol/fields/PacketType.o -c -g -O0 -DPLATFORMIO=50205 -DUNIT_TEST -DUNITY_INCLUDE_CONFIG_H -I/home/joku/SHK/avarange/common_libs/protocol -I/home/joku/SHK/avarange/common_libs/config -I.pio/build/test_native/UnityTestLib -I/home/joku/.platformio/packages/tool-unity /home/joku/SHK/avarange/common_libs/protocol/fields/PacketType.cpp
In file included from test/test_protocol.cpp:1:
/home/joku/SHK/avarange/common_libs/protocol/fields/BinaryData.h:6:10: fatal error: macros.h: No such file or directory

****************************************************************
* Looking for macros.h dependency? Check our library registry!
*
* CLI  > platformio lib search "header:macros.h"
* Web  > https://registry.platformio.org/search?q=header:macros.h
*
****************************************************************

    6 | #include <macros.h>
g++ -o .pio/build/test_native/lib18c/protocol/fields/UnitID.o -c -g -O0 -DPLATFORMIO=50205 -DUNIT_TEST -DUNITY_INCLUDE_CONFIG_H -I/home/joku/SHK/avarange/common_libs/protocol -I/home/joku/SHK/avarange/common_libs/config -I.pio/build/test_native/UnityTestLib -I/home/joku/.platformio/packages/tool-unity /home/joku/SHK/avarange/common_libs/protocol/fields/UnitID.cpp
      |          ^~~~~~~~~~
compilation terminated.
*** [.pio/build/test_native/test/test_protocol.o] Error 1
/home/joku/SHK/avarange/common_libs/protocol/fields/UnitID.cpp:2:10: fatal error: macros.h: No such file or directory

****************************************************************
* Looking for macros.h dependency? Check our library registry!
*
* CLI  > platformio lib search "header:macros.h"
* Web  > https://registry.platformio.org/search?q=header:macros.h
*
****************************************************************

    2 | #include <macros.h>
      |          ^~~~~~~~~~
compilation terminated.
/home/joku/SHK/avarange/common_libs/protocol/fields/PacketType.cpp:2:10: fatal error: macros.h: No such file or directory

****************************************************************
* Looking for macros.h dependency? Check our library registry!
*
* CLI  > platformio lib search "header:macros.h"
* Web  > https://registry.platformio.org/search?q=header:macros.h
*
****************************************************************

    2 | #include <macros.h>
      |          ^~~~~~~~~~
compilation terminated.
*** [.pio/build/test_native/lib18c/protocol/fields/UnitID.o] Error 1
*** [.pio/build/test_native/lib18c/protocol/fields/PacketType.o] Error 1
====================================================================== [FAILED] Took 0.70 seconds ======================================================================

maxgerhardt · April 15, 2022, 6:39pm

Hm a file within protocol seems to need macros.h which is in the logging library, but the dependency is not recognized.

Please add

lib_deps =
   protocol
   config
   logging

to the platformio.ini and retry the previous command.

_joku · April 15, 2022, 6:44pm

seems to be the same error

Collected 1 items

Processing * in test_native environment
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Building...
LDF: Library Dependency Finder -> https://bit.ly/configure-pio-ldf
LDF Modes: Finder ~ chain+, Compatibility ~ soft
Framework incompatible library /home/joku/SHK/avarange/common_libs/logging
More details about "Library Compatibility Mode": https://docs.platformio.org/page/librarymanager/ldf.html#ldf-compat-mode
Library Manager: Installing logging
Warning! Could not find the package with 'logging' requirements for your system 'linux_x86_64'
Found 3 compatible libraries
Scanning dependencies...
Dependency Graph
|-- <protocol> (/home/joku/SHK/avarange/common_libs/protocol)
|   |-- <config> (/home/joku/SHK/avarange/common_libs/config)
|-- <config> (/home/joku/SHK/avarange/common_libs/config)
Building in release mode
g++ -o .pio/build/test_native/test/test_protocol.o -c -g -O0 -DPLATFORMIO=50205 -DUNIT_TEST -DUNITY_INCLUDE_CONFIG_H -Iinclude -Isrc -I/home/joku/SHK/avarange/common_libs/protocol -I/home/joku/SHK/avarange/common_libs/config -I.pio/build/test_native/UnityTestLib -I/home/joku/.platformio/packages/tool-unity test/test_protocol.cpp
gcc -o .pio/build/test_native/test/tmp_pio_test_transport.o -c -g -O0 -DPLATFORMIO=50205 -DUNIT_TEST -DUNITY_INCLUDE_CONFIG_H -Iinclude -Isrc -I/home/joku/SHK/avarange/common_libs/protocol -I/home/joku/SHK/avarange/common_libs/config -I.pio/build/test_native/UnityTestLib -I/home/joku/.platformio/packages/tool-unity test/tmp_pio_test_transport.c
g++ -o .pio/build/test_native/lib18c/protocol/fields/PacketType.o -c -g -O0 -DPLATFORMIO=50205 -DUNIT_TEST -DUNITY_INCLUDE_CONFIG_H -I/home/joku/SHK/avarange/common_libs/protocol -I/home/joku/SHK/avarange/common_libs/config -I.pio/build/test_native/UnityTestLib -I/home/joku/.platformio/packages/tool-unity /home/joku/SHK/avarange/common_libs/protocol/fields/PacketType.cpp
g++ -o .pio/build/test_native/lib18c/protocol/fields/UnitID.o -c -g -O0 -DPLATFORMIO=50205 -DUNIT_TEST -DUNITY_INCLUDE_CONFIG_H -I/home/joku/SHK/avarange/common_libs/protocol -I/home/joku/SHK/avarange/common_libs/config -I.pio/build/test_native/UnityTestLib -I/home/joku/.platformio/packages/tool-unity /home/joku/SHK/avarange/common_libs/protocol/fields/UnitID.cpp
/home/joku/SHK/avarange/common_libs/protocol/fields/PacketType.cpp:2:10: fatal error: macros.h: No such file or directory

****************************************************************
* Looking for macros.h dependency? Check our library registry!
*
* CLI  > platformio lib search "header:macros.h"
* Web  > https://registry.platformio.org/search?q=header:macros.h
*
****************************************************************

    2 | #include <macros.h>
      |          ^~~~~~~~~~
compilation terminated.
*** [.pio/build/test_native/lib18c/protocol/fields/PacketType.o] Error 1
g++ -o .pio/build/test_native/lib18c/protocol/packets/AddressablePacket.o -c -g -O0 -DPLATFORMIO=50205 -DUNIT_TEST -DUNITY_INCLUDE_CONFIG_H -I/home/joku/SHK/avarange/common_libs/protocol -I/home/joku/SHK/avarange/common_libs/config -I.pio/build/test_native/UnityTestLib -I/home/joku/.platformio/packages/tool-unity /home/joku/SHK/avarange/common_libs/protocol/packets/AddressablePacket.cpp
In file included from test/test_protocol.cpp:1:
/home/joku/SHK/avarange/common_libs/protocol/fields/BinaryData.h:6:10: fatal error: macros.h: No such file or directory

****************************************************************
* Looking for macros.h dependency? Check our library registry!
*
* CLI  > platformio lib search "header:macros.h"
* Web  > https://registry.platformio.org/search?q=header:macros.h
*
****************************************************************

    6 | #include <macros.h>
      |          ^~~~~~~~~~
compilation terminated.
*** [.pio/build/test_native/test/test_protocol.o] Error 1
/home/joku/SHK/avarange/common_libs/protocol/fields/UnitID.cpp:2:10: fatal error: macros.h: No such file or directory

****************************************************************
* Looking for macros.h dependency? Check our library registry!
*
* CLI  > platformio lib search "header:macros.h"
* Web  > https://registry.platformio.org/search?q=header:macros.h
*
****************************************************************

    2 | #include <macros.h>
      |          ^~~~~~~~~~
compilation terminated.
*** [.pio/build/test_native/lib18c/protocol/fields/UnitID.o] Error 1
In file included from /home/joku/SHK/avarange/common_libs/protocol/packets/GenericPacket.h:4,
                 from /home/joku/SHK/avarange/common_libs/protocol/packets/AddressablePacket.h:3,
                 from /home/joku/SHK/avarange/common_libs/protocol/packets/AddressablePacket.cpp:1:
/home/joku/SHK/avarange/common_libs/protocol/fields/NetworkNumber.h:5:10: fatal error: macros.h: No such file or directory

****************************************************************
* Looking for macros.h dependency? Check our library registry!
*
* CLI  > platformio lib search "header:macros.h"
* Web  > https://registry.platformio.org/search?q=header:macros.h
*
****************************************************************

    5 | #include <macros.h>
      |          ^~~~~~~~~~
compilation terminated.
*** [.pio/build/test_native/lib18c/protocol/packets/AddressablePacket.o] Error 1
====================================================================== [FAILED] Took 1.46 seconds ======================================================================

it seems to find it but it is “framework incompatible”. what does that mean?

maxgerhardt · April 15, 2022, 6:46pm

Can you place an explicit #include <macros.h> in the test .cpp file?

_joku · April 15, 2022, 6:49pm

same error unfortunately

maxgerhardt · April 15, 2022, 6:51pm

Can you show how protocol/fields/PacketType.cpp includes macros.h? Are there any conditional inclusion macros in there like in Library dependency not recognized in conditional include · Issue #4152 · platformio/platformio-core · GitHub?

_joku · April 15, 2022, 6:58pm

PacketType.cpp just includes it without any #ifdefs etc
but macros.h itself has some. Could that be the problem?
here is macros.h for reference:

#pragma once

#if defined(UNIT_TEST)
#include <iostream>
#define PRINT_OUT std::cout
#define ENDL std::endl
#else
// clang-format off
#include "Streaming.h"
#include "PriUint64.h"
// clang-format on

#define PRINT_OUT Serial
#define ENDL "\n"
#endif

#ifdef DEBUG_ON
#define dbgln(...)                                                                                 \
  do {                                                                                             \
    PRINT_OUT << __VA_ARGS__ << ENDL;                                                              \
  } while (0)

#define dbg_only(code)                                                                             \
  do {                                                                                             \
    code                                                                                           \
  } while (0)
#else
#define dbgln(...)
#define dbg_only(code)
#endif

#if defined(UNIT_TEST)

#define VERIFY(expression)                                                                         \
  do {                                                                                             \
    if (!(expression)) {                                                                           \
      dbgln("ASSERTION FAILED: " #expression);                                                     \
      dbgln("in " << __FILE__ << ":" << __LINE__);                                                 \
      throw;                                                                                       \
    }                                                                                              \
  } while (0)

#else

#define VERIFY(expression)                                                                         \
  do {                                                                                             \
    if (!(expression)) {                                                                           \
      dbgln("ASSERTION FAILED: " #expression);                                                     \
      dbgln("in " << __FILE__ << ":" << __LINE__);                                                 \
                                                                                                   \
      for (;;) {                                                                                   \
        digitalWrite(LED_BUILTIN, LOW);                                                            \
        delayMicroseconds(500000);                                                                 \
        digitalWrite(LED_BUILTIN, HIGH);                                                           \
        delayMicroseconds(500000);                                                                 \
      }                                                                                            \
    }                                                                                              \
  } while (0)

#endif

#define VERIFY_NOT_REACHED() VERIFY(false)

maxgerhardt · April 15, 2022, 7:07pm

No, the LDF should have recognized the PacketType.cpp → macros dependency…

If you replace the #pragma once with traditional include guards in some critical header files (the path from the test.cpp to packettype.h and in macros.h) does that help?

_joku · April 15, 2022, 7:27pm

no didn’t change anything

maxgerhardt · April 15, 2022, 7:30pm

Open a CLI and execute pio upgrade --dev and retry. If it’s not solved after that, please open an issue at Issues · platformio/platformio-core · GitHub with the reference project.

_joku · April 15, 2022, 7:37pm

yeah that didnt fix it.
I will try to make an issue in the upcoming days.
but tysm for your help
have a nice day!