Linking issues with a custom wrapper library and submodules in nested folders

dmelendrez · March 17, 2025, 8:37am

Dear all

I am currently working on a project for a controller board (ESP32-based) where all the peripherals are abstracted using “Hardware Managers” which then are merged and controlled by a “Wrapper” library. This way, I can call specific peripherals and execute intricate actions using a single line (in theory). Please note that I am trying to hide the actual name of my project by replacing sensitive names using xxx while trying to keep a logical naming convention to convey my case.

I have successfully completed the design of this wrapper and peripherals managers on a VSCode+pio project following the recommended folder organization. This was done on a “barebones” approach by having all dependencies and libraries inside the lib/ folder, as shown in this project structure:

where I implemented a bunch of envs to have manual control over the compilation of specific files, including tests and examples. The working `platform.ini" file looks something like this (partially):

[env_defaults]
framework = arduino
platform = espressif32
board = esp32dev

build_type = release

monitor_speed = 115200
upload_speed = 921600
lib_deps =
    Wire
    adafruit/Adafruit BusIO@^1.17.0

board_upload.flash_size = 16MB
board_build.f_cpu = 240000000L

build_flags = 
	-I/lib/xxx_Controller (see note)
	-I/lib/Adafruit_HUSB238
	-I/lib/EEPROM_UID
	-I/lib/I2C_EEPROM
	-I/lib/MCP23017_RT
	-I/lib/RTClib
	-I/lib/SD
	-I/lib/modbus-esp8266/src
	-I/lib/TCA9548
	-I/lib/PCA9632
    -DBOARD_HAS_PSRAM
	-O3

lib_ldf_mode = deep+

[env:eeeprom_uid_test]
extends = env_defaults
platform = espressif32
board = esp32dev
build_src_filter = +<EEPROMUIDTest/>

Note: The first build_flag calls “lib/xxx_Controller” whilst the filename is “xxx_Controller_Wrapper.h”, however the class is: “class xxx_Controller”. Apologies for this small inconsistency, I did not notice this until now since everything compiles just fine.

Now, here is where the linking errors occur and please bear with me.

For the sake of portability (internal in my organization), I decided to “pack” my solution into a GitHub repository where the managers headers and source files are places inside include and src folders, respectively and the external libraries used by the managers are contained as submodules inside a nested lib/ folder so that everything needed for development is held within the same wrapper repo with the following structure:

Wrapper_Controller
|
|–examples
| |–Example_Manager1
| |–Example_Manager2
| |–…etc
|–include {contains all .h files}
| |–Wrapper_Controller.h
| |–Manager1.h
| |–Manager2.h
| |–…etc
|–src {contains all .cpp files}
| |–Wrapper_Controller.cpp
| |–Manager1.cpp
| |–Manager2.cpp
| |–…etc
|–lib {contains all git submodules}
| |–Submodule1
| |–Submodule2
| |–Submodule3
| |–…etc

I cloned this repository inside the `lib’ folder from a new project where I plan to continue the development of the controller firmware (where I will use the wrapper and other libraries) as well as work on corrections of the same wrapper code.

My new project structure using the GitHub repo looks like this:

One of my several attempts to configure an .ini file is this:

[env:esp32dev]
platform = espressif32
board = esp32dev
framework = arduino

board_upload.flash_size = 16MB
board_build.f_cpu = 240000000L

build_type = release

monitor_speed = 115200
upload_speed = 921600

lib_deps = adafruit/Adafruit BusIO@^1.17.0

lib_ldf_mode = deep+

; If using submodules, ensure the submodule is pulled correctly
lib_extra_dirs = lib/xxx_Wrapper_Controller/lib

build_flags = 
    -I lib/xxx_Wrapper_Controller/include
    -I lib/xxx_Wrapper_Controller/lib/Adafruit_HUSB238
    -I lib/xxx_Wrapper_Controller/lib/EEPROM_UID
    -I lib/xxx_Wrapper_Controller/lib/I2C_EEPROM
    -I lib/xxx_Wrapper_Controller/lib/MCP23017_RT
    -I lib/xxx_Wrapper_Controller/lib/RTClib/src
    -I lib/xxx_Wrapper_Controller/lib/SD
    -I lib/xxx_Wrapper_Controller/lib/SdFat-Adafruit-Fork
    -I lib/xxx_Wrapper_Controller/lib/modbus-esp8266/src
    -I lib/xxx_Wrapper_Controller/lib/TCA9548
    -I lib/xxx_Wrapper_Controller/lib/PCA9632
 
    -DBOARD_HAS_PSRAM
    -O3

My project “parses” correctly. I can use the context menus and navigate to all source and header files using the options “Go to definitions” and “Go to declaration” There are no squiggly lines anywhere in the code, however, I get tons of linking issues, something along the lines of:

“Undefined reference to: Manager1::method1…”
“Undefined reference to: Manager2::methodN”
…
“Undefined reference to: xxx_Wrapper::Constructor”
"“Undefined reference to: xxx_Wrapper::~Destructor”

which makes me think that I am missing something to point pio to the source files.

I have tried moving all .h files together with the .cpp files inside the src/ folder, however, the problems persist.

I have also tried including the src/ folder in the build_flags as:

build_flags = -I lib/xxx_Wrapper_Controller/src

Please note that all submodules are present and have been initialised.

I would appreciate any pointers (no pun intended), suggestions and comments.
Perhaps I need to rethink my folder structure, right?

Cheers

EDIT: My solution does include a library.json file inside the cloned wrapper library, however, I am not sure how impactful it is for a successful compilation.

UPDATE 1: I have tried EVERY solution I have come across on the forum to no avail. I have moved the .cpp files along with the .h files inside the src/ folder from the rapper library. After doing this, I faced EXACTLY the same issue reported here regarding Adafruit_I2CDevice.h missing in one of my managers. I added such header to the manager and the linking issues returned. It is important to emphasize that this issue did NOT occur in my previous development with all the files exposed inside the lib/ folder.

I can confirm that the verbose compilation output shows that NONE of the .cpp files from my solution gets compiled since I cannot see any xxx.cpp.o file from my custom library and managers.

UPDATE 2: My issue and situation seem to be closely related to the one described in this thread where the user worked on a similarly nested project. There, the solution provided by @maxgerhardt (if I understood correctly) was to bring all source files to the root of the lib folder. However, in my case, this is not possible since my library is a git repo and the “nested libraries” are part or it (as submodules). I will try putting all files at the root of the wrapper library and report back shortly.

UPDATE 3 (Succes!(?)): Without moving any file and leaving my wrapper library “as is” I managed to get all .cpp files to compile and linking is now working!

This is my updated .ini file:

[env:esp32dev]
platform = espressif32
board = esp32dev
framework = arduino

board_upload.flash_size = 16MB
board_build.f_cpu = 240000000L

build_type = release

monitor_speed = 115200
upload_speed = 921600

lib_extra_dirs = lib/xxx_Wrapper_Controller/lib

build_src_filter =
  +<**/*.cpp>
  +<lib/xxx_Wrapper_Controller/lib/*.cpp>
   
build_flags = 
    -I lib/xxx_Wrapper_Controller/include
    -I lib/xxx_Wrapper_Controller/lib/Adafruit_HUSB238
    -I lib/xxx_Wrapper_Controller/lib/EEPROM_UID
    -I lib/xxx_Wrapper_Controller/lib/I2C_EEPROM
    -I lib/xxx_Wrapper_Controller/lib/MCP23017_RT
    -I lib/xxx_Wrapper_Controller/lib/RTClib/src
    -I lib/xxx_Wrapper_Controller/lib/SD
    -I lib/xxx_Wrapper_Controller/lib/SdFat-Adafruit-Fork/src
    -I lib/xxx_Wrapper_Controller/lib/modbus-esp8266/src
    -I lib/xxx_Wrapper_Controller/lib/TCA9548
    -I lib/xxx_Wrapper_Controller/lib/PCA9632
 
    -DBOARD_HAS_PSRAM
    -O3

lib_ldf_mode = deep+

lib_deps = 
    Wire
    SPI
    adafruit/Adafruit BusIO@^1.17.0

Now I wonder if these settings make any sense and if the compilation is accurate (yeah I am a bit paranoid now. I have spent too much time on this ) I strongly believe that the lines: lib_ldf_mode and build_src_filter were the trick that saved my project,

My final question is: How do I convert these latest additions into configuration settings inside the library.json so that any project that uses my wrapper compiles without having to add them on each ini file?

sivar2311 · March 18, 2025, 5:33am

You should start by reading this: Creating Library — PlatformIO latest documentation

We recommend to use src folder for your C/C++ source files and include folder for your headers. You can also have nested sub-folders in src or include .
├── examples
│   └── echo
├── include
│   └── HelloWorld.h
├── library.json
└── src
   └── HelloWorld.cpp

Then simplify your file structure until it corresponds to the example above.

External libraries are included via dependencies. So delete them from your wrapper library to reduce the complexity and size.

The SD library is part of Espressif32 Arduino Framework. This should not be listed as a lib_dep.

You’re using some kind of EEPROM library. EEPROM is only emulated on an ESP32. Instead you should use the Preferences library which uses the Non Volatile Storage (NVS).

Here is how I start creating libraries:

First create a Test project as a framework for using and testing the library.
Inside that project I create a folder “my_library” where I create a subolder with the final name of my library (“MyLib” in this example).

Using lib_deps = symlink://my_library/MyLib allows to include and test the library:

Also external dependencies are downloaded automatically this way

dmelendrez · March 18, 2025, 6:06am

Dear @sivar2311

Thanks for your response.

I agree that my folder structure is not optimal. However, this is a custom implementation of hardware abstraction layers that requires every aspect of the wrapper library to be self-contained. You are absolutely right that I should probably detach the dependencies and call them in the .ini file to be “pulled” externally.

I will definitely evaluate such an approach. However, for the time being, we need to stick to something similar to this to continue developing adding corrections and new features to all libraries including the wrappers and managers. This is still a work in progress.

Now, some clarifications regarding my choice of libraries.

I agree that the SD library is part of the Arduino framework, however, this one is a custom fork that has features that cannot be merged into the main library. In particular, we need to pass a pointer to an alternative serial port to the library to print the contents of the SD card on any port. I indeed submitted such a pull request, however, it did not pass since the tests failed only for the Arduino Leonardo Board (PR 148).

Similarly, I am not sure what you meant to say regarding the EEPROM library. My system has two types of EEPROMs: one UID EEPROM based on a custom library and a second library for a typical 1Mbit EEPROM developed by Rob Tillaart

EDIT: I forgot to mention that, based on my last update, I managed to get the code to fully compile without any issues, however, I wonder how “maintainable” my solution is. I will keep using this successful result and report back in case I face other issues. Finally, thanks for sharing your workflow. I have indeed experimented with symlinks, however, I remember running into other issues when sharing the library. My colleagues couldn’t get cloned repos works for some strange reasons.

Cheers

sivar2311 · March 18, 2025, 6:13am

No in the platformio.ini but in the dependencies section of your library.json

Ok

The ESP32 doesn’t have an EEPROM. It is just emulated.
On an ESP32 you have NVS and the Preferences library wich offers a key value storage.

These are PlatformIO Symlinks, not linux symlinks - just to be clear.

The symlink was meant to be used locally to test the library. If you host your library on github the workflow is different.
Create a library repository
Create a test project repository
In the testproject you can simply include the library by

lib_deps = 
  http://github.com/username/repository

This will also respect the dependencies in the library.json.

dmelendrez · March 18, 2025, 6:18am

@sivar2311

Very insightful. Thank you once again for your kind help.

There’s a lot to be learnt from the community.

P.S. Apologies for the typos and broken English. I am thinking faster than I can type.