Build fails when using symlink to link libraries of a depended on library in main project

Hi!

I have a project where we use several own developed private libraries. All these libraries are stored in an Azure repository which unfortunately means that I am unable to link them via URLs in platformio.ini. We tried to find a solution where we have only one copy of the source code for each library in our main project and the solution we came up with is to make the depended on libraries local libraries of one of the libraries and then make that one library a local library of the main project. In source control (Azure/git) we have one main repository which then links the depended on repositories as submodules. The directory structure looks something like this:

MainProject
 |--lib
      |--TheOneProject
             | -- lib
                   |-- DependencyLib1
                   |-- DependencyLib2

TheOneProject and MainProject are both directly dependent on DependencyLib1 and DependencyLib2.

Now if I use in platformio.ini

lib_ldf_mode = chain+
lib_extra_dirs = lib/TheOneProject/lib

then all works fine, but as we know lib_extra_dirs is about to disappear. So I tried to use lib_deps instead. The following works:

lib_deps = 
    file://lib/TheOneProject/lib/DependencyLib1
    file://lib/TheOneProject/lib/DependencyLib2

This however causes a problem with changes in these libraries not reflected in the MainProject. If I use

lib_deps = 
    symlink://lib/TheOneProject/lib/DependencyLib1
    symlink://lib/TheOneProject/lib/DependencyLib2

Then the dependencies are found, the Library Manager crates the symlink files in the
.pio/libdeps/<env-name> folder, but when it comes to compiling the code, header files in these libraries are not found and the build fails.

Interestingly, if I copy these libraries somewhere else, outside of lib/TheOneProject/lib folder (and modify the symlink path accordingly), then the symlinks and compilation works, but this recreates the problem of duplication that we wanted to avoid in the first place.

So my questions are:

  1. How can I make symlinks work for libraries of libraires?
  2. If (1) is not possible, how can I set up my MainProject so that it “sees” the libraries of the TheOneProject?
  3. If (2) is not possible either, how can I link to repositories in Azure?
  4. Is there any other solution to the problem of wanting to use just one local copy of a library shared between multiple dependent projects without being able to link an external source?

Any help is much appreciated!

Update 1
I tried moving out TheOneProject from the designated private library lib folder.

lib_deps = 
    symlink://myprivatelib/TheOneProject
    symlink://myprivatelib/TheOneProject/lib/DependencyLib1
    symlink://myprivatelib/TheOneProject/lib/DependencyLib2

Same result - Library Manager installs the symlinks but the build fails. It looks like being a private library of another library makes the source files of the libraries invisible to the main project. If however DependencyLib1 and DependencyLib2 is moved out of the lib folder from under TheOneProject then compilation of MainProject succeeds.

So it appeared that if I treat these common dependencies of MainProject and TheOneProject as external library dependencies in both projects I could do with one copy only of the source code of the dependencies. So I tried this:

lib_deps = 
    symlink://myprivatelib/TheOneProject
    symlink://myprivatelib/TheOneProject/myprivatelib//DependencyLib1
    symlink://myprivatelib/TheOneProject/myprivatelib//DependencyLib2

But this one also fails. As it appears, as soon as the symlink path to one library is included in the path to another library the build will fail.

So right now I am stuck and cannot really see a way now on how to create coherent repositories for MainProject and TheOneProject in a way that both will be self contained (i.e. containing all the source for successful compilation) without having a duplicate of the dependency libraries when MainProject includes TheOneProject.

Could you move TheOneProject outside lib folder?

As you can see (Update 1 above) I tried that. It appears that as soon as a depended on library is in a subfolder of another depended on library that path is not used and the build fails. It does not matter if it is in lib or another library, the mechanism is the same.

Now I tried to add the paths also as build flags, i.e.:

build_flags =
    -I myprivatelib/TheOneProject/myprivatelib/DependencyLib1
    -I myprivatelib/TheOneProject/myprivatelib/DependencyLib2/include

which made the compilation succeed, but with this approach it is linking that fails. The files in DependencyLib2 are not compiled, no object files are created. The difference in structure between DependencyLib1 and DependencyLib2 is that in DependencyLib1 both .h and .cxx files are in the root while in DependencyLib2, include files are in an ìnclude/<namespace name> folder and source files are in a src/<namespace name> folder.

To verify that the library structure is the problem I removed these extra build flags and moved only DependencyLib2 to a completely different path and tried with:

lib_deps = 
    symlink://myprivatelib/TheOneProject
    symlink://myprivatelib/TheOneProject/myprivatelib/DependencyLib1
    symlink://DIFFERENTpath/TheOneProject/myprivatelib/DependencyLib2

And voila, everything works as it should!

Now to see if my theory on shared path I tried:

lib_deps = 
    symlink://myprivatelib/TheOneProject
    symlink://myprivatelib/TheOneProject/myprivatelib//DependencyLib1
    symlink://myprivatelib/DependencyLib2

and to my surprise this also worked. So I moved it one level down the shared path:

lib_deps = 
    symlink://myprivatelib/TheOneProject
    symlink://myprivatelib/TheOneProject/myprivatelib//DependencyLib1
    symlink://myprivatelib/TheOneProject/DependencyLib2

And this now failed at compilation. As a negative test I tried:

lib_deps = 
    symlink://myprivatelib/TheOneProject
    symlink://myprivatelib/TheOneProject/DependencyLib1
    symlink://myprivatelib/DependencyLib2

and it worked. For last I tried:

lib_deps = 
    symlink://myprivatelib/TheOneProject
    symlink://myprivatelib/DependencyLib1
    symlink://myprivatelib/TheOneProject/DependencyLib2

which failed again, thankfully showing at least a consistent behavior. So as it appears, if a library contains another library where the source files are not directly in the root, then the path of this contained library is somehow ignored.

TheOneProject is also organized the same way as DependencyLib2, so I tested if using DependencyLib1 as the “One” project makes any difference:

lib_deps = 
    symlink://myprivatelib/DependencyLib1
    symlink://myprivatelib/DependencyLib1/TheOneProject
    symlink://myprivatelib/DependencyLib1/DependencyLib2

but it does not make any difference. Header files in DependencyLib2 are not found and compilation fails.

Based on the above findings it feels like a bug (in LDF?) to me.

Could you provide a simple PlatformIO project to reproduce this issue? You can host it on Dropbox or somewhere else. Thanks!

Here’s a demo project: GitHub - margaretselzer/LibraryLinkDemo: A simple demo project to deminstrate a build/linking problem on the PlatformIO platform

When built successfully, it produces a bunch of irrelevant warnings which I did not care to fix, but when it can be built it works. When it fails to build then it behaves slightly differently than the project I am really working with as in the demo project even the library which does not have subdirectories fails to build. So ultimately, as it appears, the reason for compiling/linking not working is not the subdirectories in a library, it is something else.

The project as is will fail to build. I marked in platformio.ini how it can be made to build:

  1. use the deprecated lib_extra_dirs or
  2. put the sub-libraries on a different path. You can try these options by commenting out the relevant sections in platformio.ini.

While thinking on a solution I came up with a question. Using my above example, if I put in a new library in the lib folder next to TheOneProject - let’s called it SecondProject - which is also dependent on the same libraries as TheOneProject, then SecondProject will build within MainProject even if those dependencies are not expressly specified within SecondProject. So can it be a solution that all the dependent libraries are simply put in the lib folder under MainProject and as long as they are compiled as part of MainProject they will all build without a problem?

Then the second question is whether it is possible to specify these “parallell” dependencies in some file in each of these libraries, e.g. in library.json with a relative path so that each of these libraries will expect that all the dependent libraries they need will be present in the same folder where they themselves are located? Again with my example, TheOneProject, DependencyLib1 and DependencyLib2 would all be located in the same folder and when TheOneProjectis built without the MainProject (this is the case when unit tests are run for example), the project will know where to expect to find those to dependent libraries?

If PlatformIO expects everything to be organised in libraries (which of course is not a bad thing) it also needs to offer a good way to manage interdependencies while allowing that only one copy of any given library exists within the project - even with private libraries. While using different versions of a library within the same project might have some legitimate use cases (e.g. when different libraries used in the main project use different versions of a shared library), it is still something that is extremely error prone and should be the exception rather than the rule.

Update/New question
I noticed that the size of the binary (usage of flash) will be different depending on if the libraries are in the lib folder or somewhere externally and then symlinked via lib_deps. The symlink way consumes more flash. In my case it is 768 bytes more. It is not much of a problem on a Due, but is this justifiable?

I settled on a project structure where all libraries that depend on each other are in a separate repository. So the project structure looks like this:

MainProject
 |--TheOneProject
           |-- DependencyLib1
           |-- DependencyLib2

In the MainProject I use symlink in the platformio.ini file to all the DenedencyLibX projects and the same applies to the DependencyLibX projects themselves as well.

NEW PROBLEM: Infinite triggering of project configuration
However, I have new issue. One of the libs I am dependent on is ArduinoJson. I cannot link the GitHub repo so there is a local repo which is symlink:ed the same way as the rest of the libs. This structure looks like this:

MainProject (Depends on all three libs under TheOneProject)
 |--TheOneProject
           |-- ArduinoJson
           |-- DependencyLib1  (depends on ArduinoJson and DependencyLib2)
           |-- DependencyLib2  (no external dependency)

In this structure if ArduinoJson is symlink:ed, it will trigger an infinite loop of project configuration updates. Also, when I build MainProject with every build the ArduinoJson lib is installed - twice:

Library Manager: Installing symlink://../ArduinoJson
Library Manager: ArduinoJson@6.19.4 has been installed!
Verbose mode can be enabled via `-v, --verbose` option
LDF: Library Dependency Finder -> https://bit.ly/configure-pio-ldf
LDF Modes: Finder ~ chain, Compatibility ~ soft
Library Manager: Installing symlink://../ArduinoJson
Library Manager: ArduinoJson@6.19.4 has been installed!
Found 4 compatible libraries
Scanning dependencies...
Dependency Graph
|-- DependencyLib1 @ 2.0.0
|-- DependencyLib2 @ 1.0.6
Building in debug mode

Note, that ArduinoJson does not even appear in the dependency tree. However, if I change the symlink to file, then I get rid of the infinite project configuration loop and ArduinoJson suddenly appears in the dependency tree as well.

LDF: Library Dependency Finder -> https://bit.ly/configure-pio-ldf
LDF Modes: Finder ~ chain, Compatibility ~ soft
Found 5 compatible libraries
Scanning dependencies...
Dependency Graph
|-- ArduinoJson @ 6.19.4
|-- DependencyLib1  @ 2.0.0
|-- DependencyLib2  @ 1.0.6
Building in debug mode

I did not post this as a new topic because it might be linked to the first problem I reported.