Confused about what lib actually does for re-usability

[wanderingsquirrel]

Hello,

I am gonna apologise in advance for wildly missing the point and being extremely stupid….but I am confused as to the purpose of the lib folder. I think I am missing something obvious and perhaps overthinking. Yes, I have read the lib/README.md but it seems so simplistic as to be kinda useless. Thinking about the purpose of it confuses matters further. I am just gonna present my understanding and hopefully someone can tell me why I am wrong and illuminate me. Full disclosure I am not super comfortable with how Cpp manages distribution of code and dependency management and find the build systems a bit mysterious beyond the very basics

My question:

What does separating out my code there actually give me and why does this exist because it would seem it is not for the obvious reasons? …I would imagine that the answer should be enforced boundaries in my code and code reuse. This is the reason we separate code out afterall in any language, but this lib seems to do none of these/do none of these well…..

1. Doesn’t Seem to Offer any Reuse advantages Over Subfolders in Src:

Normally when you separate code out into a library it is supposed to give you re-use. I mean if I were working in a rust repos I could turn a particularly large module into its own crate for reuse and then reference it in another project via something like:

[dependencies]
my_crate = { git = “``https://github.com/username/my_crate.git”``, package = “subcrate_name” }

but if I am understanding correctly, there is no way built in to reuse stuff from the lib folder in another project; you have to do it manually in some form. We are reduced to copy and paste/git submodules which is kinda gross…..If we are doing that, then we might as well just separate out our code under src/ into folders for logical organisation and then use gitsubmodules and copy paste with this whcih seems equally crude.

So it seems that lib offers no advantages for code reuse at all over subfolders under src. I guess that it does offer the granular ability for libraries to carry their dependencies with them for reuse which is useful, but without a way to distribute them and without a way to enforce no coupling with other libraries or pio managed libraries this doesn’t seem very good…(explained in more detail in next point)

2. lib contained libraries do not have any enforced library boundaries

I thought that perhaps a weakness of having everything in organised subfolders of src is there is no enforced code boundaries (everything can use everything ) leading to everything getting messy and interdependent.

So I thought that perhaps lib was intended so that you had to declare explicitly all dependencies of a library so that - for example - when working in projectA that contains the lib/libA you can’t accidentally make it dependent on lib/libB without explicitly declaring it in the library.json as a dependency and thus break projectB which was using libA (perhaps as a git submodule) without realising it. In other words, it prevents accidental coupling.

This would be a nice use and explanation for lib, but it seems that things will work even if you don’t declare the dependencies in the json, so it isn’t this. So for enforcing boundaries it is no better than subfolders in src when it comes to boundaries between othe rproject local code and/or external dependencies which is obviously disappointing.

3. Perhaps it is nothing to do with code reuse and is just for build system clarity so that code native to the repos is not compiled when not needed?

from the README.md:

The PlatformIO Library Dependency Finder will find automatically dependent

libraries by scanning project source files.

More information about PlatformIO Library Dependency Finder

So the dependency finder can find whcih libraries your code actually depends on (perhaps you have a library that is only used in your code in certain compilation modes)…. but then if you aren’t using anything from that library in a given mode then it will get weeded out by the linker anyway if I am understanding at the point where we stitch the .o files together….so this hardly does anything…?

it also seems that the build process in Scons that pio uses is clever enough not to build every subfolder under src each time when there are only changes to a few files, so it is not as if separating out this code into libraries somehow enables better dependency tracking between files for incremental compilation….

Conclusion:

Probably missing something extremely obvious, but the only thing that I can think of is that it allows you to add dependencies to your project in a granular way and tie dependencies to the code that actually uses it so that when that modular code is distributed it brings with it all of its dependencies. This is great and all! However, without enforcing include paths limited ONLY to explicit dependencies this is kinda half baked….and if you git submodule a lib from inside a project then as far as I know pio isn’t clever enough to download any local dependencies it had in the project it came from…rendering this kinda useless again. So I am guessing it isn’t for this reason. Is it something simpler? Is it build process related?

thanks for any guidance

[normandunbar]

Briefly…

In days of old, libraries could be global, but this caused problems when different platforms used the same library name. Installing into global namespace could/would/did overwrite things.

Having your libraries local to a project means that the project doesn’t have the problem of its libraries being trampled by some other project’s libraries.

You can still install libraries “globally”, there’s a couple of platformio.ini options:

• lib_extra_dirs (See lib_extra_dirs — PlatformIO latest documentation).
• globallib_dir (See globallib_dir — PlatformIO latest documentation).

Unfortunately, both are deprecated and will (probably) be removed from the next major release. However, you can still set up “global” libraries in a shared directory, and reference those from your platformio.ini file.

Assume you have a number of projects using the same library and the same version of it. You would normally specify the library in the lib_deps parameter, or an online path to it etc, and have it included, separately, fo each project.

However, if you wish to share the same library then do this:

1. Set up a shared directory. Call it, for example, shared_libraries.
2. Within shared_libraries, place the code for each shared library in a separate directory—one per library. For example, shared_libraries/LED and shared_libraries/myLib etc.
3. In each project, add this to the lib_deps:

   lib_deps =
      LEDLib=symlink://path/to/shared-libraries/LED
   

4. Compile!

All your projects will use the same shared library. Whenever a library is updated, you will need to be sure that the update doesn’t break any project with uses it.

Now, you may say that the Arduino IDE manages very well with shared libraries, and it does, however, even that is moving to using “profiles” to ensure that if you ever have to compile a sketch, say 5 years down the line, then your profile will ensure that you get the exact same library versions that you used when it was last compiled.

Currently, the IDE doesn’t “surface” this, you have to use the underlying “arduino-cli” app. You create a profile for each project and that will contain the libraries used and the versions of same.

HTH

Cheers,
Norm.

[normandunbar]

UPDATE:

Sorry, I forgot to add:

When using lib_extra_dirs you only needed to specify the parent level directory. When that is gone, and no more available, each of your shared libraries will have to be specified individually within lib_deps.

It’s a bit more involved using lib_deps though, you need a library manifest file. See pio pkg install — PlatformIO latest documentation for details.

Cheers,
Norm.

[mitra42]

There may be other uses, but I can tell you how I use it …..

I work mostly on the Frugal-IoT library, but I need test applications for that. I can’t build in the examples directory because it will install the release version of my library, not the one I’m working on. I also need my library, and examples to work in Arduino IDE as there are uses who prefer that.

So instead my structure looks like follows, and I wish there was a simpler way of doing it !!!

• platform.ini // with dependency on `Frugal-IoT`, and src_dir = . commented out
• src
  • main.cpp // The example app I’m testing with
  • other src files specific to this example
• lib
  • Frugal-IoT // Git clone of the library, I push it directly from here
    • examples
      • anexample
        • anexample.ino // So it works with Arduino examples
    • src
      • the library code