Finding libraries in subfolders/submodules on git

I’m setting up a repository of helpful libraries for users to download in one fell swoop when using our boards/code. (GitHub - EnviroDIY/Libraries: Arduino libraries used with the EnviroDIY Mayfly data logger board) Because many of them are external, that repository is really just a grouping of submodules, along with an easy-to-download zip file (because git-archive doesn’t include submodules in the default download button).

If try to install the git repository directly (pio install GitHub - EnviroDIY/Libraries: Arduino libraries used with the EnviroDIY Mayfly data logger board), it downloads everything, including all of the submodules, but behaves as if it installed a single library called “Libraries” with no source code instead of having installed a whole collection of libraries. None of the libraries can be accessed by the compiler.

I tried generating a library.json file within the directory that simply lists the dependent libraries, (Libraries/library.json at master · EnviroDIY/Libraries · GitHub) but this json file is completely ignored.

If I try to install the zip file of all of the compressed libraries (pio install https://github.com/EnviroDIY/Libraries/raw/master/libraries.zip), only the alphabetically first library within the zip is found and installed. Nothing else is decompressed or installed.

Any suggestions for what I’m doing wrong or should I put this up as a bug in the issue tracker?

PIO handles VCS url as a single library. library.json and etc will not help here. Solutions:

  1. You can register your libraries in PIO Library Registry Redirecting...
  2. You can clone GitHub - EnviroDIY/Libraries: Arduino libraries used with the EnviroDIY Mayfly data logger board to $project_folder and add this folder as an extra directory for PIO Library Dependency Finder. See lib_extra_dirs.
cd pio-project
git clone https://github.com/EnviroDIY/Libraries.git EnviroDIYLibs

In platformio.ini

[platformio]
lib_extra_dirs = EnviroDIYLibs

Oh, Thank you! Actually, just adding “lib_extra_dirs = .piolibdeps/EnviroDIYLibs” and keeping the direct git reference in the library worked really well.

2 Likes

Excellent idea/hook! :blush:

Thanks! This was exactly what I was looking for :slight_smile:

This doesn’t seem to be working anymore with the current update.

Could you explain in details?

I’m sorry, that was a terrible message. I’m having build problems now with it not recognizing (any) installed libraries and I’m kind-of baffled as to why. Since you improved the package manager last week, all the dependencies install properly from the JSON in my git repo. That’s awesome and makes setting up the lib_extra_dirs to find the subfolders unnecessary. But somehow now my build is failing at the first “include” statement it hits and my “library dependency graph” is empty. I’m hoping it’s something else I’ve managed to set up wrong and I’ll get it figured out again soon. I’ll update then.

Could you provide a simple project to reproduce this issue?

I created a new simple ‘blink’ project… and it worked fine. It’s still failing when I try to build examples in the library I’m working on. Deleting and recreating the .piolibdeps and .pioenv folders made no difference. Right now I suspect it’s a conflict between the “lib_dir = .” I’m using in the [platformio] section (because I’m working on the library itself) and the libraries themselves in the .piolibdeps directory. Here is the library dependency graph I am getting:

Library Dependency Graph
|-- < src>
| |-- <.piolibdeps>
|-- <.piolibdeps>

Please don’t override lib_dir. Use lib_extra_dirs = ., .piolibdeps/EnviroDIYLibs

I still haven’t been able to get my project to work so I went back to the simple blink project I’d created and started working forward. There is definitely a conflict in using the project folder itself as a library and any other library installed in the project directory (ie, within .piolibdeps). As soon as I add “lib_extra_dirs = .” (or “lib_dir = .”) to the platformio configuration, it fails to find any other library installed in the .piolibdeps or lib folders or even the global library installation folder. Built in libraries and those in other folders referenced with lib_extra_dirs are found, though. (ie, using “lib_extra_dirs = C:\Users\s\Documents\Arduino\Libraries”)

Here’s what I have that’s failing:

test/platformio.ini:

[env:mayfly]
platform = atmelavr
board = mayfly
framework = arduino
lib_extra_dirs = .
lib_deps =
    https://github.com/SodaqMoja/RTCTimer

test/src/lib_test.ino:

#include <RTCTimer.h>
void setup(){}
void loop(){}

Should I elevate this to an issue on GitHub?

I spent some time staring at the changes you made, and I admit I really cannot follow why it broke what I was doing, but it did.

This no longer works to access submodules/subfolders from git. Only the very first submodule is installed. Prior to 3.3.0a17 it worked. (I am not positive the last version it worked on.) Using a separate git clone works fine.

lib_extra_dirs = .piolibdeps/EnviroDIYLibs
lib_deps =
     EnviroDIYLibs=https://github.com/EnviroDIY/Libraries.git

Using “lib_extra_dirs = .” (or “lib_dir = .”) causes all libraries in the global, lib, or .piolibdeps folders to be missed. This can be solved by adding “lib_ignore = .piolibdeps, .pioenvs” to the environment.

The dependencies are installed correctly from a library.json in a VCS URL.

You are right. PIO Package Manager was significantly rewritten in a17. Now, it has a right behavior. You pass raw repository/archive/local folder/etc and it will look for the ROOT of this source that you have passed. Sometimes source code is not located in ROOT of folder.

In this case, PIO Package Manager relies on library manifests. See currently supported list: https://github.com/platformio/platformio-core/blob/develop/platformio/managers/lib.py#L38

Summary. Just add one of manifest to the root of your repo.

P.S: You can also pass to installer RAW archive, like https://github.com/EnviroDIY/Libraries/blob/master/libraries.zip?raw=true

I just clone submodule in lib folder and it is working without anything extra

git submodule add https://github.com/roboter/ArduinoMenu lib/ArduinoMenu