Lib_extra_dirs, lib_deps - Now, how is it work anyway?

[prampec]

I use lib_extra_dir with success in my project to point to a specific library. But that is goes against the specification:

It should be a path to storage that contains libraries grouped by folders.

But personally I use it to include a specific library by pointing this path to the base folder of this lib and not specifying lib_deps.
Now, when I want to combine with lib_deps, things go wrong. Library in the lib_deps is found and downloaded as expected, but Dependency Graph does not contain the lib any more.

It start to work again, if I include the pio/libdeps folder in the lib_extra_dirs like this:

lib_extra_dirs =
   .pio/libdeps/d1/
   ../mylibs/
lib_deps =
  MQTT
  IotWebConf

Same stands, if libs/ folder is not empty.
It seams like in these cases libdeps_dir is not used by the Library Dependency Finder any more.

[maxgerhardt]

Do you have a concrete minimal reproducable example which showcases the issue?

[prampec]

I want to make platformIO folder from an existing example of a library (in our case IotWebConf library). So I check out the source code, and start to create a folder from the root of this library with name ‘pio_examples/IotWebConf06MqttApp’, and start building up PIO from there as follows.

I create a directory structure like this (Only the platformio.ini is new here everything else is just symlink to it’s original location.):

./lib
./lib/IotWebConf -> ../../../
./platformio.ini
./src
./src/main.cpp -> ../../../examples/IotWebConf06MqttApp/IotWebConf06MqttApp.ino

platformio.ini is this:

[env:d1]
platform = espressif8266
board = d1
framework = arduino
monitor_speed = 115200
lib_deps =
  MQTT
  IotWebConf

Compiler output:

> Executing task in folder IotWebConf06MqttApp: pio run --environment d1 <

Processing d1 (platform: espressif8266; board: d1; framework: arduino)
---------------------------------------------------------------------------------------------------------------------------------------------------------
Verbose mode can be enabled via `-v, --verbose` option
CONFIGURATION: https://docs.platformio.org/page/boards/espressif8266/d1.html
PLATFORM: Espressif 8266 2.6.2 > WEMOS D1 R1
HARDWARE: ESP8266 80MHz, 80KB RAM, 4MB Flash
PACKAGES: toolchain-xtensa 2.40802.200502 (4.8.2), framework-arduinoespressif8266 3.20704.0 (2.7.4), tool-esptool 1.413.0 (4.13), tool-esptoolpy 1.20800.0 (2.8.0)
LDF: Library Dependency Finder -> http://bit.ly/configure-pio-ldf
LDF Modes: Finder ~ chain, Compatibility ~ soft
Looking for MQTT library in registry
Conflict: More than one library has been found by request {"name": "MQTT", "requirements": null}:

<...>

Found: https://platformio.org/lib/show/617/MQTT
LibraryManager: Installing id=617
Using cache: /Volumes/Data/ext/kelemenb/.platformio/.cache/d9/23fa68fd8ed789f7f0efa8a2de96c3d9
MQTT @ 2.4.8 has been successfully installed!
Found 31 compatible libraries
Scanning dependencies...
Dependency Graph
|-- <IotWebConf> 2.3.2
|   |-- <DNSServer> 1.1.1
|   |   |-- <ESP8266WiFi> 1.0
|   |-- <ESP8266WebServer> 1.0
|   |   |-- <ESP8266WiFi> 1.0
|   |-- <ESP8266WiFi> 1.0
|   |-- <EEPROM> 1.0
|   |-- <ESP8266mDNS> 1.2
|   |   |-- <ESP8266WiFi> 1.0
Building in release mode
Compiling .pio/build/d1/src/main.cpp.o
Generating LD script .pio/build/d1/ld/local.eagle.app.v6.common.ld
src/main.cpp:39:18: fatal error: MQTT.h: No such file or directory

**************************************************************
* Looking for MQTT.h dependency? Check our library registry!
*
* CLI  > platformio lib search "header:MQTT.h"
* Web  > https://platformio.org/lib/search?query=header:MQTT.h
*
**************************************************************

 #include <MQTT.h>
                  ^
compilation terminated.
Compiling .pio/build/d1/liba63/ESP8266WiFi/BearSSLHelpers.cpp.o
Compiling .pio/build/d1/liba63/ESP8266WiFi/CertStoreBearSSL.cpp.o
Compiling .pio/build/d1/liba63/ESP8266WiFi/ESP8266WiFi.cpp.o
Compiling .pio/build/d1/liba63/ESP8266WiFi/ESP8266WiFiAP.cpp.o
Compiling .pio/build/d1/liba63/ESP8266WiFi/ESP8266WiFiGeneric.cpp.o
Compiling .pio/build/d1/liba63/ESP8266WiFi/ESP8266WiFiGratuitous.cpp.o
*** [.pio/build/d1/src/main.cpp.o] Error 1
============================================================== [FAILED] Took 5.63 seconds ==============================================================

Environment    Status    Duration
-------------  --------  ------------
d1             FAILED    00:00:05.629
========================================================= 1 failed, 0 succeeded in 00:00:05.629 =========================================================
The terminal process "pio 'run', '--environment', 'd1'" terminated with exit code: 1.

Terminal will be reused by tasks, press any key to close it.

I see, that ‘.pio/libdeps/d1/MQTT’ is created by the build, and contains all the source code of this lib:
./.pio/libdeps
./.pio/libdeps/d1
./.pio/libdeps/d1/MQTT
./.pio/libdeps/d1/MQTT/.editorconfig
./.pio/libdeps/d1/MQTT/.gitignore
./.pio/libdeps/d1/MQTT/.travis.yml
./.pio/libdeps/d1/MQTT/CMakeLists.txt
./.pio/libdeps/d1/MQTT/LICENSE.md
./.pio/libdeps/d1/MQTT/Makefile
./.pio/libdeps/d1/MQTT/README.md
…
./.pio/libdeps/d1/MQTT/library.properties
./.pio/libdeps/d1/MQTT/src
./.pio/libdeps/d1/MQTT/src/MQTT.h
./.pio/libdeps/d1/MQTT/src/MQTTClient.cpp
./.pio/libdeps/d1/MQTT/src/MQTTClient.h
./.pio/libdeps/d1/MQTT/src/lwmqtt
./.pio/libdeps/d1/MQTT/src/lwmqtt/client.c
./.pio/libdeps/d1/MQTT/src/lwmqtt/helpers.c
./.pio/libdeps/d1/MQTT/src/lwmqtt/helpers.h
./.pio/libdeps/d1/MQTT/src/lwmqtt/lwmqtt.h
./.pio/libdeps/d1/MQTT/src/lwmqtt/packet.c
./.pio/libdeps/d1/MQTT/src/lwmqtt/packet.h
./.pio/libdeps/d1/MQTT/src/lwmqtt/string.c

Now I modify platformio.ini as follows, and the build will finish successfully this time:

[env:d1]
platform = espressif8266
board = d1
framework = arduino
monitor_speed = 115200
lib_extra_dirs =
  .pio/libdeps/d1/
lib_deps =
  MQTT
  IotWebConf

[maxgerhardt]

@prampec

lib_deps =
  MQTT
  IotWebConf

Does this happen too when the proper lib_deps declarations are used, with owner name and version?

lib_deps = 
     256dpi/MQTT @ ^2.4.8
     prampec/IotWebConf @ ^2.3.2

[prampec]

‘prampec/IotWebConf @ ^2.3.2’ in lib_deps does not make sense, as I want LDF to find it in the lib folder, and not to have it downloaded.

When I try to use ’ 256dpi/MQTT @ ^2.4.8’ I’ve got other serious download errors, that are not connected to the topic. (Btw. the correct format might be ‘256dpi/arduino-mqtt@^2.4.8’ ? But it is also not usable.) E.g.:

AssertionError: Package version 72fb1d7 doesn't satisfy requirements ^2.4.8

[maxgerhardt]

@prampec

Btw. the correct format might be ‘256dpi/arduino-mqtt@^2.4.8’ ?

Nope since the library declares its name as

256dpi/arduino-mqtt/blob/72fb1d70158b8745321744a73c07470341afa3a3/library.properties#L1-L2

I copied the expressions from PlatformIO Registry.

@prampec

AssertionError: Package version 72fb1d7 doesn't satisfy requirements ^2.4.8

This seems weird. What does PIO Home say about the core version?

[prampec]

PIO Home is not actually working inside my IDE (VSCode@MacOS), but starting it from command line is shows: Home 3.0.1 - Core 4.1.0

[maxgerhardt]

@prampec

Core 4.1.0

Okay that explains a lot in regards to why the new lib_deps declarations don’t work. 4.1.0 is from September 2019, the current stable version is 5.0.3 from November 2020, which had major overhauls done (there’s a 4.x → 5 migration guide).

I suggest you completely remove the VSCode PlatformIO extension and re-install it, to get hopefully the newest core version. If it still behaves erratically in the new version, a bug can be filed to address that, but not in the outdated version.

[prampec]

I have PlatformIO IDE plugin installed in VSCode plugin v2.2.1. I have removed it, and then reinstalled it (it took 1 sec!), and nothing has changed after IDE restart.

It seams like I was forced somehow to install PIO from Homebrew, and that is why I was stuck on an older version. Now, after updated to Core v5.0.3 this issue is not reproducible any more. Thanks for the support!

It is strange tho, that availability of new version is not reported to me somehow. (And btw. PIO Home is still not available from the VSCode ide.)

[maxgerhardt]

@prampec

Now, after updated to Core v5.0.3 this issue is not reproducible any more

Yay

@prampec

(And btw. PIO Home is still not available from the VSCode ide.)

Then it’s not 100% over yet. What exactly happens when you go to the sidebar and do Quick Access → PIO Home → Open? (while having all manual instances of pio home closed)

Does Help → Toggle Developer Tools → Console reveal any error messages?

[prampec]

I’ve opened a new ticket with this:

PlatformIO Home does not open inside VSCode when *not* using "Builtin PIOCore"

When I open PIO Home from VSCode it’s just loading until timeout. Console contains a lot of message, the last ones are “PlatformIO Core is not installed”. That might explains some problem… I have PlatformIO installed from Homebrew. In the VSCode extension settings I have the “Use Builtin PIOCore” unchecked. pio command can be run from the console (also the ‘pio home’), and all other functions of the IDE integration seams to be working (e.g build, upload, monitor). PIO Core ver…