LDF dependency downloads are incomplete

mbarney · November 17, 2020, 5:42pm

Description

My project has 1 library specified in lib_deps, and that library has another 20+ dependencies listed in its library.json manifest. However, only 3 of those dependencies are being automatically downloaded by Library Dependency Finder.

This problem only seems to occur with IDE v2.1.1 and later.

To Reproduce

Create a new project from PIO Home->New Project, and select the board EnviroDIY Mayfly.
Replace the contents of platformio.ini with the following, and save it:
[env:mayfly]
platform = atmelavr
board = mayfly
framework = arduino
lib_deps = EnviroDIY_ModularSensors@0.25.1
lib_ldf_mode = deep+
lib_ignore = RTCZero
build_flags = -DSDI12_EXTERNAL_PCINT
Click Build.

Results

When you save the ini file, LDF installs only the following into .pio\libdeps\mayfly:
EnviroDIY_ModularSensors and 3 of its dependencies. They happen to be the first 3 dependencies listed in that library’s manifest.
When you build, it fails with the error:
In file included from .pio\libdeps\mayfly\EnviroDIY_ModularSensors\src\modems\DigiXBee3GBypass.cpp:11:0: .pio\libdeps\mayfly\EnviroDIY_ModularSensors\src\modems\DigiXBee3GBypass.h:37:27: fatal error: TinyGsmClient.h: No such file or directory

Expected Behavior

LDF installs EnviroDIY_ModularSensors as well as all 23 of its dependencies.
Build succeeds.

Notes

I can avoid the problem by downgrading the IDE as follows:

Downgrade the PlatformIO IDE extension to v2.1.0 or earlier.
Close VSCode.
Delete the .pio folder.
Reopen the project in VSCode.

Then, LDF downloads all dependencies as expected. Any newer version of the extension, including 2.2.0, produces the condition described above, under Results.

My Environment

PlatformIO Core 5.0.2
PlatformIO Home 3.3.1
PlatformIO IDE extension 2.2.0
VSCode 1.51.0
Windows 10 Pro 64-bit

Note that I also updated to Core v5.0.3 and IDE v2.2.1, with no change in behavior.

maxgerhardt · November 17, 2020, 7:42pm

When I copy your platformio.ini and some basic example code, pio run does

pio run
Processing mayfly (platform: atmelavr; board: mayfly; framework: arduino)
------------------------------------------------------------------------------------------------------------------------
Verbose mode can be enabled via `-v, --verbose` option
CONFIGURATION: https://docs.platformio.org/page/boards/atmelavr/mayfly.html
PLATFORM: Atmel AVR (2.2.0) > EnviroDIY Mayfly
HARDWARE: ATMEGA1284P 8MHz, 16KB RAM, 127KB Flash
DEBUG: Current (simavr) On-board (simavr)
PACKAGES:
 - framework-arduino-avr 5.0.0
 - toolchain-atmelavr 1.50400.190710 (5.4.0)
LDF: Library Dependency Finder -> http://bit.ly/configure-pio-ldf
LDF Modes: Finder ~ deep+, Compatibility ~ soft
Library Manager: Installing EnviroDIY_ModularSensors @ 0.25.1
Library Manager: EnviroDIY_ModularSensors @ 0.25.1 has been installed!
Library Manager: Installing dependencies...
Library Manager: Installing EnviroDIY_DS3231 @ =1.3.2
Library Manager: EnviroDIY_DS3231 @ 1.3.2 has been installed!
Library Manager: Installing EnableInterrupt @ =1.1.0
Library Manager: EnableInterrupt @ 1.1.0 has been installed!
Library Manager: Installing SdFat @ =1.1.2
Library Manager: SdFat @ 1.1.2 has been installed!
Library Manager: Installing git+https://github.com/soligen2010/Adafruit_ADS1X15.git
git version 2.29.0.windows.1
Cloning into 'C:\Users\Maxi\.platformio\.cache\tmp\pkg-installing-ucc0o_ab'...
Library Manager: Adafruit ADS1X15 @ 1.2.0+sha.7d67b45 has been installed!
Library Manager: Installing Adafruit AM2315 @ =2.0.0
Library Manager: Adafruit AM2315 @ 2.0.0 has been installed!
Library Manager: Installing Adafruit BME280 Library @ =2.0.0
Library Manager: Adafruit BME280 Library @ 2.0.0 has been installed!
Library Manager: Installing dependencies...
Library Manager: Installing Adafruit Unified Sensor
Library Manager: Adafruit Unified Sensor @ 1.1.4 has been installed!
Library Manager: Installing DHT sensor library @ =1.3.8
Library Manager: DHT sensor library @ 1.3.8 has been installed!
Library Manager: Installing dependencies...
Library Manager: Installing Adafruit INA219 @ =1.0.6
Library Manager: Adafruit INA219 @ 1.0.6 has been installed!
Library Manager: Installing Adafruit MPL115A2 @ =1.1.2
Library Manager: Adafruit MPL115A2 @ 1.1.2 has been installed!
Library Manager: Installing Adafruit Unified Sensor @ =1.1.2
Library Manager: Adafruit Unified Sensor @ 1.1.2 has been installed!
Library Manager: Installing dependencies...
Library Manager: Installing Adafruit ADXL343
Library Manager: Adafruit ADXL343 @ 1.3.0 has been installed!
Library Manager: Installing dependencies...
Library Manager: Installing Adafruit BusIO
Library Manager: Adafruit BusIO @ 1.6.0 has been installed!
Library Manager: Installing DallasTemperature @ =3.8.1
Library Manager: DallasTemperature @ 3.8.1 has been installed!
Library Manager: Installing dependencies...
Library Manager: Installing OneWire
Library Manager: OneWire @ 2.3.5 has been installed!
Library Manager: OneWire @ 2.3.5 is already installed
Library Manager: Installing SDI-12 @ =2.1.0
Library Manager: SDI-12 @ 2.1.0 has been installed!
Library Manager: Installing KellerModbus @ =0.2.1
Library Manager: KellerModbus @ 0.2.1 has been installed!
Library Manager: Installing dependencies...
Library Manager: Installing SensorModbusMaster @ >=0.6.2
Library Manager: SensorModbusMaster @ 0.6.8 has been installed!
Library Manager: Installing MS5803 @ =0.1.2
Library Manager: MS5803 @ 0.1.2 has been installed!
Library Manager: Installing PubSubClient @ =2.7
Library Manager: PubSubClient @ 2.7.0 has been installed!
Library Manager: Installing RTCZero @ =1.6.0
Library Manager: RTCZero @ 1.6.0 has been installed!
Library Manager: SensorModbusMaster @ 0.6.8 is already installed
Library Manager: Installing git+https://github.com/EnviroDIY/TinyGSM.git
git version 2.29.0.windows.1
Cloning into 'C:\Users\Maxi\.platformio\.cache\tmp\pkg-installing-24vd7ou0'...
Library Manager: TinyGSM @ 0.10.9+sha.1576ecd has been installed!
Library Manager: Installing YosemitechModbus @ =0.2.2

Downloading  [########################------------]   66%
Downloading  [########################------------]   68%
Downloading  [#########################-----------]   69%
Downloading  [#########################-----------]   71%
Downloading  [##########################----------]   73%
Downloading  [##########################----------]   74%
Downloading  [###########################---------]   76%
Downloading  [############################--------]   77%
Downloading  [############################--------]   79%
Downloading  [#############################-------]   80%
Downloading  [#############################-------]   82%
Downloading  [##############################------]   84%
Downloading  [##############################------]   85%
Downloading  [###############################-----]   87%
Downloading  [################################----]   88%
Downloading  [################################----]   90%
Downloading  [#################################---]   92%
Downloading  [#################################---]   93%
Downloading  [##################################--]   95%
Downloading  [##################################--]   96%
Downloading  [###################################-]   98%
Downloading  [####################################]  100%
Library Manager: YosemitechModbus @ 0.2.2 has been installed!
Library Manager: Installing dependencies...
Library Manager: SensorModbusMaster @ 0.6.8 is already installed
Found 33 compatible libraries

so all libraries are installed, including TinyGSM. Compilation then succeeds too.

RAM:   [          ]   0.1% (used 9 bytes from 16384 bytes)
Flash: [          ]   0.4% (used 510 bytes from 130048 bytes)
============================ [SUCCESS] Took 45.49 seconds ============================

that is with the very latest PIO core obtained via pio ugprade --dev (ver. 5.0.4a2 as of now).

Now there was no VSCode IDE involved here. If I rm -rf .pio again and press the Build button in VSCode, I get…

> Executing task in folder nano33ble: C:\Users\Maxi\AppData\Local\Programs\Python\Python37\Scripts\pio.exe run <

Processing mayfly (platform: atmelavr; board: mayfly; framework: arduino)
---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

Verbose mode can be enabled via `-v, --verbose` option
CONFIGURATION: https://docs.platformio.org/page/boards/atmelavr/mayfly.html
PLATFORM: Atmel AVR (2.2.0) > EnviroDIY Mayfly   
HARDWARE: ATMEGA1284P 8MHz, 16KB RAM, 127KB Flash
DEBUG: Current (simavr) On-board (simavr)        
PACKAGES:
 - framework-arduino-avr 5.0.0
 - toolchain-atmelavr 1.50400.190710 (5.4.0)     
LDF: Library Dependency Finder -> http://bit.ly/configure-pio-ldf
LDF Modes: Finder ~ deep+, Compatibility ~ soft
Found 15 compatible libraries
Scanning dependencies...
Dependency Graph
|-- <EnviroDIY_ModularSensors> 0.25.1
|   |-- <EnviroDIY_DS3231> 1.3.2     
|   |   |-- <Wire> 1.0
|   |-- <EnableInterrupt> 1.1.0      
|   |-- <SdFat> 1.1.2
|   |   |-- <SPI> 1.0
|   |-- <YosemitechModbus>
|   |-- <SPI> 1.0
|   |-- <Wire> 1.0
Building in release mode

…somehow the PIO extension must have downloaded some (incomplete set) of dependencies here in .pio\libdeps after I removed the folder and pio run then further doesn’t verify that all dependencies are installed, and then fails. That’s also on the latest 2.2.1 extension version.

When inside VSCode and I remove the .pio\libdeps folder in the GUI and then pio run, compilation proceeds succesfully and installs all dependencies using pio run.

@ivankravets, does the PIO VSCode extension attempt some auto-download of libraries when e.g. the platformio.ini is reloaded? Does it respect lib_ldf_mode = deep+ then? Or is this caused by something else. (I’m doing a direct @ here because the linked github issue hasn’t moved since 11 days )

Also note that reportededly an extension downgrade to 2.1.0 works to restore the good behavior.

ivankravets · November 18, 2020, 9:14pm

I moved that issue to the Core

github.com/platformio/platformio-core

LDF only downloading some of the dependencies listed in library manifest

opened 12:07AM - 07 Nov 20 UTC

closed 11:54AM - 23 Dec 20 UTC

mbarneytu

help wanted ldf

### Description My project has 1 library specified in lib_deps, and that librar…y has another 20+ dependencies listed in its library.json manifest. However, only 3 of those dependencies are being automatically downloaded by Library Dependency Finder. ### To Reproduce 1. Create a new project from PIO Home->New Project, and select the board EnviroDIY Mayfly. 2. Replace the contents of platformio.ini with the following, and save it: `[env:mayfly]` `platform = atmelavr` `board = mayfly` `framework = arduino` `lib_deps = EnviroDIY_ModularSensors@0.25.1` `lib_ldf_mode = deep+` `lib_ignore = RTCZero` `build_flags = -DSDI12_EXTERNAL_PCINT` 3. Click Build. ### Results - When you save the ini file, LDF installs only the following into .pio\libdeps\mayfly: EnviroDIY_ModularSensors and 3 of its dependencies. They happen to be the first 3 dependencies listed in that library's [manifest.](https://github.com/EnviroDIY/ModularSensors/blob/master/library.json) - When you build, it fails with the error: `In file included from .pio\libdeps\mayfly\EnviroDIY_ModularSensors\src\modems\DigiXBee3GBypass.cpp:11:0: .pio\libdeps\mayfly\EnviroDIY_ModularSensors\src\modems\DigiXBee3GBypass.h:37:27: fatal error: TinyGsmClient.h: No such file or directory` ### Expected Behavior - LDF installs EnviroDIY_ModularSensors as well as all 23 of its dependencies. - Build succeeds. ### Notes I can avoid the problem by downgrading the IDE as follows: 1. Downgrade the PlatformIO IDE extension to v2.1.0 or earlier. 2. Close VSCode. 3. Delete the .pio folder. 4. Reopen the project in VSCode. Then, LDF downloads all dependencies as expected. Any newer version of the extension, including 2.2.0, produces the condition described above, under Results. ### My Environment PlatformIO Core 5.0.2 PlatformIO Home 3.3.1 PlatformIO IDE extension 2.2.0 VSCode 1.51.0 Windows 10 Pro 64-bit

So, we will investigate. In case, the manifest looks strange for me.

SRGDamia1 · December 10, 2020, 4:25pm

So, we will investigate. In case, the manifest looks strange for me.

I’m sorry, this is my library and it has a lot of dependencies.

Within the manifest json’s dependency section, I added some extra fields (author, note, url) that were just intended to be extra information for any curious users. I assumed the parser would just skip or ignore the extra fields. It doesn’t seem like they’re the problem, but they would make the manifest “look strange”. If you think they’re the cause of the trouble, I can remove them. If there’s something else strange, let me know!

mbarney · December 23, 2020, 4:21pm

This is fixed in the latest dev version of PIO, 5.0.4b1. I did a pio upgrade --dev , then followed my original ‘steps to reproduce’, and dependencies download properly and build succeeds. Thanks!