Pio compiles dependencies for all platforms when only a specific platform specified per dependency

as-iotex · November 27, 2021, 11:34am

Hello

I have developed this library: GitHub - iotexproject/arduino-sdk

The library supports multiple platforms (esp8266, esp32, atmelsam). For atmelsam ONLY, the library needs the dependencies FlashStorage and WiFiNINA. These are specified in library.json as follows:

"dependencies":
    [
      {
        "name": "FlashStorage",
        "platforms": ["atmelsam"]
      },
      {
        "name": "WiFiNINA",
        "platforms": ["atmelsam"]
      }
    ]

However, when building any of the examples for esp8266 or esp32 platforms, platformio tries to build those dependencies too. This causes a build error because they are not compatible

In order to reproduce:

Clone the repo
Try to build any of the examples in the examples folder for esp8266 or esp32 environments

Note: as the library is not published yet to pio manager, you’ll need to add a lib_extra_dirs entry pointing to the directory where you have cloned the repository

Expected behavior: the FlashStorage and WiFiNINA libraries are not compiled and the example builds
Actual behavior: the FlashStorage and WiFiNINA libraries are compiled causing a build error

Am I missing something?

Thanks for the help

jcw · November 27, 2021, 4:13pm

See Redirecting...

By default, this value is set to lib_compat_mode = soft and means that LDF will check only for framework compatibility.

maxgerhardt · November 28, 2021, 5:41am

The library.json can also declare the strict mode instead of the user having to do it in the project’s platformio.ini, per documentation.

{
   "build": {
      "libCompatMode": "strict"
   }
}

as-iotex · November 28, 2021, 11:12am

Thank you for your quick replies
Unfortunately they don´t make any change

I tried adding lib_compat_mode = strict to platformio.ini and also "libCompatMode": "strict" as maxgerhardt suggested. The result is the same. Pio still tries to build the dependencies for esp32 and esp8266 even if the platform specified on them is atmelsam

I also tried changing the value of libLDFMode but nothing seems to make a change

jcw · November 28, 2021, 1:03pm

Odd. Unfortunately, I can’t replicate your build:

CONFIGURATION: https://docs.platformio.org/page/boards/espressif8266/esp_wroom_02.html
PLATFORM: Espressif 8266 (3.2.0) > ESP-WROOM-02
HARDWARE: ESP8266 80MHz, 80KB RAM, 2MB Flash
PACKAGES:
 - framework-arduinoespressif8266 3.30002.0 (3.0.2)
 - tool-esptool 1.413.0 (4.13)
 - tool-esptoolpy 1.30000.201119 (3.0.0)
 - toolchain-xtensa 2.100300.210717 (10.3.0)
Converting SignMessage.ino
LDF: Library Dependency Finder -> https://bit.ly/configure-pio-ldf
LDF Modes: Finder ~ deep+, Compatibility ~ strict
Library Manager: Installing IoTeX-blockchain-client
Warning! Could not find the package with 'IoTeX-blockchain-client' requirements for your system 'darwin_arm64'
Found 35 compatible libraries
Scanning dependencies...
No dependencies
Building in release mode
Compiling .pio/build/esp8266/src/SignMessage.ino.cpp.o
Compiling .pio/build/esp8266/FrameworkArduino/Print.cpp.o
Compiling .pio/build/esp8266/FrameworkArduino/Schedule.cpp.o
Compiling .pio/build/esp8266/FrameworkArduino/StackThunk.cpp.o
Compiling .pio/build/esp8266/FrameworkArduino/Stream.cpp.o
Compiling .pio/build/esp8266/FrameworkArduino/StreamSend.cpp.o
Compiling .pio/build/esp8266/FrameworkArduino/Tone.cpp.o
Compiling .pio/build/esp8266/FrameworkArduino/TypeConversion.cpp.o
/Users/jcw/Desktop/arduino-sdk/examples/SignMessage/SignMessage.ino:3:10: fatal error: IoTeX-blockchain-client.h: No such file or directory

Note the Compatibility ~ strict which I added to SignMessage/platformio.ini.
It looks like the above issue is not related to this.

as-iotex · November 28, 2021, 9:13pm

The library is not published to platformio library manager yet
In order for platformio to recognize the library you need to clone it manually from github and then add a lib_extra_dirs entry pointing to the directory where you have cloned the repo.

Alternatively you could set the PLATFORMIO_LIB_EXTRA_DIRS environment variable before calling platformio run

For example, if you are using bash:

git clone https://github.com/iotexproject/arduino-sdk.git
# Set the value of PLATFORMIO_LIB_EXTRA_DIRS to the current directory
export PLATFORMIO_LIB_EXTRA_DIRS=`pwd`
cd arduino-sdk/examples/GetAccountMetadata
platformio run --environment nano_33_iot
platformio run --environment esp32

nano_33_iot builds fine
esp32 build fails because it tries to build WifiNINA. See the output below (with strict mode):

export PLATFORMIO_LIB_EXTRA_DIRS="~/Documents/Arduino/libraries" && pio run --environment esp32
Processing esp32 (platform: espressif32; board: esp32dev; framework: arduino)
-------------------------------------------------------------------------------------------------------------------------------------------------
Verbose mode can be enabled via `-v, --verbose` option
CONFIGURATION: https://docs.platformio.org/page/boards/espressif32/esp32dev.html
PLATFORM: Espressif 32 (1.12.1) > Espressif ESP32 Dev Module
HARDWARE: ESP32 240MHz, 320KB RAM, 4MB Flash
DEBUG: Current (esp-prog) External (esp-prog, iot-bus-jtag, jlink, minimodule, olimex-arm-usb-ocd, olimex-arm-usb-ocd-h, olimex-arm-usb-tiny-h, olimex-jtag-tiny, tumpa)
PACKAGES: 
 - framework-arduinoespressif32 3.10004.200129 (1.0.4) 
 - tool-esptoolpy 1.20600.0 (2.6.0) 
 - toolchain-xtensa32 2.50200.80 (5.2.0)
Converting GetAccountMetadata.ino
LDF: Library Dependency Finder -> http://bit.ly/configure-pio-ldf
LDF Modes: Finder ~ deep+, Compatibility ~ strict
Found 28 compatible libraries
Scanning dependencies...
Dependency Graph
|-- <IoTeX-blockchain-client> 0.0.1
|   |-- <WiFiNINA> 1.8.13
|   |   |-- <SPI> 1.0
|   |-- <HTTPClient> 1.2
|   |   |-- <WiFiNINA> 1.8.13
|   |   |   |-- <SPI> 1.0
|   |   |-- <WiFiClientSecure> 1.0
|   |   |   |-- <WiFiNINA> 1.8.13
|   |   |   |   |-- <SPI> 1.0
|   |-- <WiFiClientSecure> 1.0
|   |   |-- <WiFiNINA> 1.8.13
|   |   |   |-- <SPI> 1.0
|   |-- <EEPROM> 1.0.3
|-- <WiFiNINA> 1.8.13
|   |-- <SPI> 1.0
Building in release mode
Compiling .pio/build/esp32/src/GetAccountMetadata.ino.cpp.o
Compiling .pio/build/esp32/lib9da/WiFiNINA/WiFi.cpp.o
Compiling .pio/build/esp32/lib9da/WiFiNINA/WiFiClient.cpp.o
Compiling .pio/build/esp32/lib9da/WiFiNINA/WiFiServer.cpp.o
Compiling .pio/build/esp32/lib9da/WiFiNINA/WiFiStorage.cpp.o
Compiling .pio/build/esp32/lib9da/WiFiNINA/WiFiUdp.cpp.o
Compiling .pio/build/esp32/lib9da/WiFiNINA/utility/spi_drv.cpp.o
Compiling .pio/build/esp32/lib9da/WiFiNINA/utility/wifi_drv.cpp.o
In file included from /Users/__/Documents/Arduino/libraries/WiFiNINA/src/WiFiStorage.h:23:0,
                 from /Users/__/Documents/Arduino/libraries/WiFiNINA/src/WiFi.h:38,
                 from /Users/__/Documents/Arduino/libraries/iotex-client/examples/GetAccountMetadata/GetAccountMetadata.ino:4:
/Users/__/Documents/Arduino/libraries/WiFiNINA/src/utility/wifi_drv.h:293:12: error: 'PinStatus' does not name a type
     static PinStatus digitalRead(uint8_t pin);
            ^
In file included from /Users/__/Documents/Arduino/libraries/WiFiNINA/src/WiFi.cpp:21:0:
/Users/__/Documents/Arduino/libraries/WiFiNINA/src/utility/wifi_drv.h:293:12: error: 'PinStatus' does not name a type
     static PinStatus digitalRead(uint8_t pin);
            ^
In file included from /Users/__/Documents/Arduino/libraries/WiFiNINA/src/WiFiClient.cpp:30:0:
/Users/__/Documents/Arduino/libraries/WiFiNINA/src/utility/wifi_drv.h:293:12: error: 'PinStatus' does not name a type
     static PinStatus digitalRead(uint8_t pin);
            ^
*** [.pio/build/esp32/lib9da/WiFiNINA/WiFi.cpp.o] Error 1
*** [.pio/build/esp32/lib9da/WiFiNINA/WiFiClient.cpp.o] Error 1
In file included from /Users/__/Documents/Arduino/libraries/iotex-client/src/protobuf/pb_api.h:15:0,
                 from /Users/__/Documents/Arduino/libraries/iotex-client/src/api/rpc.h:9,
                 from /Users/__/Documents/Arduino/libraries/iotex-client/src/api/wallet/wallets.h:5,
                 from /Users/__/Documents/Arduino/libraries/iotex-client/src/api/api.h:7,
                 from /Users/__/Documents/Arduino/libraries/iotex-client/src/connection/connection.h:5,
                 from /Users/__/Documents/Arduino/libraries/iotex-client/src/account/account.h:4,
                 from /Users/__/Documents/Arduino/libraries/iotex-client/src/IoTeX-blockchain-client.h:4,
                 from /Users/__/Documents/Arduino/libraries/iotex-client/examples/GetAccountMetadata/GetAccountMetadata.ino:17:
/Users/__/Documents/Arduino/libraries/iotex-client/src/helpers/json_helper.h: In member function 'virtual void iotex::json::Uint64JsonProp::appendValueToString(String&)':
/Users/__/Documents/Arduino/libraries/iotex-client/src/helpers/json_helper.h:228:36: warning: format '%lu' expects argument of type 'long unsigned int', but argument 3 has type 'uint64_t {aka long long unsigned int}' [-Wformat=]
   sprintf(buf, "%lu", _value.uint64);
                                    ^
/Users/__/Documents/Arduino/libraries/iotex-client/src/helpers/json_helper.h:228:36: warning: format '%lu' expects argument of type 'long unsigned int', but argument 3 has type 'uint64_t {aka long long unsigned int}' [-Wformat=]
In file included from /Users/__/Documents/Arduino/libraries/WiFiNINA/src/WiFiStorage.h:23:0,
                 from /Users/__/Documents/Arduino/libraries/WiFiNINA/src/WiFi.h:38,
                 from /Users/__/Documents/Arduino/libraries/WiFiNINA/src/WiFiServer.cpp:28:
/Users/__/Documents/Arduino/libraries/WiFiNINA/src/utility/wifi_drv.h:293:12: error: 'PinStatus' does not name a type
     static PinStatus digitalRead(uint8_t pin);
            ^
/Users/__/Documents/Arduino/libraries/iotex-client/examples/GetAccountMetadata/GetAccountMetadata.ino: In function 'void initWiFi()':
/Users/__/Documents/Arduino/libraries/iotex-client/examples/GetAccountMetadata/GetAccountMetadata.ino:31:10: error: 'class WiFiClass' has no member named 'mode'
     WiFi.mode(WIFI_STA);
          ^
/Users/__/Documents/Arduino/libraries/iotex-client/examples/GetAccountMetadata/GetAccountMetadata.ino:31:15: error: 'WIFI_STA' was not declared in this scope
     WiFi.mode(WIFI_STA);
               ^
*** [.pio/build/esp32/lib9da/WiFiNINA/WiFiServer.cpp.o] Error 1
In file included from /Users/__/Documents/Arduino/libraries/iotex-client/src/contract/contract.h:4:0,
                 from /Users/__/Documents/Arduino/libraries/iotex-client/src/IoTeX-blockchain-client.h:7,
                 from /Users/__/Documents/Arduino/libraries/iotex-client/examples/GetAccountMetadata/GetAccountMetadata.ino:17:
/Users/__/Documents/Arduino/libraries/iotex-client/src/abi/abi.h: At global scope:
/Users/__/Documents/Arduino/libraries/iotex-client/src/abi/abi.h:41:20: warning: 'iotex::abi::EthereumTypeNameToStringLT' defined but not used [-Wunused-variable]
 static const char* EthereumTypeNameToStringLT[] = {"uint",  "int",   "address", "bool",  "fixed",
                    ^
*** [.pio/build/esp32/src/GetAccountMetadata.ino.cpp.o] Error 1
In file included from /Users/__/Documents/Arduino/libraries/WiFiNINA/src/WiFiStorage.h:23:0,
                 from /Users/__/Documents/Arduino/libraries/WiFiNINA/src/WiFiStorage.cpp:20:
/Users/__/Documents/Arduino/libraries/WiFiNINA/src/utility/wifi_drv.h:293:12: error: 'PinStatus' does not name a type
     static PinStatus digitalRead(uint8_t pin);
            ^
In file included from /Users/__/Documents/Arduino/libraries/WiFiNINA/src/WiFiUdp.cpp:29:0:
/Users/__/Documents/Arduino/libraries/WiFiNINA/src/utility/wifi_drv.h:293:12: error: 'PinStatus' does not name a type
     static PinStatus digitalRead(uint8_t pin);
            ^
*** [.pio/build/esp32/lib9da/WiFiNINA/WiFiStorage.cpp.o] Error 1
*** [.pio/build/esp32/lib9da/WiFiNINA/WiFiUdp.cpp.o] Error 1
In file included from /Users/__/Documents/Arduino/libraries/WiFiNINA/src/WiFiStorage.h:23:0,
                 from /Users/__/Documents/Arduino/libraries/WiFiNINA/src/WiFi.h:38,
                 from /Users/__/Documents/Arduino/libraries/WiFiNINA/src/utility/spi_drv.cpp:25:
/Users/__/Documents/Arduino/libraries/WiFiNINA/src/utility/wifi_drv.h:293:12: error: 'PinStatus' does not name a type
     static PinStatus digitalRead(uint8_t pin);
            ^
/Users/__/Documents/Arduino/libraries/WiFiNINA/src/utility/spi_drv.cpp: In static member function 'static void SpiDrv::begin()':
/Users/__/Documents/Arduino/libraries/WiFiNINA/src/utility/spi_drv.cpp:94:24: error: 'PINS_COUNT' was not declared in this scope
       if (SLAVERESET > PINS_COUNT) {
                        ^
/Users/__/Documents/Arduino/libraries/WiFiNINA/src/utility/spi_drv.cpp:103:15: error: 'NINA_GPIO0' was not declared in this scope
       pinMode(NINA_GPIO0, OUTPUT);
               ^
/Users/__/Documents/Arduino/libraries/WiFiNINA/src/utility/spi_drv.cpp: In static member function 'static int SpiDrv::available()':
/Users/__/Documents/Arduino/libraries/WiFiNINA/src/utility/spi_drv.cpp:66:25: error: 'NINA_GPIO0' was not declared in this scope
 #define NINA_GPIOIRQ    NINA_GPIO0
                         ^
/Users/__/Documents/Arduino/libraries/WiFiNINA/src/utility/spi_drv.cpp:597:25: note: in expansion of macro 'NINA_GPIOIRQ'
     return (digitalRead(NINA_GPIOIRQ) != LOW);
                         ^
In file included from /Users/__/Documents/Arduino/libraries/WiFiNINA/src/utility/wifi_drv.cpp:27:0:
/Users/__/Documents/Arduino/libraries/WiFiNINA/src/utility/wifi_drv.h:293:12: error: 'PinStatus' does not name a type
     static PinStatus digitalRead(uint8_t pin);
            ^
*** [.pio/build/esp32/lib9da/WiFiNINA/utility/spi_drv.cpp.o] Error 1
/Users/__/Documents/Arduino/libraries/WiFiNINA/src/utility/wifi_drv.cpp:1063:1: error: 'PinStatus' does not name a type
 PinStatus WiFiDrv::digitalRead(uint8_t pin)
 ^
*** [.pio/build/esp32/lib9da/WiFiNINA/utility/wifi_drv.cpp.o] Error 1

Please me know if you are able to reproduce the error following my instructions

jcw · November 29, 2021, 12:07am

For pio run -e nano_33_iot, I get errors:

LDF Modes: Finder ~ deep+, Compatibility ~ strict
Found 12 compatible libraries
Scanning dependencies...
No dependencies
Building in release mode
Compiling .pio/build/nano_33_iot/src/GetAccountMetadata.ino.cpp.o
Compiling .pio/build/nano_33_iot/FrameworkArduinoVariant/variant.cpp.o
Compiling .pio/build/nano_33_iot/FrameworkArduino/Reset.cpp.o
Compiling .pio/build/nano_33_iot/FrameworkArduino/SERCOM.cpp.o
Compiling .pio/build/nano_33_iot/FrameworkArduino/Tone.cpp.o
Compiling .pio/build/nano_33_iot/FrameworkArduino/USB/CDC.cpp.o
Compiling .pio/build/nano_33_iot/FrameworkArduino/USB/USBCore.cpp.o
Compiling .pio/build/nano_33_iot/FrameworkArduino/USB/samd21_host.c.o
/Users/jcw/Desktop/arduino-sdk/examples/GetAccountMetadata/GetAccountMetadata.ino:12:14: fatal error: WiFiNINA.h: No such file or directory

Hrmph, I don’t even get to pio run -e esp32. Note the “No dependencies” message.
I’m sorry, there are too many moving parts for me to pursue this issue any further.
I suggest stripping your project way down to get to the core of the problem.

as-iotex · November 29, 2021, 11:20am

Thank you again

The behavior in your machine was completely different to mine. So I did more investigation. I found something very weird and confusing with PlatformIO, which is what is causing platformio to always build the FlashStorage and WiFiNINA libraries. I suspect this is a bug in platformio and shoulg go into a Github issue.

Here’s my setup:

In order to make development easier, I have the repo in my Arduino IDE libraries directory.
When I build with Arduino IDE, the WiFININA and FlashStorage libraries are downloaded in the same directory as my library. So, here’s the file structure:

/Users/__/Documents/Arduino/libraries/
├── FlashStorage
├── WiFiNINA
├── arduino-sdk
└── readme.txt

Now when I build any example from arduino-sdk/examples with platformio, it detects FlashStorage and WiFiNINA as dependencies. This happens even if I remove all the "dependencies" from library.json. Here’s the output:

|-- <IoTeX-blockchain-client> 0.0.1
|   |-- <HTTPClient> 1.2
|   |   |-- <WiFiNINA> 1.8.13
|   |   |   |-- <SPI> 1.0
|   |   |-- <WiFiClientSecure> 1.0
|   |   |   |-- <WiFiNINA> 1.8.13
|   |   |   |   |-- <SPI> 1.0
|   |-- <WiFiNINA> 1.8.13
|   |   |-- <SPI> 1.0
|   |-- <WiFiClientSecure> 1.0
|   |   |-- <WiFiNINA> 1.8.13
|   |   |   |-- <SPI> 1.0
|   |-- <EEPROM> 1.0.3
|-- <WiFiNINA> 1.8.13
|   |-- <SPI> 1.0

This explains why my build for nano_33_iot was working, while esp32 was failing. The FlashStorage and WiFiNINA libraries were always built, no matter what library.json said!

In order to replicate exactly your scenario. I cloned the repo to a new folder (outside of the arduino libraries directory) . After that, I run the same commands I posted above. I get exactly the same behavior you have seen.
Ie. the nano_33_iot build fails, while esp32 and esp8266 build successfully

The problem here though, is that platformio does not recognize WiFiNINA and FlashStorage as dependencies.