How to properly include libraries in custom libraries?

mark.lamneck · April 3, 2024, 12:52pm

I am developing an open source library for ESP’s on Github. The library itself has libdeps to other libraries, in particular ESPAsyncWebserver. Now I am adding my library to a new project with the following line in plarformio.ini.

lib_deps = https://github.com/...

If I just add the ESPAsyncWebserver dependency to my library.json, I get some compile errors. I also have some compile errors for missing files like Wifi.h or EEprom.h and sometimes FS.h. At the moment it’s compiling with the following dependecies in my library.json.

  "dependencies":[
    { "name":"ESP8266WiFi", "platforms":"espressif8266" },
    { "name":"WiFi", "platforms":"espressif32" },
    { "name":"EEPROM", "platforms":["espressif32","espressif8266"] },
    { "name":"ESPAsyncTCP", "owner": "me-no-dev"},
    { "name":"ESP Async WebServer", "owner": "me-no-dev", "version" : "^1.2.3"},
  ]

The question is why do I have to add stuff like WiFi, EEprom (included in some files in my own library) to this list and what is the recommended way to add ESPAsyncWebserver with all chained dependencies. I also experienced that a working solution of the above dependency list did not work anymore when I changed something in my own code which has nothing to do with this at all. Maybe because of a different compilation order of cpp’s? Anyway I would highly appreciate some advise on how to properly do this.

sivar2311 · April 4, 2024, 4:11am

Libraries that exist in the Arduino core (WiFi, EEPROM etc) must not be specified!

Your library seems to be intended for both ESP32 and ESP8266.
Please make sure that you specify the correct platforms in your library.json:

  "platforms": [
    "espressif8266",
    "espressif32"
  ]

You should also specify the platforms correctly for the dependencies:

  "dependencies" : [
    { "name": "ESP Async WebServer", 
      "owner": "me-no-dev", 
      "version" : "^1.2.3",
      "platforms": [ "espressif32", "espressif8266" ]
    }
  ]

mark.lamneck · April 4, 2024, 10:07am

Thanks for the answer. I tried it again and if I remove WiFi as dependency it does not compile with the error ESP8266WiFi.h: No such file or directory. If it’s in the dependencies it works fine. However I tried to find a minimal example and it was working. Now I’ve added another dependency to a library from me which is dependency-free and I get the error again. I also find again that if I change code i.e. order of includes it changes the behavior. Sometimes it compiles, sometimes not and sometimes with different error messages.

platformio.ini:

[env:d1_mini_lite]
platform = espressif8266
board = d1_mini_lite
framework = arduino
lib_deps = https://github.com/mLamneck/MinimalPlatformIOdepIssue.git

main.cpp

#include "uTypedef.h"

void setup() {
}

void loop() {
  // put your main code here, to run repeatedly:
}

does not compile, because ESP8266WiFi.h is not found. The full project compiles at the moment (with everything in the dependency array WiFi, FS, EEprom,…) but I am afraid that it can break at any time because I don’t understand what’s happening. How can I debug this better?

sivar2311 · April 4, 2024, 11:51am

main.cpp

#include <Arduino.h>
#include <fileA.h>

void setup() {
    Serial.begin(115200);
    Serial.println("Hello World");
}

void loop() {
}

platformio.ini

; PlatformIO Project Configuration File
;
;   Build options: build flags, source filter
;   Upload options: custom upload port, speed and extra flags
;   Library options: dependencies, extra library storages
;   Advanced options: extra scripting
;
; Please visit documentation for the other options and examples
; https://docs.platformio.org/page/projectconf.html

[env]
lib_deps = 
  https://github.com/sivar2311/MinimalPlatformIOdepIssue

[env:esp12e]
platform = espressif8266
board = esp12e
framework = arduino

[env:esp32dev]
platform = espressif32
board = esp32dev
framework = arduino

Compiles and build fine for me!

Processing esp12e (platform: espressif8266; board: esp12e; framework: arduino)
...
Checking size .pio\build\esp12e\firmware.elf
Advanced Memory Usage is available via "PlatformIO Home > Project Inspect"
RAM:   [===       ]  34.3% (used 28088 bytes from 81920 bytes)
Flash: [===       ]  25.4% (used 265175 bytes from 1044464 bytes)
Building .pio\build\esp12e\firmware.bin
Creating BIN file ".pio\build\esp12e\firmware.bin" using "C:\Users\boris\.platformio\packages\framework-arduinoespressif8266\bootloaders\eboot\eboot.elf" and ".pio\build\esp12e\firmware.elf"
===================================================================================================================================================================================================== [SUCCESS] Took 10.49 seconds =====================================================================================================================================================================================================

Processing esp32dev (platform: espressif32; board: esp32dev; framework: arduino)
...
Checking size .pio\build\esp32dev\firmware.elf
Advanced Memory Usage is available via "PlatformIO Home > Project Inspect"
RAM:   [=         ]   6.5% (used 21408 bytes from 327680 bytes)
Flash: [==        ]  20.1% (used 263669 bytes from 1310720 bytes)
Building .pio\build\esp32dev\firmware.bin
esptool.py v4.5.1
Creating esp32 image...
Merged 2 ELF sections
Successfully created esp32 image.

Environment    Status    Duration
-------------  --------  ------------
esp12e         SUCCESS   00:00:10.488
esp32dev       SUCCESS   00:00:12.480
===================================================================================================================================================================================================== 2 succeeded in 00:00:22.968 =====================================================================================================================================================================================================

Maybe some functions should be implemented in “fileA.cpp” like setupWebServer() which get’s called by the main.cpp to be on the safe side.

Where does this come from?

mark.lamneck · April 4, 2024, 12:19pm

Yes if I do it like this, it works fine. But as soon as I include “uTypedef.h” which is coming from another lib from the dependency list and has nothing to do with WiFi at all, I get the error. Here is the library.json for the Minimal Example lib…

{
    "name": "Minimal Example",
    "description":"Minimal Example to show it's not compiling without Wifi in dependencies",
    "keywords":"",
    "authors":
    {
      "name": "Mark Lamneck",
      "maintainer": true
    },
 
    "repository":
    {
      "type": "git",
      "url": "toBeAdded"
    },
	"version": "0.0.0",
    "frameworks": "arduino",
    "platforms": ["espressif8266", "espressif32"],
    "dependencies":[
	      { "name":"ESP Async WebServer", "owner": "me-no-dev", "platforms": [ "espressif32", "espressif8266" ], "version" : "^1.2.3"},
        { "name":"SDDS", "version": "https://github.com/mLamneck/SDDS"}
    ]
}

In the Arduino IDE it works fine. And if I add all the dependencies like WiFi, EEprom, FS, … it is also working fine.

sivar2311 · April 4, 2024, 12:28pm

I’ve made a fork of your test librarary, added a setupWebServer which starts the webserver on port 80. See here: GitHub - sivar2311/MinimalPlatformIOdepIssue

I’ve also created a test project see here: GitHub - sivar2311/MinimalPlatformIOdepIssueTest which calls the setupWebServer() function from fileA.h / .cpp

Everything works as expected on ESP8266 and ESP32.

sivar2311 · April 4, 2024, 12:29pm

Ah ok… let me have a look at this…

mark.lamneck · April 4, 2024, 12:33pm

I get this warning with a verbose build

warning

And the following errors for I guess all cpp’s that include WiFi.

If have also checked the content of c_cpp_properties.json for both cases (with uTypedef not compiling and without uTypedef compiling) and it’s the same.

How can I further debug this?

sivar2311 · April 4, 2024, 12:39pm

There is some “kind” of issue with the ESPAsyncWebServer library.
If you have it in your lib_deps, you MUST include it in your source files!
Otherwise you’ll get errors like this, or “FS.h” not found and some more.

Try my example, it should compile for you without issues.

About the uTypesdef.h:
You should include it by using <> instead of "" : #include <uTypesdef.h>
BUT:
This header file is tricky!
It defines a macro “on” which collides with AsyncWebserver::on() function!
So, naming a macro “on” is not a good idea

mark.lamneck · April 4, 2024, 12:59pm

There is some “kind” of issue with the ESPAsyncWebServer library.
If you have it in your lib_deps , you MUST include it in your source files!
Otherwise you’ll get errors like this, or “FS.h” not found and some more.

What do you mean by it must be included in the source files? It is included in one of the source files (in the real library). Or do you mean it must be included in the main.cpp?

This header file is tricky!
It defines a macro “on ” which collides with AsyncWebserver::on() function!
So, naming a macro “on ” is not a good idea

Yes I know. I said the same about AsyncWebserver::on . No I don’t want to change the name of this macro, because it is a very neat way to specify event listeners. On the other hand I don’t understand why a macro can crash with a class member function, but it’s C++ right!? Anyway I came around the crash with the following wrapper around the webserver.

#pragma push_macro("on")

  #undef on
  #include <ESPAsyncWebServer.h>

  class TwebServer : public AsyncWebServer{
    public:
      using AsyncWebServer::AsyncWebServer;

      //wrapper around AsyncWebServers on method, because it crashes with out handy on macro...
      //route is a better name anyway
      void route(const char* uri, WebRequestMethodComposite method, ArRequestHandlerFunction onRequest){
        AsyncWebServer::on(uri,method,onRequest);
      }
  };

#pragma pop_macro("on")

At the moment my library compiles fine for ESP32 and ESP8266 with the following library.json.

{
    "name": "SDDS_ESP_Extension",
    "description":"extension for SDDS providing web features on ESP32 and ESP8266",
    "keywords":"",
    "authors":
    {
      "name": "Mark Lamneck",
      "maintainer": true
    },
 
    "repository":
    {
      "type": "git",
      "url": "https://github.com/mLamneck/SDDS_ESP_Extension"
    },
	"version": "0.0.0",
    "frameworks": "arduino",
    "platforms": ["espressif8266", "espressif32"],
    "dependencies":[
      { "name":"ESP8266WiFi", "platforms":"espressif8266" },
      { "name":"WiFi", "platforms":"espressif32" },
      { "name":"FS", "platforms":"espressif32" },
      { "name":"AsyncTCP", "owner": "me-no-dev"},
      { "name":"ESP Async WebServer", "owner": "me-no-dev", "version" : "^1.2.3"},
      { "name":"SDDS", "version": "https://github.com/mLamneck/SDDS"}
    ]
}

But for me it feels kind of buggy. Is there anything wrong with the way I did this? What is the recommended way to do it? I read the documentation about Library Management on platformIO and I think I did it accordingly but maybe I missed something?

sivar2311 · April 4, 2024, 1:08pm

Take this example:

platformio.ini:

[env:esp32dev]
platform = espressif32@6.5.0
board = esp32dev
framework = arduino
lib_deps = me-no-dev/ESP Async WebServer

main.cpp:

#include <Arduino.h>

void setup() {
}

void loop() {
}

Will result a bunch of errors caused by ESP Async WebServer:

In file included from .pio/libdeps/esp32dev/ESP Async WebServer/src/AsyncEventSource.h:29,
                 from .pio/libdeps/esp32dev/ESP Async WebServer/src/AsyncEventSource.cpp:21:
.pio/libdeps/esp32dev/ESP Async WebServer/src/ESPAsyncWebServer.h:27:10: fatal error: FS.h: No such file or directory

************************************************************
* Looking for FS.h dependency? Check our library registry!
*
* CLI  > platformio lib search "header:FS.h"
* Web  > https://registry.platformio.org/search?q=header:FS.h
*
************************************************************

 #include "FS.h"
          ^~~~~~
compilation terminated.
In file included from .pio/libdeps/esp32dev/ESP Async WebServer/src/SPIFFSEditor.h:3,
                 from .pio/libdeps/esp32dev/ESP Async WebServer/src/SPIFFSEditor.cpp:1:
.pio/libdeps/esp32dev/ESP Async WebServer/src/ESPAsyncWebServer.h:27:10: fatal error: FS.h: No such file or directory

************************************************************
* Looking for FS.h dependency? Check our library registry!
*
* CLI  > platformio lib search "header:FS.h"
* Web  > https://registry.platformio.org/search?q=header:FS.h
*
************************************************************

 #include "FS.h"
          ^~~~~~
compilation terminated.
*** [.pio\build\esp32dev\libbe6\ESP Async WebServer\AsyncEventSource.cpp.o] Error 1
*** [.pio\build\esp32dev\libbe6\ESP Async WebServer\SPIFFSEditor.cpp.o] Error 1
In file included from .pio/libdeps/esp32dev/ESP Async WebServer/src/AsyncWebSocket.h:32,
                 from .pio/libdeps/esp32dev/ESP Async WebServer/src/AsyncWebSocket.cpp:22:
.pio/libdeps/esp32dev/ESP Async WebServer/src/ESPAsyncWebServer.h:27:10: fatal error: FS.h: No such file or directory

************************************************************
* Looking for FS.h dependency? Check our library registry!
*
* CLI  > platformio lib search "header:FS.h"
* Web  > https://registry.platformio.org/search?q=header:FS.h
*
************************************************************

 #include "FS.h"
          ^~~~~~
compilation terminated.
*** [.pio\build\esp32dev\libbe6\ESP Async WebServer\AsyncWebSocket.cpp.o] Error 1
In file included from .pio/libdeps/esp32dev/ESP Async WebServer/src/WebRequest.cpp:21:
.pio/libdeps/esp32dev/ESP Async WebServer/src/ESPAsyncWebServer.h:27:10: fatal error: FS.h: No such file or directory

************************************************************
* Looking for FS.h dependency? Check our library registry!
*
* CLI  > platformio lib search "header:FS.h"
* Web  > https://registry.platformio.org/search?q=header:FS.h
*
************************************************************

 #include "FS.h"
          ^~~~~~
compilation terminated.
*** [.pio\build\esp32dev\libbe6\ESP Async WebServer\WebRequest.cpp.o] Error 1
In file included from .pio/libdeps/esp32dev/ESP Async WebServer/src/WebHandlers.cpp:21:
.pio/libdeps/esp32dev/ESP Async WebServer/src/ESPAsyncWebServer.h:27:10: fatal error: FS.h: No such file or directory

************************************************************
* Looking for FS.h dependency? Check our library registry!
*
* CLI  > platformio lib search "header:FS.h"
* Web  > https://registry.platformio.org/search?q=header:FS.h
*
************************************************************

 #include "FS.h"
          ^~~~~~
compilation terminated.
*** [.pio\build\esp32dev\libbe6\ESP Async WebServer\WebHandlers.cpp.o] Error 1
In file included from .pio/libdeps/esp32dev/ESP Async WebServer/src/WebResponses.cpp:21:
.pio/libdeps/esp32dev/ESP Async WebServer/src/ESPAsyncWebServer.h:27:10: fatal error: FS.h: No such file or directory

************************************************************
* Looking for FS.h dependency? Check our library registry!
*
* CLI  > platformio lib search "header:FS.h"
* Web  > https://registry.platformio.org/search?q=header:FS.h
*
************************************************************

 #include "FS.h"
          ^~~~~~
compilation terminated.
In file included from .pio/libdeps/esp32dev/ESP Async WebServer/src/WebServer.cpp:21:
.pio/libdeps/esp32dev/ESP Async WebServer/src/ESPAsyncWebServer.h:27:10: fatal error: FS.h: No such file or directory

************************************************************
* Looking for FS.h dependency? Check our library registry!
*
* CLI  > platformio lib search "header:FS.h"
* Web  > https://registry.platformio.org/search?q=header:FS.h
*
************************************************************

 #include "FS.h"
          ^~~~~~
compilation terminated.
*** [.pio\build\esp32dev\libbe6\ESP Async WebServer\WebResponses.cpp.o] Error 1
*** [.pio\build\esp32dev\libbe6\ESP Async WebServer\WebServer.cpp.o] Error 1
================================================================================================================================================================== [FAILED] Took 3.56 seconds ==================================================================================================================================================================

 *  The terminal process "C:\Users\boris\.platformio\penv\Scripts\platformio.exe 'run', '--environment', 'esp32dev'" terminated with exit code: 1. 
 *  Terminal will be reused by tasks, press any key to close it.

Whereby the same platformio.ini and this main.cpp compiles without issues:

#include <Arduino.h>
#include <ESPAsyncWebServer.h>
void setup() {
}

void loop() {
}

The only difference is #include <ESPAsyncWebServer>!

That’s what I meant by, if you have ESP AsyncWeb Server in your lib_deps you MUST include it - and the same goes for dependencies in your library.json!

Edit:

No, this can also happens in the source code of your library. But it must be somewhere included in the entire project!

Take a look at my example! It is not included in main.cpp but in fileA.h.

sivar2311 · April 4, 2024, 1:13pm

mark.lamneck:

"dependencies":[
      { "name":"ESP8266WiFi", "platforms":"espressif8266" },
      { "name":"WiFi", "platforms":"espressif32" },
      { "name":"FS", "platforms":"espressif32" },
      { "name":"AsyncTCP", "owner": "me-no-dev"},
      { "name":"ESP Async WebServer", "owner": "me-no-dev", "version" : "^1.2.3"},
      { "name":"SDDS", "version": "https://github.com/mLamneck/SDDS"}
    ]

Remove WiFi, FS and AsyncTCP!
WiFi and FS are core libraries.
AsyncTCP is a dependencie defined by ESPAsyncWebServer and handled by it’s own library.json.

mark.lamneck · April 4, 2024, 1:28pm

I know, but it’s not working without. That’s exactly my question and what makes me nervous. It doesn’t find the includes if I don’t specify it explicitly in the dependency list. But only in a more “complex” scenario, like I’ve posted. In the documentation of my library I have the following installation instruction for platformIO.

Add the GitHub link to this repository as a lib dependency to your platformio.ini file, as shown in the following example:

[env:myEspEnv]
platform = espressif32
board = esp32dev
framework = arduino
lib_deps = https://github.com/mLamneck/SDDS_ESP_Extension

and this should work reliable. At the moment it seems like, but because I don’t understand why I have to manually add AsyncTCP in the dependency list and all the other stuff I doubt that this is a stable solution. How can I further debug this issue?

sivar2311 · April 4, 2024, 1:33pm

No you don’t have to add AsyncTCP as dependencie as long as you #include <ESPAsyncWebServer.h> somewhere in your project.

That’s an issue caused by ESPAsyncWebServer library!

If you need ESPAsyncWebServer in your library you have to do 2 things:

put it in the dependencies
include ESPAsyncWebServer.h somewhere in your library’s code

But only doing step 1 only, without step 2 you’ll get a bunch of errors.

sivar2311 · April 4, 2024, 1:40pm

The dependencies are missing the platform attribute.
Also this library has issues on ESP32 as there might be no LED_BUILTIN defined (ESP32DEV doesn’t have LED_BUILTIN)

mark.lamneck · April 4, 2024, 1:40pm

Yes I understand, but I have both requirements fulfilled in the real lib. WebSpike.cpp includes WebSpike.h and this includes Webserver.h which in turn includes ESPAsyncWebServer.h. And as posted, I have it ESP Async WebServer in the dependency list. Anyway it doesn’t compile without specifying all the stuff.

sivar2311 · April 4, 2024, 1:42pm

Add the missing platform attribute.
See library.json of my fork from your fist example:

...
    "dependencies":[
	      { "name":"ESP Async WebServer", "owner": "me-no-dev", "platforms": [ "espressif32", "espressif8266" ], "version" : "^1.2.3"},
        { "name":"SDDS", "version": "https://github.com/mLamneck/SDDS"}
    ]
...

mark.lamneck · April 4, 2024, 1:47pm

I already added the platform information but it doesn’t help. And I’m quite sure that I’ve read in the documentation that this information is not evaluated anyway.

sivar2311 · April 4, 2024, 1:49pm

ESPAsyncWebServer has different dependencies to AsyncTCP for different platforms. See GitHub - me-no-dev/ESPAsyncWebServer: Async Web Server for ESP8266 and ESP32

For ESP8266 it requires ESPAsyncTCP To use this library you might need to have the latest git versions of ESP8266 Arduino Core

For ESP32 it requires AsyncTCP to work To use this library you might need to have the latest git versions of ESP32 Arduino Core

To make this all work automatically with just a single dependency in your library, it must know on what platform ESPAsyncWebServer is included.

Sorry, I don’t know how to explain this better.

Adding the platform attribute to your your dependencies should do the trick.
So the LDF is able to know on what platform which libraries are requried.

sivar2311 · April 4, 2024, 1:50pm

Try the example i made for your.
Does it work for you, or do you get the same issues?