How to integrate an esspressif library into an arduino framework project

lkyslevin · March 3, 2023, 9:03am

Hi, I have an ESP32 board with the following .ini file:

platform = espressif32
board = firebeetle32
framework = arduino

I know that the arduino frameworks provides access to many libraries from the espressif platform, however, not afaik to GitHub - espressif/esp-aws-iot: AWS IoT SDK for ESP32 based chipsets . I was wondering if it is possible to make use of the source code available in this repo within my project?

Since I do not have any experience in Cmake and esp-idf I do not know how to properly build the source/library content and link it to my arduino framework project.

Hope someone can provide me with some intel if this is possible and explain me the way of properly building/linking such “external” libraries.

Slev1n

maxgerhardt · March 3, 2023, 9:37am

Do an ESP-IDF + Arduino-ESP32 build, just just like

shows, then add the esp-aws-iot component, just

does and the documentation describes.

lkyslevin · March 3, 2023, 3:04pm

Thank you for your answer, but I am not completely sure what you want me to do.

I followed the README instructions from the blink example and I builded it successfully within a seperate pio project in an arduino framework - was this what you wanted me to do?

then add the esp-aws-iot component

Well I was not sure what you meant by that either but I checked the documentation and found the example I think you wanted me to find:

# The following lines of boilerplate have to be in your project's CMakeLists
# in this exact order for cmake to work correctly
cmake_minimum_required(VERSION 3.16)

include($ENV{IDF_PATH}/tools/cmake/project.cmake)
list(APPEND EXTRA_COMPONENT_DIRS esp-aws-iot)
project(subscribe_publish)

Since my project is in the arduino framework, there are no CMakeLists, so I added the espidf framework and I do have two frameworks now: framework = arduino, espidf

After that I copied the esp-aws-iot folder to my project following the instrcution from the documentation and added the lines from above into my CMakeLists.txt file in the src-folder:

Using EXTRA_COMPONENT_DIRS option in the root CMakeLists.txt file. This option represents a list of extra directories to search for components.

I tried building the blink example again but CMake is throwing some errors - here is the terminal output:

Warning! Arduino framework as an ESP-IDF component doesn’t handle the variant field! The default esp32 variant will be used.
Reading CMake configuration…
– Project is not inside a git repository, or git repository has no commits; will not use ‘git describe’ to determine PROJECT_VER.
– Building ESP-IDF components for target esp32
– Found Git: C:/Program Files/Git/cmd/git.exe (found version “2.29.2.windows.1”)
– Configuring incomplete, errors occurred!
See also “C:/0_GithubRepos/ESP-IDF/.pio/build/firebeetle32/CMakeFiles/CMakeOutput.log”.

fatal: not a git repository (or any of the parent directories): .git
CMake Error at .pio/build/firebeetle32/CMakeFiles/git-data/grabRef.cmake:48 (file):
file failed to open for reading (No such file or directory):
C:/0_GithubRepos/ESP-IDF/.pio/build/firebeetle32/CMakeFiles/git-data/head-ref
Call Stack (most recent call first):
C:/Users/wes39504/.platformio/packages/framework-espidf/tools/cmake/third_party/GetGitRevisionDescription.cmake:80 (include)
C:/Users/wes39504/.platformio/packages/framework-espidf/tools/cmake/third_party/GetGitRevisionDescription.cmake:90 (get_git_head_revision)
C:/Users/wes39504/.platformio/packages/framework-espidf/tools/cmake/project.cmake:59 (git_describe)
C:/Users/wes39504/.platformio/packages/framework-espidf/tools/cmake/project.cmake:380 (__project_get_revision)
CMakeLists.txt:3 (project)

CMake Error at C:/Users/wes39504/.platformio/packages/framework-espidf/tools/cmake/component.cmake:219 (message):
CMake Warning (dev) at
.pio/build/firebeetle32/build_properties.temp.cmake:8:
Syntax Warning in cmake code at column 51

Argument not separated from preceding token by whitespace.
Call Stack (most recent call first):
C:/Users/wes39504/.platformio/packages/framework-espidf/tools/cmake/scripts/component_get_requirements.cmake:3 (include)
This warning is for project developers. Use -Wno-dev to suppress it.

fatal: Needed a single revision

fatal: Needed a single revision

CMake Error at
C:/Users/wes39504/.platformio/packages/tool-cmake/share/cmake-3.16/Modules/ExternalProject.cmake:1029
(define_property):
define_property command is not scriptable
Call Stack (most recent call first):
C:/Users/wes39504/.platformio/packages/framework-espidf/tools/cmake/kconfig.cmake:1 (include)
C:/Users/wes39504/.platformio/packages/framework-espidf/tools/cmake/idf.cmake:41 (include)
C:/Users/wes39504/.platformio/packages/framework-espidf/tools/cmake/project.cmake:12 (include)
src/CMakeLists.txt:5 (include)
C:/Users/wes39504/.platformio/packages/framework-espidf/tools/cmake/scripts/component_get_requirements.cmake:106 (include)
C:/Users/wes39504/.platformio/packages/framework-espidf/tools/cmake/scripts/component_get_requirements.cmake:124 (__component_get_requirements)
Call Stack (most recent call first):
C:/Users/wes39504/.platformio/packages/framework-espidf/tools/cmake/build.cmake:488 (__component_get_requirements)
C:/Users/wes39504/.platformio/packages/framework-espidf/tools/cmake/project.cmake:384 (idf_build_process)
CMakeLists.txt:3 (project)

So I am quite sure this is not want you wanted me to do, but maybe you could elaborate what I could do to use the library in my current arduino framework based pio project.

maxgerhardt · March 4, 2023, 11:58am

I don’t get any such errors when I just created the proejct. See

lkyslevin · March 5, 2023, 3:43pm

Your example you compiles fine on my computer as well but I think you did not copy the aws-iot stuff into the blink example, did you?

I made a fresh start.

I created a pio project using the espidf-arduino-blink folder
I adapted the platformio.ini to

[env:firebeetle32]
platform = espressif32
board = firebeetle32
framework = arduino, espidf
build_flags =
       -D CONFIG_BLINK_GPIO=2
monitor_speed = 115200

I created a folder called components in the root directory of the project (same hierachy level as src or lib folder)
I copied the complete esp-aws-iot folder into the components folder
In root-folder’s!!! CMakeLists.txt file I replaced the project name with: project(arduino_with_aws) like in your example.
I compiled and it worked.
I added the function parseUnsignedInteger32Value() and #include "aws_iot_json_utils.h"in the header to see if I can actually access the libraries.
I compiled and it worked.

Thank you for your help @maxgerhardt . My goal was to understand how to incorporate further espidf libraries into my arduino framework and I think I can handle it now - will come back if I have further questions.

EDIT: If I want to add something like https://github.com/espressif/esp-aws-iot/tree/master/libraries to my project, could I simply add those libraries into my component folder, take on example from the repo’s example folder as main.cpp and use both plattforms (arduino, espidf) again?

tinkerphu · October 1, 2023, 8:13pm

Same Issue.

I am currently trying the same experiment but keep on failing.

I would like to keep my Arduino code and add a tiny esspressif part to it.
But once I change from framework = arduino to framework = arduino,espidf I can not compile anymore.
The compiler cannot find Serial.

I tried a completely new project with the PlatformIO wizard in VSCode, it has the same issue. (Actually, even there were a couple issues more in the sdkconfig: CONFIG_FREERTOS_HZ was set to 100 instead of the required 1000, and the board was set to esp32s2 instead of the given adafruit_qtpy_esp32s2.)

Even the example espidf-arduino-blink is not compiling on my machine. Same issue:

The compiler does not recognize Serial as an object. There is only Serial0 and Serial1.
I first tried with Serial0 but it is not what I need. Serial0 and Serial1 are HardwareSerials.
I believe, I need USBCDC Serial to have the same behavior as before with Arduino.
Serial can be found, if I add

	-D ARDUINO_USB_MODE=0
	-D ARDUINO_USB_CDC_ON_BOOT=1
	-D CONFIG_TINYUSB_ENABLED=1
	-D CONFIG_TINYUSB_CDC_ENABLED=1

to the build flags.

However, then the compiler cannot find tusb.h, which is included from USER/.platformio/packages/framework-arduinoespressif32/cores/esp32/esp32-hal-tinyusb.c:36
I found this file in USER.platformio\packages\framework-arduinoespressif32\tools\sdk\esp32s2\include\arduino_tinyusb\tinyusb\src
But I do not understand how to append it to the configuration. I was hoping there is a better way to fix this than adding lots of build-flags to include the files.

Anybody with experience to share?

my Platform.ini looks like this:

[env]
platform = espressif32
framework = arduino,espidf
build_flags = 
	-D ARDUINO_USB_MODE=0
	-D ARDUINO_USB_CDC_ON_BOOT=1
	-D CONFIG_TINYUSB_ENABLED=1
	-D CONFIG_TINYUSB_CDC_ENABLED=1

[env:adafruit_qtpy_esp32s2]
board = adafruit_qtpy_esp32s2

maxgerhardt · October 2, 2023, 8:57am

If you used the reference project, then that would have not happened, because its sdkconfig.defaults sets the required values.

Maybe you need this instead in your project’s components folder?

tinkerphu · October 7, 2023, 9:36pm

Thanks for the hint @maxgerhardt.
I finally got it worked out. This was quite a marathon and it got me quite alerted about the so called ‘reliable continuity’ of PlatformIO. Probably Espressif also contributed its fair share to this disaster.

So if you happen to work with ESP32-S2 (in particular adafruit_qtpy_esp32s2, exchange this string wherever you see it with your own variant) and want to work with Arduino framework in combination with Espressif library functions (framework = arduino, espidf), this is hopefully a helpful cookbook:

I started from an Arduino sketch. I assume, you could also go from PlatformIO/CreateProject.

The first thing that fails is CONFIG_FREERTOS_HZ=100, once corrected, the non-existing Serial is the big blocker. But also, the pin numbers are not correctly set by default…

open sdkconfig.adafruit_qtpy_esp32s2.
find CONFIG_FREERTOS_HZ=100,
correct it to CONFIG_FREERTOS_HZ=1000
(yes, it is set in the sdkconfig.default, but that is useless because defaults are not taken if it is already set wrongly in the variant version of sdkconfig).
in sdkconfig.adafruit_qtpy_esp32s2,
find CONFIG_ARDUINO_VARIANT="esp32s2" and
replace it with CONFIG_ARDUINO_VARIANT="adafruit_qtpy_esp32s2".
– IGNORE the build warnings:
Warning! Arduino framework as an ESP-IDF component doesn't handle the 'variant' field! The default 'esp32' variant will be used.
The warning is VERY CONFUSING / ANNOYING (did I forget something to get rid of this warning?).
Solve the Serial issue. Don’t get fooled by the compilers suggestion to use Serial1. Serial1 points to the RX/TX pins on the board but we want the usb serial port (USBCDC device).
a) Get the sources as @maxgerhardt suggested. Hopefully the link stays valid. No idea how to get to this link. (How did you know about it @maxgerhardt ?).
– Create a folder compontents on the root of your project, next to src, lib and include.
– Put the folder arduino_tinyusb into it. (PlatformIO auto-magically takes this folder into account for the build.)
b) Get more sources: get the code from Home - TinyUSB (direct link / zip )
– Create an additional folder tinyusb inside compontents/arduino_tinyusb (next to src and include)
– copy the folders hw and src from the downloaded zip into compontents/arduino_tinyusb/tinyusb
c) open sdkconfig.default
– add these lines (maybe only the first is really needed)
This should cause a change in sdkconfig.adafruit_qtpy_esp32s2 after compiling (CONFIG_TINYUSB=y should be inserted).

CONFIG_TINYUSB=y
CONFIG_TINYUSB_ENABLED=y
CONFIG_TINYUSB_CDC_ENABLED=y

open sdkconfig.default
Add CONFIG_AUTOSTART_ARDUINO=y

===
This should now enable you to compile.

I still haven’t figured out how to add the initialization of the variant to the project and have it called automatically:
USER\.platformio\packages\framework-arduinoespressif32\variants\adafruit_qtpy_esp32s2\variant.cpp
(This initialization is required for my board to have the WS2812 LED to be powered on.)

– Please drop me a line if you know.

Also, if you’re using the macro LED_BUILTIN as used by the Arduino Blink example sketch, it will not work.
Add Adafruit_NeoPixel library and manually set

  pinMode     (NEOPIXEL_POWER, OUTPUT);
  digitalWrite(NEOPIXEL_POWER, NEOPIXEL_POWER_ON);

and create an object

Adafruit_NeoPixel led(1, PIN_NEOPIXEL);

and use it as the library docs suggest.

Hope I can save you some work with this description. If you read some sarcasm or frustration between the lines - this issue spoiled me three week-ends while searching for a solution!
And thanks PlatformIO Team! I do realize that you are at war in Ukraine and that my problems are tiny compared to yours. Glory to Ukraine.

my platformio.ini:

[env:adafruit_qtpy_esp32s2]
platform = espressif32
board = adafruit_qtpy_esp32s2
framework = arduino, espidf
lib_deps =adafruit/Adafruit NeoPixel@^1.11.0