Using esp-idf library within the Arduino Framework (ESP32)

Just a disclaimer, I’m a novice in terms of C++ build systems and I’m trying to learn about how PlatformIO works in building libraries in VS Code for the ESP32.

My issue is that I have a library written for the esp-idf framework (https://github.com/espressif/esp-homekit-sdk) but I want to use the Arduino framework as base and not have Arduino as a component. Mainly because it’s easier for me to understand as I think the building systems are different and also because I would like one esp-idf component and many Arduino library so I think it just makes sense.

I’ve tried using the library.json file, adding lib_deps and lib_extra_dirs and various other techniques. My original method was to use git submodule to add the repo into the lib folder and add the components directory into lib_extra_dirs. This causes the library in question to have loads of internal build errors. Adding all the folders indivudally works but causes an unable to find reference issue (which I can recreate if needed).

I think the way to do it is to get cmake to run on esp_hap_core which should build everything, but that doesn’t work correctly when using the Arduino framework. When using the esp-idf framework, I found instructions on adding a components folder which made everything buuild correctly.

Does anyone have any resources, tutorials or advice on how to do this? I’d post code examples but I’ve tried several different solutions I thought it would be best to start from scratch.

Feel free to let me know if I’m going about this the wrong way. Does it makes sense to just use the esp-idf framework with Arduino libraries? Are they fundementally different? Does Arduino use cmake like esp-idf does with PlatformIO (which like I said I’m a novice at).

The base example for ESP-IDF + Arduino as a component is at platform-espressif32/examples/espidf-arduino-blink at develop · platformio/platform-espressif32 · GitHub. Maybe you can start from that and then add esp-homekit-sdk as a component, just as you say you haved done previously which worked.

Well if it’s a ESP-IDF library (/component) and you must absolutely use it, the best way would be to keep using the ESP-IDF build system. This also enables the intended configuration way of that component via menuconfig / KConfig files. Of course you can copy all the code files into a new library but then the configuration via the CONFIG_.. macros also has to be re-written somehow, generating additional work for you.

Indeed, ESP-IDF is based on the CMake build system as you’ve already said. The way the Arduino IDE builds the core can be considered its own build-system and is implemented in Java in the IDE, so it’s not using CMake. PlatformIO builds ESP-IDF projects by also calling in the exact same CMake build system (that’s why you still have to create CMakeLists.txt files etc although we have a platformio.ini config file). Arduino-ESP32 projects are built by using (Python) SCons build system, setup in such a way that the compilation process and option matches that of what the Arduino IDE would do.

However, the most important fact about the Arduino-ESP32 core in that regard is that “Arduino as a component of ESP-IDF” and “just the Arduino-ESP32 core” are the same thing semantically, just built a little bit differently. The Arduino-ESP32 core uses a precompiled ESP-IDF version as its entire base. They don’t re-implement the same peripheral drivers etc. from scratch, they just use ESP-IDF and their APIs. You can e.g. see the precompiled libraries in https://github.com/espressif/arduino-esp32/tree/master/tools/sdk/esp32/lib where each .a file is a static library which maps to one precompiled component from the ESP-IDF version the core is based on. Then the additional Arduino API layer is compiled on top of that, together with your sketch, and linked together with those precompiled libs. Any Arduino-ESP32 sketch can also call the APIs which stem from the used ESP-IDF core directly and this will work.

Of course it still makes sense to do what PIO does with the framework = espidf, arduino option, that is: Compiling ESP-IDF from source gives the user the ability to change the configuration of the ESP-IDF core. When using pure Arduino-ESP32, users are stuck with what the precompiled version has been configured for – alas this may be sufficient for most, but not in some special cases.

You just said that your library, esp-homekit-sdk, is an ESP-IDF library but you want to use the Arduino framework, did you switch the words here? Either way, as I’ve explained above, since the Arduino core is an extension of ESP-IDF, and Arduino libraries are an extension of Arduino, this will work. In that example project we’re demonstrating how to use the Arduino WiFi library with ESP-IDF as base and Arduino as a component.

Also, have you seen GitHub - Mixiaoxiao/Arduino-HomeKit-ESP32: [Deprecated] Native Apple HomeKit accessory implementation for the ESP32 Arduino core.? An Arduino library for the ESP32 claiming to implement HomeKit functionality. If you want to 100% stay in Arduino, you may want to try and use their APIs. Yes, the github says they’re “deprecated” because GitHub - espressif/esp-homekit-sdk exists, but still this an Arduino library and the other is a ESP-IDF component, so I think it has a right to exist separately.

2 Likes

Wow this was an amazing response thank you so much! This has helped me a lot in understand Arduino / C++ / IoT development!

Just to make sure I understand, having Arduino as a component is essentially the same as using Arduino except with a different build system and would allow for the same functionality.

So the solution would be to use Arduino as a component of ESP-IDF as that way it ensures the SDK is built properly from source (due to configuration files etc) which can then be accessed via my sketch? And the reason it isn’t working by simply adding the libraries is because of the fact the Arduino build system has no idea how to handle the structure of the SDK source? This would explain why when I managed to include all the directories in lib_extra_dirs it “built” but had issues with referencing due to lack of proper configuration that would normally happen during the build steps. These steps could technically be recreated in PlatformIO but that doesn’t seem worth it!

In terms of this, to make an “Arduino port” could I simply compile the SDK to get the static .a files and build an Arduino wrapper over those? Not that I would do this, just out of curiosity.

I have seen this but I thought it would be best to stick with the official SDK, I wanted wrapper to the “official” implementation. Also I wanted to try and learn something which I have!

Exactly.

If you want to use an Arduino sketch plus ESP-IDF from source and esp-homekit-sdk, yes. The latter should be added as a ESP-IDF component, and the Arduino sketch can freely access the APIs of the HomeKit SDK.

Treating esp-homekit-sdk as an Arduino library by doing a lib_extra_dirs won’t work since it’s an ESP-IDF component and requires a different build system (ESP-IDF’s), you’re right.

Exactly. As I’ve e.g. referenced the espidf-arduino-blink example which uses the special Arduino core branch https://github.com/espressif/arduino-esp32.git#idf-release/v4.0 (see platformio.ini), you can use the exact same sdkconfig that was used by the Arduino-ESP32 core people to generate their precompiled libraries (just to get close the original ESP-IDF config), additionally stick the esp-homekit-sdk in a pure ESP-IDF v4.0.1 project (see note regarding version) and configure it with menuconfig. Compilation of that project should then result in the lib...a files being generated together with your new libesp-homekit-sdk.a. Than that file can be copied standalone in a pure framework = arduino project, and using build_flags, you can add the header file paths of the new component with -I flags, the search path to the .a file with a -L flag and finally a -lesp-homekit-sdk to force linkage of the firmware with libesp-homekit-sdk.a. And that enables then to have that component in a precompiled form in a Arduino project, without having to compile the component or ESP-IDF from source.
EDIT: Once you have a precompiled file .a and the .h files from the SDK, actually you can create a proper Arduino library for it, using those files plus a library.json library definition file – then you can put this folder in lib/ of the project and use it just like that, without the need to do build_flags in the platformio.ini, since that can all be specified in the library.json. Would be pretty comfortable.

This is brilliant, It took me a few days of searching and trying things to find this solution. I have one question, in the the project root there is a file sdkconfig.defaults and I have tried setting CONFIG_AUTOSTART_ARDUINO to y and to 0 (because I want to use tasks) but neither changes the value in .pio\build\config\sdkconfig.h - I have looked and tried to work out how the files get generated but I just can’t find it.

(Just to confirm I am using VS Code + PlatformIO and targetting an az-delivery-devkit-v4 with the espidf-arduino-wifiscan example project)

I think the data from the sdkconfig.defaults is only loaded when you after that remove the sdkconfig file, forcing a regeneration of the file. But that will also cause you to lose other settings you’ve made to the sdkconfig file, so if that’s the case you should just call into menuconfig as documented and change and save the setting there. Beware of this caveat when doing so.

1 Like

@maxgerhardt When using framework = espidf, arduino it makes Arduino into a component, but it does not go into a folder called Component, right? Where does that mapping occur to tell the underlying build system that Arduino is not in a Component folder but instead in a Src folder. I ask this because I’m confused where to put other components. Do I make a Components folder in the main project folder? And will the build system be able to find that? I’m really just trying to understand where to add additional components.

Yes, that is what should be done.

Arduino as a component in the framework = espidf, arduino case is happening in

and

Thanks for the great answer! If someone has published a component on Github, is there a way to tell PIO in the .ini file to use it, or do I actually need to clone it into the Components folder?

This is the only way. lib_deps in the platformio.ini can’t be used because of missing features like #453.