OTA updates in ESP32, using VS Code, PlatformIO and Espressif 32 platform

Hello lovely people,

I’ve inherited some code that uses an earlier version of the ESP-AWS-IoT framework (release v3.1.x), at GitHub - espressif/esp-aws-iot at release/v3.1.x) for adding MQTT communications to an ESP32, allowing it to communicate with AWS over MQTT and provide a system with IoT-type functionality (remote changing of settings, reporting of data from the system, etc.).

That part all works well, so I didn’t want to change it, but now I need to add OTA functionality to that, with the intention of using the methods described on AWS, where a signed firmware update is deployed to a device registered as a thing on AWS, using the OTA Update Manager service (which creates an job on AWS that notifies the device that an update is available, then the OTA Agent on the ESP32 publishes requests to AWS IoT, to obtain updates using the MQTT protocol), as described in all of this stuff here: FreeRTOS Over-the-Air Updates - FreeRTOS

The earlier release of the ESP-AWS-IoT framework doesn’t seem to have any OTA parts to it, though, and while the latest version (here: GitHub - espressif/esp-aws-iot: AWS IoT SDK for ESP32 based chipsets) does have OTA (and looks like it might be able to do all of the things mentioned above, integrating with AWS jobs to allow OTA updates to be easily deployed), it is very different to the earlier version (all of the MQTT functions have changed, as well as many variables and structures being changed or introduced).

So I’m not sure whether I should translate the old, working code to use the functionality, variables, etc. of the new version, or try and take just the OTA parts from the new code, to add into the older version.

So far I have tried adding in some OTA/demo functionality from the new version, but it’s a slog, as the versions are so different, and the ESP32, AWS, VS Code, PlatformIO and all of that are still quite new to me. I can’t seem to import the code as one submodule (or similar) and get PlatformIO to locate the files (both for the IntelliSense-type functionality and when building), so that leads to a slow dismantling and rearranging of the files, which is taking forever.

I found an excellent translation of one of the esp-idf OTA examples by @maxgerhardt into code that works on PlatformIO, here: ESP32 OTA default examples don't work in PlatformIO but those are more about obtaining the firmware from a local server.

I wonder if there’s anyone out there who has been following developments in these repositories, and can give any ideas on how best to proceed with this?

Thank you for any help you can give.

1 Like

The main readme page for esp-aws-iot states it just needs ESP-IDF v4.3 as a base, which coincidentally is the latest support version (v.4.3.2 to be specific).

Have you tried starting with the blink example, then adding in all the files from esp-aws-iot/examples/ota/ota_mqtt at master · espressif/esp-aws-iot · GitHub, then copying every library from esp-aws-iot/libraries at master · espressif/esp-aws-iot · GitHub into the components/ folder of the PlatformIO project, like other example projects do?

Hi Max,

Thank you for the reply. While I’ve tried quite a few things, without much success, I haven’t tried that.

It’s good to see that the latest supported version in PlatformIO matches what is required by that repository. I’ll update, and give what you suggest a try.

There are a few pifalls like hardcoded paths that you would need to change when copying them into the PlatformIO project (/../../../libraries to /components)

and you have to inform PlatformIO about a changed partition table that they set here

as documented here.

Same goes for the embedded files documented right below that, with an example here.

Thank you for the extra tips. Having a bad day here - ran a cleanup command suggested by PlatformIO, and it seemed to break the whole thing. I’ll have another go once I’ve tried reinstalling VS Code and PlatformIO for the third time!

Probably a pio system prune command, or which one exactly? What did it break afterwards? (After removing some packages, it is normal that one gets e.g. intellisense errors, you have to build the project again to redownload these).

I’ve done the project conversion and some surface-level testing at GitHub - maxgerhardt/pio-espidf-aws-mqtt-ota: PlatformIO compilable version of https://github.com/espressif/esp-aws-iot/tree/master/examples/ota/ota_mqtt.

The conversion was straightforward as described above except for the embedded files inclusion. The logic had to be moved from CMakeLists.txt to main/CMakeLists.txt with a changed first argument and after the idf_component_register() call to work, as also noted in How to embed binary files into esp32 firmware - #29 by mikemoy. Further the initial compilation didn’t start at all because I somehow had to apply the fix as shown in Use empty ASM and CXX flags if config empty by maxgerhardt · Pull Request #823 · platformio/platform-espressif32 · GitHub.

If you just download the repo and open it in VSCode, it should be compilable + uploadable. Of course, you need to configure it in accordance to the README to get any further.

Yes, it was pio system prune. I wish that hadn’t been suggested! It seemed to get rid of the build command, and building existing projects also seemed to be broken. I’m still trying to get back to my original setup after reinstall.

Thank you very much for the extra work to convert the code to something that PlatformIO is happy with! You’re a star! I don’t think I would have been able to work out those issues, so those are great clues for how to import these kinds of code chunks if I need to do any more.

I shall give it a try once I get VS Code and PlatformIO back to a working state.

If you have done no local modifications in <home folder>/.platformio (to e.g. framework files, or platform scripts etc.), the easiest and cleanest way to reinstall VSCode is to uninstall the PlatformIO VSCode extension, then remove <home folder>/.platformio completely, reopen VSCode and reinstall the PlatformIO extension until completion, then restart VSCode again.

Ah, thank you for that tip. I found that the reinstall didn’t work properly unless I deleted the .platformio directory. I also deleted the .vscode directory in the same location, for good measure, before reinstalling VS Code, PlatformIO and the latest Espressif 32 platform (seems to be giving me 4.3.0 - hopefully that is sufficient).

I am back in action and have the project building successfully. Thank you again for your excellent help.

I’m not sure I understand the fix that was applied, or the response you got from valeros (at Use empty ASM and CXX flags if config empty by maxgerhardt · Pull Request #823 · platformio/platform-espressif32 · GitHub), but I’m very glad you had some magic up your sleeve for that!

I’ve updated the repo with a fix as valeros indicated – due to how PlatformIO uses the ESP-IDF build system doing a set() on the EXTRA_COMPONENT_DIRS variable is not good, since it overwrites the special components PlatformIO has injected to get some build information. Sadly all official ESP-IDF examples I’ve seen use set() instead of the needed list(APPEND ..) here, but now there is at least a proper error message and a link to the documentation if PlatformIO encounters that situation.

So I can only recommend now to not use my special fork, but the official stable on with the above applied fix, as I’ve done above.

Thank you very much for the update and the notes.

Though it builds successfully, I do seem to have an issue when building that I immediately get a set of error messages, marked in red (part of the path removed here, replaced with ... ):

[WinError 3] The system cannot find the path specified: 'C:\\Users\\ ... \\pio-espidf-aws-mqtt-ota\\.pio\\build\\esp32dev\\esp-idf\\Device-Defender-for-AWS-IoT-embedded-sdk\\CMakeFiles\\__idf_Device-Defender-for-AWS-IoT-embedded-sdk.dir\\Device-Defender-for-AWS-IoT-embedded-sdk' 
Please manually remove the file `C:\Users\ ... \pio-espidf-aws-mqtt-ota\.pio\build\esp32dev\esp-idf\Device-Defender-for-AWS-IoT-embedded-sdk\CMakeFiles\__idf_Device-Defender-for-AWS-IoT-embedded-sdk.dir\Device-Defender-for-AWS-IoT-embedded-sdk`
[WinError 3] The system cannot find the path specified: 'C:\\Users\\ ... \\pio-espidf-aws-mqtt-ota\\.pio\\build\\esp32dev\\esp-idf\\Device-Defender-for-AWS-IoT-embedded-sdk\\CMakeFiles\\__idf_Device-Defender-for-AWS-IoT-embedded-sdk.dir\\Device-Defender-for-AWS-IoT-embedded-sdk' 
Please manually remove the file `C:\Users\ ... \pio-espidf-aws-mqtt-ota\.pio\build\esp32dev\esp-idf\Device-Defender-for-AWS-IoT-embedded-sdk\CMakeFiles\__idf_Device-Defender-for-AWS-IoT-embedded-sdk.dir\Device-Defender-for-AWS-IoT-embedded-sdk`
[WinError 145] The directory is not empty: 'C:\\Users\\ ... \\pio-espidf-aws-mqtt-ota\\.pio\\build\\esp32dev\\esp-idf\\Device-Defender-for-AWS-IoT-embedded-sdk\\CMakeFiles\\__idf_Device-Defender-for-AWS-IoT-embedded-sdk.dir' 
Please manually remove the file `C:\Users\ ... \pio-espidf-aws-mqtt-ota\.pio\build\esp32dev\esp-idf\Device-Defender-for-AWS-IoT-embedded-sdk\CMakeFiles\__idf_Device-Defender-for-AWS-IoT-embedded-sdk.dir`
[WinError 145] The directory is not empty: 'C:\\Users\\ ... \\pio-espidf-aws-mqtt-ota\\.pio\\build\\esp32dev\\esp-idf\\Device-Defender-for-AWS-IoT-embedded-sdk\\CMakeFiles' 
Please manually remove the file `C:\Users\ ... \pio-espidf-aws-mqtt-ota\.pio\build\esp32dev\esp-idf\Device-Defender-for-AWS-IoT-embedded-sdk\CMakeFiles`
[WinError 145] The directory is not empty: 'C:\\Users\\ ... \\pio-espidf-aws-mqtt-ota\\.pio\\build\\esp32dev\\esp-idf\\Device-Defender-for-AWS-IoT-embedded-sdk'
Please manually remove the file `C:\Users\ ... \pio-espidf-aws-mqtt-ota\.pio\build\esp32dev\esp-idf\Device-Defender-for-AWS-IoT-embedded-sdk`
[WinError 3] The system cannot find the path specified: 'C:\\Users\\ ... \\pio-espidf-aws-mqtt-ota\\.pio\\build\\esp32dev\\esp-idf\\Device-Shadow-for-AWS-IoT-embedded-sdk\\CMakeFiles\\__idf_Device-Shadow-for-AWS-IoT-embedded-sdk.dir\\Device-Shadow-for-AWS-IoT-embedded-sdk\\source'
Please manually remove the file `C:\Users\ ... \pio-espidf-aws-mqtt-ota\.pio\build\esp32dev\esp-idf\Device-Shadow-for-AWS-IoT-embedded-sdk\CMakeFiles\__idf_Device-Shadow-for-AWS-IoT-embedded-sdk.dir\Device-Shadow-for-AWS-IoT-embedded-sdk\source`
[WinError 3] The system cannot find the path specified: 'C:\\Users\\ ... \\pio-espidf-aws-mqtt-ota\\.pio\\build\\esp32dev\\esp-idf\\Device-Shadow-for-AWS-IoT-embedded-sdk\\CMakeFiles\\__idf_Device-Shadow-for-AWS-IoT-embedded-sdk.dir\\Device-Shadow-for-AWS-IoT-embedded-sdk\\source'
Please manually remove the file `C:\Users\ ... \pio-espidf-aws-mqtt-ota\.pio\build\esp32dev\esp-idf\Device-Shadow-for-AWS-IoT-embedded-sdk\CMakeFiles\__idf_Device-Shadow-for-AWS-IoT-embedded-sdk.dir\Device-Shadow-for-AWS-IoT-embedded-sdk\source`
[WinError 145] The directory is not empty: 'C:\\Users\\ ... \\pio-espidf-aws-mqtt-ota\\.pio\\build\\esp32dev\\esp-idf\\Device-Shadow-for-AWS-IoT-embedded-sdk\\CMakeFiles\\__idf_Device-Shadow-for-AWS-IoT-embedded-sdk.dir\\Device-Shadow-for-AWS-IoT-embedded-sdk'
Please manually remove the file `C:\Users\ ... \pio-espidf-aws-mqtt-ota\.pio\build\esp32dev\esp-idf\Device-Shadow-for-AWS-IoT-embedded-sdk\CMakeFiles\__idf_Device-Shadow-for-AWS-IoT-embedded-sdk.dir\Device-Shadow-for-AWS-IoT-embedded-sdk`
[WinError 145] The directory is not empty: 'C:\\Users\\ ... \\pio-espidf-aws-mqtt-ota\\.pio\\build\\esp32dev\\esp-idf\\Device-Shadow-for-AWS-IoT-embedded-sdk\\CMakeFiles\\__idf_Device-Shadow-for-AWS-IoT-embedded-sdk.dir'
Please manually remove the file `C:\Users\ ... \pio-espidf-aws-mqtt-ota\.pio\build\esp32dev\esp-idf\Device-Shadow-for-AWS-IoT-embedded-sdk\CMakeFiles\__idf_Device-Shadow-for-AWS-IoT-embedded-sdk.dir`
[WinError 145] The directory is not empty: 'C:\\Users\\ ... \\pio-espidf-aws-mqtt-ota\\.pio\\build\\esp32dev\\esp-idf\\Device-Shadow-for-AWS-IoT-embedded-sdk\\CMakeFiles'
Please manually remove the file `C:\Users\ ... \pio-espidf-aws-mqtt-ota\.pio\build\esp32dev\esp-idf\Device-Shadow-for-AWS-IoT-embedded-sdk\CMakeFiles`
[WinError 145] The directory is not empty: 'C:\\Users\\ ... \\pio-espidf-aws-mqtt-ota\\.pio\\build\\esp32dev\\esp-idf\\Device-Shadow-for-AWS-IoT-embedded-sdk'
Please manually remove the file `C:\Users\ ... \pio-espidf-aws-mqtt-ota\.pio\build\esp32dev\esp-idf\Device-Shadow-for-AWS-IoT-embedded-sdk`
[WinError 3] The system cannot find the path specified: 'C:\\Users\\ ... \\pio-espidf-aws-mqtt-ota\\.pio\\build\\esp32dev\\esp-idf\\ota-for-aws-iot-embedded-sdk\\CMakeFiles\\__idf_ota-for-aws-iot-embedded-sdk.dir\\ota-for-aws-iot-embedded-sdk\\source\\dependency\\3rdparty\\tinycbor'
Please manually remove the file `C:\Users\ ... \pio-espidf-aws-mqtt-ota\.pio\build\esp32dev\esp-idf\ota-for-aws-iot-embedded-sdk\CMakeFiles\__idf_ota-for-aws-iot-embedded-sdk.dir\ota-for-aws-iot-embedded-sdk\source\dependency\3rdparty\tinycbor`
[WinError 3] The system cannot find the path specified: 'C:\\Users\\ ... \\pio-espidf-aws-mqtt-ota\\.pio\\build\\esp32dev\\esp-idf\\ota-for-aws-iot-embedded-sdk\\CMakeFiles\\__idf_ota-for-aws-iot-embedded-sdk.dir\\ota-for-aws-iot-embedded-sdk\\source\\dependency\\3rdparty\\tinycbor'
Please manually remove the file `C:\Users\ ... \pio-espidf-aws-mqtt-ota\.pio\build\esp32dev\esp-idf\ota-for-aws-iot-embedded-sdk\CMakeFiles\__idf_ota-for-aws-iot-embedded-sdk.dir\ota-for-aws-iot-embedded-sdk\source\dependency\3rdparty\tinycbor`
[WinError 145] The directory is not empty: 'C:\\Users\\ ... \\pio-espidf-aws-mqtt-ota\\.pio\\build\\esp32dev\\esp-idf\\ota-for-aws-iot-embedded-sdk\\CMakeFiles\\__idf_ota-for-aws-iot-embedded-sdk.dir\\ota-for-aws-iot-embedded-sdk\\source\\dependency\\3rdparty'
Please manually remove the file `C:\Users\ ... \pio-espidf-aws-mqtt-ota\.pio\build\esp32dev\esp-idf\ota-for-aws-iot-embedded-sdk\CMakeFiles\__idf_ota-for-aws-iot-embedded-sdk.dir\ota-for-aws-iot-embedded-sdk\source\dependency\3rdparty`
[WinError 145] The directory is not empty: 'C:\\Users\\ ... \\pio-espidf-aws-mqtt-ota\\.pio\\build\\esp32dev\\esp-idf\\ota-for-aws-iot-embedded-sdk\\CMakeFiles\\__idf_ota-for-aws-iot-embedded-sdk.dir\\ota-for-aws-iot-embedded-sdk\\source\\dependency'
Please manually remove the file `C:\Users\ ... \pio-espidf-aws-mqtt-ota\.pio\build\esp32dev\esp-idf\ota-for-aws-iot-embedded-sdk\CMakeFiles\__idf_ota-for-aws-iot-embedded-sdk.dir\ota-for-aws-iot-embedded-sdk\source\dependency`
[WinError 145] The directory is not empty: 'C:\\Users\\ ... \\pio-espidf-aws-mqtt-ota\\.pio\\build\\esp32dev\\esp-idf\\ota-for-aws-iot-embedded-sdk\\CMakeFiles\\__idf_ota-for-aws-iot-embedded-sdk.dir\\ota-for-aws-iot-embedded-sdk\\source'
Please manually remove the file `C:\Users\ ... \pio-espidf-aws-mqtt-ota\.pio\build\esp32dev\esp-idf\ota-for-aws-iot-embedded-sdk\CMakeFiles\__idf_ota-for-aws-iot-embedded-sdk.dir\ota-for-aws-iot-embedded-sdk\source`
[WinError 145] The directory is not empty: 'C:\\Users\\ ... \\pio-espidf-aws-mqtt-ota\\.pio\\build\\esp32dev\\esp-idf\\ota-for-aws-iot-embedded-sdk\\CMakeFiles\\__idf_ota-for-aws-iot-embedded-sdk.dir\\ota-for-aws-iot-embedded-sdk'
Please manually remove the file `C:\Users\ ... \pio-espidf-aws-mqtt-ota\.pio\build\esp32dev\esp-idf\ota-for-aws-iot-embedded-sdk\CMakeFiles\__idf_ota-for-aws-iot-embedded-sdk.dir\ota-for-aws-iot-embedded-sdk`
[WinError 145] The directory is not empty: 'C:\\Users\\ ... \\pio-espidf-aws-mqtt-ota\\.pio\\build\\esp32dev\\esp-idf\\ota-for-aws-iot-embedded-sdk\\CMakeFiles\\__idf_ota-for-aws-iot-embedded-sdk.dir'
Please manually remove the file `C:\Users\ ... \pio-espidf-aws-mqtt-ota\.pio\build\esp32dev\esp-idf\ota-for-aws-iot-embedded-sdk\CMakeFiles\__idf_ota-for-aws-iot-embedded-sdk.dir`
[WinError 145] The directory is not empty: 'C:\\Users\\ ... \\pio-espidf-aws-mqtt-ota\\.pio\\build\\esp32dev\\esp-idf\\ota-for-aws-iot-embedded-sdk\\CMakeFiles'
Please manually remove the file `C:\Users\ ... \pio-espidf-aws-mqtt-ota\.pio\build\esp32dev\esp-idf\ota-for-aws-iot-embedded-sdk\CMakeFiles`
[WinError 145] The directory is not empty: 'C:\\Users\\ ... \\pio-espidf-aws-mqtt-ota\\.pio\\build\\esp32dev\\esp-idf\\ota-for-aws-iot-embedded-sdk'
Please manually remove the file `C:\Users\ ... \pio-espidf-aws-mqtt-ota\.pio\build\esp32dev\esp-idf\ota-for-aws-iot-embedded-sdk`        
[WinError 145] The directory is not empty: 'C:\\Users\\ ... \\pio-espidf-aws-mqtt-ota\\.pio\\build\\esp32dev\\esp-idf' 
Please manually remove the file `C:\Users\ ... \pio-espidf-aws-mqtt-ota\.pio\build\esp32dev\esp-idf`
[WinError 145] The directory is not empty: 'C:\\Users\\ ... \\pio-espidf-aws-mqtt-ota\\.pio\\build\\esp32dev' 
Please manually remove the file `C:\Users\ ... \pio-espidf-aws-mqtt-ota\.pio\build\esp32dev`
[WinError 145] The directory is not empty: 'C:\\Users\\ ... \\pio-espidf-aws-mqtt-ota\\.pio\\build' 
Please manually remove the file `C:\Users\ ... \pio-espidf-aws-mqtt-ota\.pio\build`

I’m not sure if that’s specific to the paths on my PC, or whether it is a problem generally?

ESP-IDF and CMake generate absurdly long paths, you might be running into Windows MAX_PATH restrictions on path lengths. Please use the “Enable Win32 Long Paths through Local Group Policy Editor” method at https://www.thewindowsclub.com/how-to-enable-or-disable-win32-long-paths-in-windows-11-10 to try and counteract it by activating Win32 Long Paths. For the setting to apply, you may need to reboot and do a clean build (remove the .pio folder of the project).

Thank you. That worked, and removed the error messages, seemingly without even the need for a reboot.

1 Like