PioArduino's hybrid compile fails due to missing file

hmbacher · October 3, 2025, 11:19am

Hi folks!
I have been working with ESP32SvelteKit - now based on pioarduino’s espressi32 platform and arduiino framework - for a couple of months, until I came to a point where the precompiled libraries where not suffiecient as I had to increase the max HTTP header size from 1024 to 2048.

I remembered pioarduino’s awsome hybrid compile feature and tried to get the header size increased by setting CONFIG_HTTPD_MAX_REQ_HDR_LEN to 2048. I did the following changes to the ESP32Sveltekit’s original platformio.ini:

Added custom_sdkconfig param
Added board_build.embed_txtfiles param

Final platformio.ini

[platformio]
description = Genius Gateway - A gateway for Hekatron Genius Plus X smoke detectors
data_dir = data
extra_configs = 
	factory_settings.ini
	features.ini
    config.ini
default_envs = seeed-xiao-esp32s3

[env]
framework = arduino, espidf
platform = https://github.com/pioarduino/platform-espressif32/releases/download/55.03.31/platform-espressif32.zip
build_flags = 
	${factory_settings.build_flags}
	${features.build_flags}
    ${config.build_flags}
    -D BUILD_TARGET=\"$PIOENV\"
    -D APP_NAME=\"Genius-Gateway\"
    -D APP_VERSION=\"0.2.3\"
    -D ESP32SVELTEKIT_RUNNING_CORE=0
    -D EMBED_WWW
    -D SERIAL_INFO
    -D CONFIG_ARDUHAL_LOG_COLORS
	-D CORE_DEBUG_LEVEL=3

custom_sdkconfig = CONFIG_HTTPD_MAX_REQ_HDR_LEN=2048 ; original 1024
    
lib_compat_mode = strict

board_build.embed_files = src/certs/x509_crt_bundle.bin

board_build.embed_txtfiles =
    managed_components/espressif__esp_insights/server_certs/https_server.crt
    managed_components/espressif__esp_rainmaker/server_certs/rmaker_mqtt_server.crt
    managed_components/espressif__esp_rainmaker/server_certs/rmaker_claim_service_server.crt
    managed_components/espressif__esp_rainmaker/server_certs/rmaker_ota_server.crt

board_ssl_cert_source = mozilla

monitor_speed = 115200
monitor_filters = 
	esp32_exception_decoder
    log2file
board_build.filesystem = littlefs
extra_scripts = 
    pre:scripts/generate_enums.py
    pre:scripts/build_interface.py
    pre:scripts/generate_cert_bundle.py
    scripts/merge_bin.py
    scripts/rename_fw.py
    scripts/save_elf.py
lib_deps = 
	ArduinoJson@>=7.0.0
    elims/PsychicMqttClient@^0.2.3

[env:seeed-xiao-esp32s3]
board = seeed_xiao_esp32s3 ; 8 MB flash, 8 MB PSRAM

When building the project with these changes from scratch, I observed quite a undeterministic behaviour. I get frontend build errors, file not found errors, etc. Every time I just restarted the build, it’s outcome / fail reason slightly changed. Finally, I was left with the following build output:

Terminal output

 *  Executing task: C:\Users\hmbac\.platformio\penv\Scripts\platformio.exe run 

Processing seeed-xiao-esp32s3 (board: seeed_xiao_esp32s3; framework: arduino; platform: https://github.com/pioarduino/platform-espressif32/releases/download/55.03.31/platform-espressif32.zip)
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Verbose mode can be enabled via `-v, --verbose` option

running: generate_enums.py
==================================================
Generating TypeScript enums from C++ headers...
==================================================
Processing AlarmBlocker.h...
  No enums found
Processing AlarmLinesService.h...
  ✓ alarm_line_acquisition → AlarmLineAcquisition
Processing cc1101.h...
  ✓ cc1101_mode → Cc1101Mode
Processing CC1101Controller.h...
  No enums found
Processing GatewayDevicesService.h...
  ✓ genius_alarm_ending → GeniusAlarmEnding
  ✓ genius_smoke_detector → GeniusSmokeDetector
  ✓ genius_radio_module → GeniusRadioModule
  ✓ genius_device_registration → GeniusDeviceRegistration
Processing GatewayMqttSettingsService.h...
  No enums found
Processing GatewaySettingsService.h...
  No enums found
Processing GeniusGateway.h...
  ✓ genius_packet_type → GeniusPacketType
Processing ThreadSafeService.h...
  No enums found
Processing VisualizerSettingsService.h...
  No enums found
Processing WSLogger.h...
  No enums found
Processing WSLoggerSettingsService.h...
  No enums found

------------------------------------------------------------
Generated 7 TypeScript enum(s) in C:\dev\esp32\platformio\genius-gateway\interface\src\lib\types\enums.ts

Enum Mapping Summary:
C++ Enum                  → TypeScript Enum      Source File
-------------------------   -------------------- ------------------------------
alarm_line_acquisition    → AlarmLineAcquisition AlarmLinesService.h
cc1101_mode               → Cc1101Mode           cc1101.h
genius_alarm_ending       → GeniusAlarmEnding    GatewayDevicesService.h
genius_smoke_detector     → GeniusSmokeDetector  GatewayDevicesService.h
genius_radio_module       → GeniusRadioModule    GatewayDevicesService.h
genius_device_registration → GeniusDeviceRegistration GatewayDevicesService.h
genius_packet_type        → GeniusPacketType     GeniusGateway.h

✓ TypeScript enums generated successfully

running: build_interface.py
Newest file: 2025-10-03 12:52:19.471231, output file: 2025-10-03 12:51:02.957521
Building interface with npm
npm install

up to date, audited 256 packages in 2s

62 packages are looking for funding
  run `npm fund` for details

7 vulnerabilities (3 low, 4 moderate)

To address all issues (including breaking changes), run:
  npm audit fix --force

Run `npm audit` for details.
npm run build

> genius-gateway@0.2.0 build
> vite build

12:52:25 [vite-plugin-svelte] WARNING: The following packages have a svelte field in their package.json but no exports condition for svelte.

svelte-focus-trap@1.2.0

Please see https://github.com/sveltejs/vite-plugin-svelte/blob/main/docs/faq.md#missing-exports-condition for details.
vite v5.4.20 building SSR bundle for production...
transforming...
/*! 🌼 daisyUI 5.0.46 */
✓ 397 modules transformed.
rendering chunks...
12:52:31 [vite-plugin-svelte] WARNING: The following packages have a svelte field in their package.json but no exports condition for svelte.

svelte-focus-trap@1.2.0

Please see https://github.com/sveltejs/vite-plugin-svelte/blob/main/docs/faq.md#missing-exports-condition for details.
vite v5.4.20 building for production...
transforming...
✓ 505 modules transformed.
rendering chunks...
computing gzip size...
.svelte-kit/output/client/_app/version.json                            0.03 kB │ gzip:  0.05 kB

... (removed some output here) ...

.svelte-kit/output/server/index.js                                                 124.42 kB
✓ built in 11.89s

Run npm run preview to preview your production build locally.

> Using @sveltejs/adapter-static
  Wrote site to "build"
  ✔ done
Converting interface to PROGMEM
Converting favicon.png

... (removed some output here) ...

Converting _app/immutable/nodes/9.js
SSL Cert Store: Certificate bundle downloaded to: ssl_certs\cacert.pem
SSL Cert Store: Parsing certificates from ssl_certs\cacert.pem
C:\dev\esp32\platformio\genius-gateway\scripts\generate_cert_bundle.py:131: CryptographyDeprecationWarning: Parsed a serial number which wasn't positive (i.e., it was negative or zero), which is disallowed by RFC 5280. Loading this certificate will cause an exception in a future release of cryptography.
  self.certificates.append(x509.load_pem_x509_certificate(crt.encode(), default_backend()))
SSL Cert Store: Successfully added 146 certificates
SSL Cert Store: Successfully added 146 certificates in total
SSL Cert Store: Successfully created src\certs\x509_crt_bundle.bin
CONFIGURATION: https://docs.platformio.org/page/boards/espressif32/seeed_xiao_esp32s3.html
PLATFORM: Espressif 32 (55.3.31) > Seeed Studio XIAO ESP32S3
HARDWARE: ESP32S3 240MHz, 320KB RAM, 8MB Flash
DEBUG: Current (cmsis-dap) External (cmsis-dap, esp-bridge, esp-builtin, esp-prog, iot-bus-jtag, jlink, minimodule, olimex-arm-usb-ocd, olimex-arm-usb-ocd-h, olimex-arm-usb-tiny-h, olimex-jtag-tiny, tumpa)
PACKAGES:
 - contrib-piohome @ 3.4.4
 - framework-arduinoespressif32 @ 3.3.0
 - framework-arduinoespressif32-libs @ 5.5.0+sha.129cd0d247
 - framework-espidf @ 3.50501.0 (5.5.1)
 - tool-cmake @ 4.0.3
 - tool-esp-rom-elfs @ 2024.10.11
 - tool-esptoolpy @ 5.1.0
 - tool-mklittlefs @ 3.2.0
 - tool-ninja @ 1.13.1
 - tool-scons @ 4.40801.0 (4.8.1)
 - toolchain-xtensa-esp-elf @ 14.2.0+20241119
*** Compile Arduino IDF libs for seeed-xiao-esp32s3 ***
*** Add "custom_sdkconfig" settings to IDF sdkconfig.defaults ***
Replace: CONFIG_ESPTOOLPY_FLASHMODE_QIO=y with: CONFIG_ESPTOOLPY_FLASHMODE_QIO=y
Replace: CONFIG_HTTPD_MAX_REQ_HDR_LEN=1024 with: CONFIG_HTTPD_MAX_REQ_HDR_LEN=2048
Reading CMake configuration...
Warning! Flash memory size mismatch detected. Expected 8MB, found 16MB!
Please select a proper value in your `sdkconfig.defaults` or via the `menuconfig` target!
LDF: Library Dependency Finder -> https://bit.ly/configure-pio-ldf
LDF Modes: Finder ~ chain, Compatibility ~ strict
Found 47 compatible libraries
Scanning dependencies...
Dependency Graph
|-- ArduinoJson @ 7.4.2
|-- PsychicMqttClient @ 0.2.3
Building in release mode
*** [.pio\build\seeed-xiao-esp32s3\.pio\build\seeed-xiao-esp32s3\https_server.crt.S.o] Source `.pio\build\seeed-xiao-esp32s3\https_server.crt.S' not found, needed by target `.pio\build\seeed-xiao-esp32s3\.pio\build\seeed-xiao-esp32s3\https_server.crt.S.o'.
================================================================================== [FAILED] Took 78.02 seconds ==================================================================================

Environment         Status    Duration
------------------  --------  ------------
seeed-xiao-esp32s3  FAILED    00:01:18.020
============================================================================= 1 failed, 0 succeeded in 00:01:18.020 =============================================================================

 *  The terminal process "C:\Users\hmbac\.platformio\penv\Scripts\platformio.exe 'run'" terminated with exit code: 1. 
 *  Terminal will be reused by tasks, press any key to close it.

Custom scripts (enum generation, frontend build, SSL cert bundle generation) seem to work fine and the pipeline seems to recognize the custom_sdkconfig settings in the platformio.ini file as well, starting the Arduino IDF libs compilation.

Now to what I could not solve:

```
Warning! Flash memory size mismatch detected. Expected 8MB, found 16MB!
Please select a proper value in your `sdkconfig.defaults` or via the `menuconfig` target!
```
I was not able to properly specifiy 8MB of flash size (board config is 8MB, but somewhere else 16MB seems to be a default value). Changing sdkconfig.defaults as proposed seems not helpful, as the file is regenarted with every build.
How do i properly specify 8MB to avoid this warning?
https_server.crt.S not found in pio build folder.
Why might this file be missing? I had the file(s) once (no idea what I did to get it) and with the file available the build continued until I was finally left with a linker error. But now, I am not able to overcome even this point.

Any help is highly appreciated!
Many thanks in advance!

maxgerhardt · October 3, 2025, 11:28am

When you download https://github.com/pioarduino/platform-espressif32/archive/refs/tags/55.03.31.zip and open the https://github.com/pioarduino/platform-espressif32/tree/55.03.31/examples/espidf-arduino-blink project inside of, does it also fail to compile?

It has the same

github.com/pioarduino/platform-espressif32

examples/espidf-arduino-blink/platformio.ini

8715c1708


      
          framework = arduino, espidf
          build_flags =
              -DCONFIG_BLINK_GPIO=2
          monitor_speed = 115200
          board_build.embed_txtfiles =
              managed_components/espressif__esp_insights/server_certs/https_server.crt
              managed_components/espressif__esp_rainmaker/server_certs/rmaker_mqtt_server.crt
              managed_components/espressif__esp_rainmaker/server_certs/rmaker_claim_service_server.crt
              managed_components/espressif__esp_rainmaker/server_certs/rmaker_ota_server.crt

instructions. So if it works in there, it might be something that’s specific to your project.

hmbacher · October 3, 2025, 3:32pm

Thanks for the idea, i tried with examples/arduino-blink first, as it uses arduino as single framework and contains custom_sdkconfig param, with the following results:

When just using the original example it perfectly built.

As many components are manually disabled/removed by custom_component_remove param, I assumed, this might be the reason for successfull built.

When commenting out the custom_component_remove -list (i.e. disabling removing of any components), the built failed with the exact same error message!

I narrowed it down to the esp_insights component.

Removing just this single component esp_insights avoids the https_server.crt.S.o related error message, bringing up a very similar error message related to rmaker_mqtt_server.crt.S.o some compile steps later!

These files seem likely related somehow to the files referenced in another example: platform-espressif32/examples/espidf-arduino-blink/platformio.ini at main · pioarduino/platform-espressif32 · GitHub , but just adding the board_build.embed_txtfiles param to the arduino-blink project did not help either.

I will continue adding espidf as additional framework (framework: arduino, espidf).

hmbacher · October 5, 2025, 1:35pm

I have had some more insights by looking at the pioarduino platform scripts:

If you sepcify arduino and epidf as framework, the pioarduino platform scripts recognize this as “Arduino as component” and switches off hybrid compile
The original error about the missing file came up by an inconsistent project configuration where ESP Insights was not added to managed_components directory (as espressif__esp_insights).

There were two ways to solve this:

Cleaning up project structure (manually deleting /.pio/build/…, /.pio/libdeps/…, /managed_components, /skdconfig.*) and regenerating (e.g. by saving platformio.ini file)

Then, if ESP Insights was added (espressif__esp_insights) I had to add board_build.embed_txtfiles = ...to the platformio.ini file (see first post).
Second option is to exculde ESP Insights and ESP Rainmaker components in platformio.ini file (see examples/arduino-blink)

Finally I was able to use the hybrid compile feature - and found out, that there is a much simpler solution!

My original motivation was to increase max HTTP header size from 1024 to 2048 bytes. Instead of recompiling IDF Libs with increased default value (CONFIG_HTTPD_MAX_REQ_HDR_LEN) and rebuilding Arduino, one can just increase max_req_hdr_len of httpd_config_t struct:

// Increase max header length
httpd_config_t config = HTTPD_DEFAULT_CONFIG();
config.max_req_hdr_len = 2048; // Default is 1024

C’est la vie!