PlatformIO Community

How to compile against ESP32-WROVER-E?

Hi,
I have been working ESP32-WROOM-32 for a long time but needed more RAM dn FLASH since the programm got too large to support two extra OTA partition next to the factory partition.

How do i correctly compile and configure the chip? Since there are many different sizes for ram (etc) i am not sure how to do this without bricking it.

What I got so far (so it’s in one place):

From ESP32 - How To Use PSRAM • ThingPulse we get compiler flags for the psram

-DBOARD_HAS_PSRAM
-mfix-esp32-psram-cache-issue

Which as far as i read it correctly just binds 4 MB into memory, so i don’t need to do anything else and it just is in memory? I found that the lower 4MB are accessible with himem.h and a memory bank switch; I need the memory to be able to serve as stack for some processes which get rather large.

From the datasheet:

I got the size, which can be different. Do I have to tell the compler that the flash/ram is that large ? or does that get done automatically when gathering the chip information ?

And a final question which is kinda out there. I am assuming i can just create a partition table and as long as i am in the size limit of the flash it is fine. right ?

Any help is greatly appreciated! I find the chip version vs revision vs chip type and dev board thing very confusing.

Well they turn on the code telling the Arduino-ESP32 core to look for that memory, not specifically 4MB. Arduino-ESP32 leans on ESP-IDF’s esp_spiram_init function (called here). ESP-IDF knows about 8MB PSRAM, too. The size is enumerated dynmamically by querrying the chip via SPI.

This still does not change the fact that the ESP32 has architectural limitations with the memory map

During the ESP-IDF startup, external RAM is mapped into the data address space, starting at address 0x3F800000 (byte-accessible). The length of this region is the same as the SPI RAM size (up to the limit of 4 MB).

This is related:

Thanks for your help!
I have followed all these instructions but as soon as i try to upload to the chip with the sdk config and compile flags i get this error:

esptool write_flash: error: argument <address> <filename>: Detected overlap at address: 0x8000 for file: /home/zweieuro/Documents/PlatformIO/Projects/master_gate/.pio/build/esp-wrover-kit/partitions.bin
Which i can’t find anywhere, are there some requirements for the partition table in order to use the ram ?
I ma using this pio.ini

[env:esp-wrover-kit]

platform = espressif32

board = esp-wrover-kit

framework = espidf

monitor_speed = 115200

monitor_filters = esp32_exception_decoder

board_build.partitions = partition_table.csv

build_type = debug

build_flags = -fpermissive -DCORE_DEBUG_LEVEL=5 -DBOARD_HAS_PSRAM -mfix-esp32-psram-cache-issue

This is my partition table:

# Name,   Type, SubType,  Offset,   Size,  Flags
nvs,      data, nvs,      0x9000,  0x4000
otadata,  data, ota,      0xd000,  0x2000
phy_init, data, phy,      0xf000,  0x1000
factory,  app,  factory,  0x10000,  1900k
ota_0,    app,  ota_0,    ,         1900k
storage,  data, spiffs,  ,        170k

and this is what my sdkconfig.defaults looks like:

CONFIG_BT_ENABLED=y
CONFIG_BTDM_CTRL_MODE_BLE_ONLY=y
CONFIG_BTDM_CTRL_MODE_BR_EDR_ONLY=n
CONFIG_BTDM_CTRL_MODE_BTDM=n
CONFIG_BT_BLUEDROID_ENABLED=y
CONFIG_BT_NIMBLE_ENABLED=n
CONFIG_BTDM_MODEM_SLEEP=n

CONFIG_FREERTOS_USE_TRACE_FACILITY=y
CONFIG_FREERTOS_GENERATE_RUN_TIME_STATS=y
CONFIG_BT_BTC_TASK_STACK_SIZE=4096
CONFIG_ESP32_PHY_CALIBRATION_AND_DATA_STORAGE=y
CONFIG_BOOTLOADER_LOG_LEVEL=5
CONFIG_ESP32_PHY_INIT_DATA_ERROR=y
CONFIG_ESP32_WIFI_STATIC_RX_BUFFER_NUM=10

CONFIG_ESP32_WIFI_DEBUG_LOG_ENABLE=n
CONFIG_ESP32_WIFI_DEBUG_LOG_DEBUG=n
# CONFIG_ESP32_WIFI_DEBUG_LOG_VERBOSE is not set
CONFIG_ESP32_WIFI_DEBUG_LOG_MODULE_ALL=n
# CONFIG_ESP32_WIFI_DEBUG_LOG_MODULE_WIFI is not set
# CONFIG_ESP32_WIFI_DEBUG_LOG_MODULE_COEX is not set
# CONFIG_ESP32_WIFI_DEBUG_LOG_MODULE_MESH is not set
CONFIG_ESP32_WIFI_DEBUG_LOG_SUBMODULE=n
CONFIG_ESP32_WIFI_DEBUG_LOG_SUBMODULE_ALL=n

# HEAP debugging
CONFIG_HEAP_POISONING_COMPREHENSIVE=n
CONFIG_HEAP_TASK_TRACKING=n

#OTA
CONFIG_BOOTLOADER_APP_ROLLBACK_ENABLE=y
#CONFIG_IDF_TARGET_ESP32=y


#SPIRAM
CONFIG_ESP32_SPIRAM_SUPPORT=y
CONFIG_SPIRAM_BOOT_INIT=y
CONFIG_SPIRAM_IGNORE_NOTFOUND=n
CONFIG_SPIRAM_USE_MALLOC=y
CONFIG_SPIRAM_TYPE_AUTO=y
CONFIG_SPIRAM_SIZE=-1
CONFIG_SPIRAM_SPEED_40M=y
CONFIG_SPIRAM_MEMTEST=y
CONFIG_SPIRAM_CACHE_WORKAROUND=y
CONFIG_SPIRAM_BANKSWITCH_ENABLE=y
CONFIG_SPIRAM_BANKSWITCH_RESERVE=4

I just can’t find any information on why the partition table is even relevant here, it works perfectly fine if the spiram configs are not present

You need to also set up the partition table in the menuconfig.

grafik

Huh?
I’ve always just done this via platformio and until now this worked perfectly as expected?
I only have platformio installed on my system, do i need to manually add the esp-idf in order to change this setting ?

So in the SDK there are these settings:

#
# Partition Table
#
CONFIG_PARTITION_TABLE_SINGLE_APP=y
# CONFIG_PARTITION_TABLE_TWO_OTA is not set
# CONFIG_PARTITION_TABLE_CUSTOM is not set
CONFIG_PARTITION_TABLE_CUSTOM_FILENAME="partitions.csv"
CONFIG_PARTITION_TABLE_FILENAME="partitions_singleapp.csv"
CONFIG_PARTITION_TABLE_OFFSET=0x8000
CONFIG_PARTITION_TABLE_MD5=y
# end of Partition Table

Which i assume should mention my partition table “partition_table.csv”

Have you checked the documentation link I provided you above? You do not need a separate ESP-IDF installation for this. You are expected to open a CLI and execute pio run -t menuconfig in it to get into the menuconfig.

Exactly.

Ok the first thing is on me i only had the extension installed not the package so my system didn’t recognize it properly, my bad.

I have repeated my configuration in the menuconfig client but still have the same error.
I also looked through the spiram config but didn’t find anything out of order since my defaults configured this correctly already (it seems).

What i don’t get is that i am offsetting my partition table first partition to 0x9000 which shouldn’t collide with the bootloader at 0x8000 at all …

if i use GitHub - majenkotech/esp32part: Small program to extract an ESP32 partition CSV file from a binary image of the partition table. to decompile my partition table i get:

Name, Type, SubType, Offset, Size, Flags

nvs, data, nvs, 0x00009000, 0x00004000,
otadata, data, ota, 0x0000d000, 0x00002000,
phy_init, data, phy, 0x0000f000, 0x00001000,
factory, app, factory, 0x00010000, 0x001db000,
ota_0, app, ota_0, 0x001f0000, 0x001db000,
storage, data, spiffs, 0x003cb000, 0x0002a800,

which looks fine

I’ve tested the espidf-blink project with your partition table and I don’t run in any issues.

[env:esp32dev]
platform = espressif32
framework = espidf
board = esp32dev
monitor_speed = 115200
build_flags =
        ; https://docs.espressif.com/projects/esp-idf/en/latest/get-started/get-started-wrover-kit.html#rgb-led
        -D CONFIG_BLINK_GPIO=2
board_build.partitions = partition_table.csv
# Name,   Type, SubType,  Offset,   Size,  Flags
nvs,      data, nvs,      0x9000,  0x4000
otadata,  data, ota,      0xd000,  0x2000
phy_init, data, phy,      0xf000,  0x1000
factory,  app,  factory,  0x10000,  1900k
ota_0,    app,  ota_0,    ,         1900k
storage,  data, spiffs,  ,        170k
>cat sdkconfig | grep PART
# CONFIG_PARTITION_TABLE_SINGLE_APP is not set
# CONFIG_PARTITION_TABLE_TWO_OTA is not set
CONFIG_PARTITION_TABLE_CUSTOM=y
CONFIG_PARTITION_TABLE_CUSTOM_FILENAME="partition_table.csv"
CONFIG_PARTITION_TABLE_FILENAME="partition_table.csv"
CONFIG_PARTITION_TABLE_OFFSET=0x8000
CONFIG_PARTITION_TABLE_MD5=y
# CONFIG_ESP32_PHY_INIT_DATA_IN_PARTITION is not set
CONFIG_SPIFFS_MAX_PARTITIONS=3
C:\Users\Max\temp\espidf-blink>pio run
Processing esp32dev (platform: espressif32; framework: espidf; board: esp32dev)
-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Verbose mode can be enabled via `-v, --verbose` option
CONFIGURATION: https://docs.platformio.org/page/boards/espressif32/esp32dev.html
PLATFORM: Espressif 32 (3.2.1) > Espressif ESP32 Dev Module
HARDWARE: ESP32 240MHz, 320KB RAM, 4MB Flash
DEBUG: Current (esp-prog) External (esp-prog, iot-bus-jtag, jlink, minimodule, olimex-arm-usb-ocd, olimex-arm-usb-ocd-h, olimex-arm-usb-tiny-h, olimex-jtag-tiny, tumpa)
PACKAGES:
 - framework-espidf 3.40201.210513 (4.2.1)
 - tool-cmake 3.16.4
 - tool-esptoolpy 1.30000.201119 (3.0.0)
 - tool-idf 1.0.1
 - tool-mconf 1.4060000.20190628 (406.0.0)
 - tool-ninja 1.9.0
 - toolchain-esp32ulp 1.22851.191205 (2.28.51)
 - toolchain-xtensa32 2.80400.210114 (8.4.0)
Reading CMake configuration...
Generating assembly for certificate bundle...
LDF: Library Dependency Finder -> http://bit.ly/configure-pio-ldf
LDF Modes: Finder ~ chain, Compatibility ~ soft
Found 0 compatible libraries
Scanning dependencies...
No dependencies
Building in release mode
Compiling .pio\build\esp32dev\src\blink.o
[...]
esptool.py v3.0
Linking .pio\build\esp32dev\firmware.elf
Retrieving maximum program size .pio\build\esp32dev\firmware.elf
Checking size .pio\build\esp32dev\firmware.elf
Advanced Memory Usage is available via "PlatformIO Home > Project Inspect"
RAM:   [          ]   3.2% (used 10464 bytes from 327680 bytes)
Flash: [=         ]   7.8% (used 150900 bytes from 1945600 bytes)
Building .pio\build\esp32dev\firmware.bin
esptool.py v3.0
============== [SUCCESS] Took 52.89 seconds ==============

If in doubt, remove the .pio folder of the project to get a clean build.

using “pio run” doesn’t produce any error for me either.
the error only occures when uploading

> Executing task in folder master_gate: platformio run <

Processing esp-wrover-kit (platform: espressif32; board: esp-wrover-kit; framework: espidf)
-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Verbose mode can be enabled via `-v, --verbose` option
CONFIGURATION: https://docs.platformio.org/page/boards/espressif32/esp-wrover-kit.html
PLATFORM: Espressif 32 (3.2.0) > Espressif ESP-WROVER-KIT
HARDWARE: ESP32 240MHz, 320KB RAM, 4MB Flash
DEBUG: Current (ftdi) On-board (ftdi) External (esp-prog, iot-bus-jtag, jlink, minimodule, olimex-arm-usb-ocd, olimex-arm-usb-ocd-h, olimex-arm-usb-tiny-h, olimex-jtag-tiny, tumpa)
PACKAGES: 
 - framework-espidf 3.40200.210118 (4.2.0) 
 - tool-cmake 3.16.4 
 - tool-esptoolpy 1.30000.201119 (3.0.0) 
 - tool-ninja 1.7.1 
 - toolchain-esp32ulp 1.22851.191205 (2.28.51) 
 - toolchain-xtensa32 2.80400.210211 (8.4.0)
Reading CMake configuration...
LDF: Library Dependency Finder -> http://bit.ly/configure-pio-ldf
LDF Modes: Finder ~ chain, Compatibility ~ soft
Found 0 compatible libraries
Scanning dependencies...
No dependencies
Building in debug mode
Retrieving maximum program size .pio/build/esp-wrover-kit/firmware.elf
Checking size .pio/build/esp-wrover-kit/firmware.elf
Advanced Memory Usage is available via "PlatformIO Home > Project Inspect"
RAM:   [==        ]  21.3% (used 69780 bytes from 327680 bytes)
Flash: [========= ]  88.6% (used 1723656 bytes from 1945600 bytes)
======================================================================================= [SUCCESS] Took 6.09 seconds =======================================================================================

Terminal will be reused by tasks, press any key to close it.

although… running it with pio run (not with the button from the pio vscode extension) does produce an error:
Processing esp-wrover-kit (platform: espressif32; board: esp-wrover-kit; framework: espidf)
-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Verbose mode can be enabled via -v, --verbose option
CONFIGURATION: Redirecting...
PLATFORM: Espressif 32 (3.2.0) > Espressif ESP-WROVER-KIT
HARDWARE: ESP32 240MHz, 320KB RAM, 4MB Flash
DEBUG: Current (ftdi) On-board (ftdi) External (esp-prog, iot-bus-jtag, jlink, minimodule, olimex-arm-usb-ocd, olimex-arm-usb-ocd-h, olimex-arm-usb-tiny-h, olimex-jtag-tiny, tumpa)
PACKAGES:
- framework-espidf 3.40200.210118 (4.2.0)
- tool-cmake 3.16.4
- tool-esptoolpy 1.30000.201119 (3.0.0)
- tool-ninja 1.7.1
- toolchain-esp32ulp 1.22851.191205 (2.28.51)
- toolchain-xtensa32 2.80400.210211 (8.4.0)
Reading CMake configuration…
LDF: Library Dependency Finder → http://bit.ly/configure-pio-ldf
LDF Modes: Finder ~ chain, Compatibility ~ soft
Found 0 compatible libraries
Scanning dependencies…
No dependencies
Building in debug mode
Generating partitions .pio/build/esp-wrover-kit/partitions.bin
Generating project linker script .pio/build/esp-wrover-kit/esp32.project.ld
Building .pio/build/esp-wrover-kit/bootloader.bin
Generating an empty partition .pio/build/esp-wrover-kit/ota_data_initial.bin
esptool.py v3.0
linker script generation failed for /home/zweieuro/.platformio/packages/framework-espidf/components/esp32/ld/esp32.project.ld.in
ERROR: failed to parse /home/zweieuro/.platformio/packages/framework-espidf/components/esp_event/linker.lf
Expected end of text, found ‘i’ (at char 0), (line:1, col:1)
*** [.pio/build/esp-wrover-kit/esp32.project.ld] Error 1

When I do a verbose upload I get

CURRENT: upload_protocol = esptool
MethodWrapper(["upload"], [".pio\build\esp32dev\firmware.bin"])
Auto-detected: COM40
"c:\users\max\appdata\local\programs\python\python38\python.exe" "C:\Users\Max\.platformio\packages\tool-esptoolpy\esptool.py" --chip esp32 --port "COM40" --baud 460800 --before default_reset --after hard_reset write_flash -z --flash_mode dio --flash_freq 40m --flash_size detect 0x1000 C:\Users\Max\temp\espidf-blink\.pio\build\esp32dev\bootloader.bin 0x8000 C:\Users\Max\temp\espidf-blink\.pio\build\esp32dev\partitions.bin 0xd000 C:\Users\Max\temp\espidf-blink\.pio\build\esp32dev\ota_data_initial.bin 0x10000 .pio\build\esp32dev\firmware.bin
esptool.py v3.0
Serial port COM40
Connecting........_
Chip is ESP32-D0WDQ6 (revision 0)
Features: WiFi, BT, Dual Core, Coding Scheme None
Crystal is 40MHz
MAC: 24:0a:c4:04:26:38
Uploading stub...
Running stub...
Stub running...
Changing baud rate to 460800
Changed.
Configuring flash size...
Auto-detected Flash size: 4MB
Compressed 25152 bytes to 15293...
Writing at 0x00001000... (100 %)
Wrote 25152 bytes (15293 compressed) at 0x00001000 in 0.4 seconds (effective 479.6 kbit/s)...
Hash of data verified.
Compressed 3072 bytes to 150...
Writing at 0x00008000... (100 %)
Wrote 3072 bytes (150 compressed) at 0x00008000 in 0.1 seconds (effective 487.0 kbit/s)...
Hash of data verified.
Compressed 8192 bytes to 31...
Writing at 0x0000d000... (100 %)
Wrote 8192 bytes (31 compressed) at 0x0000d000 in 0.1 seconds (effective 1276.4 kbit/s)...
Hash of data verified.
Compressed 151024 bytes to 78776...
Writing at 0x00010000... (20 %)
Writing at 0x00014000... (40 %)
Writing at 0x00018000... (60 %)
Writing at 0x0001c000... (80 %)
Writing at 0x00020000... (100 %)
Wrote 151024 bytes (78776 compressed) at 0x00010000 in 2.1 seconds (effective 570.2 kbit/s)...
Hash of data verified.

Leaving...
Hard resetting via RTS pin...
============== [SUCCESS] Took 25.20 seconds ==============

You are running most interestingly in the same issue as someone recently posted in ESP32 (framework-espidf 4.2.1) - linker script generation failed. Please follow the fixing steps there.

Okay so pyparsing > 2.2.0 is incompatible with the idf?
thats a bit annoying, need a virtualenv to use matplotlib now since that requires 2.2.1 but whatever.

pio run does do its job now.

uploading still causes the same error, but it gets past linking now:

> Executing task in folder master_gate: platformio run --target upload <

Processing esp-wrover-kit (platform: espressif32; board: esp-wrover-kit; framework: espidf)
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Verbose mode can be enabled via `-v, --verbose` option
CONFIGURATION: https://docs.platformio.org/page/boards/espressif32/esp-wrover-kit.html
PLATFORM: Espressif 32 (3.2.0) > Espressif ESP-WROVER-KIT
HARDWARE: ESP32 240MHz, 320KB RAM, 4MB Flash
DEBUG: Current (ftdi) On-board (ftdi) External (esp-prog, iot-bus-jtag, jlink, minimodule, olimex-arm-usb-ocd, olimex-arm-usb-ocd-h, olimex-arm-usb-tiny-h, olimex-jtag-tiny, tumpa)
PACKAGES: 
 - framework-espidf 3.40200.210118 (4.2.0) 
 - tool-cmake 3.16.4 
 - tool-esptoolpy 1.30000.201119 (3.0.0) 
 - tool-mkspiffs 2.230.0 (2.30) 
 - tool-ninja 1.7.1 
 - toolchain-esp32ulp 1.22851.191205 (2.28.51) 
 - toolchain-xtensa32 2.80400.210211 (8.4.0)
Reading CMake configuration...
LDF: Library Dependency Finder -> http://bit.ly/configure-pio-ldf
LDF Modes: Finder ~ chain, Compatibility ~ soft
Found 0 compatible libraries
Scanning dependencies...
No dependencies
Building in debug mode
Generating partitions .pio/build/esp-wrover-kit/partitions.bin
Generating project linker script .pio/build/esp-wrover-kit/esp32.project.ld
Building .pio/build/esp-wrover-kit/bootloader.bin
Generating an empty partition .pio/build/esp-wrover-kit/ota_data_initial.bin
esptool.py v3.0
Linking .pio/build/esp-wrover-kit/firmware.elf
Retrieving maximum program size .pio/build/esp-wrover-kit/firmware.elf
Checking size .pio/build/esp-wrover-kit/firmware.elf
Advanced Memory Usage is available via "PlatformIO Home > Project Inspect"
RAM:   [==        ]  21.1% (used 69164 bytes from 327680 bytes)
Flash: [========= ]  85.3% (used 1660212 bytes from 1945600 bytes)
Building .pio/build/esp-wrover-kit/firmware.bin
esptool.py v3.0
Configuring upload protocol...
AVAILABLE: esp-prog, espota, esptool, ftdi, iot-bus-jtag, jlink, minimodule, olimex-arm-usb-ocd, olimex-arm-usb-ocd-h, olimex-arm-usb-tiny-h, olimex-jtag-tiny, tumpa
CURRENT: upload_protocol = esptool
Looking for upload port...
Auto-detected: /dev/ttyUSB0
Uploading .pio/build/esp-wrover-kit/firmware.bin
usage: esptool write_flash [-h] [--erase-all]
                           [--flash_freq {keep,40m,26m,20m,80m}]
                           [--flash_mode {keep,qio,qout,dio,dout}]
                           [--flash_size FLASH_SIZE]
                           [--spi-connection SPI_CONNECTION] [--no-progress]
                           [--verify] [--encrypt]
                           [--ignore-flash-encryption-efuse-setting]
                           [--compress | --no-compress]
                           <address> <filename> [<address> <filename> ...]
esptool write_flash: error: argument <address> <filename>: Detected overlap at address: 0x8000 for file: /home/zweieuro/Documents/PlatformIO/Projects/master_gate/.pio/build/esp-wrover-kit/partitions.bin
*** [upload] Error 2

Also it is a bit weird that the vscode run has different results than the one on my system, i am guessing the extension is packages with 2.2.0 which is why the linker error only occures on my system where it was 2.2.1

running pio run --target upload --verbose
produces a lot of output, but at the end it may be relevant to this issue (everything above is just the .elf file that was generated)
    <lambda>(["upload"], [".pio/build/esp-wrover-kit/firmware.bin"])
    AVAILABLE: esp-prog, espota, esptool, ftdi, iot-bus-jtag, jlink, minimodule, olimex-arm-usb-ocd, olimex-arm-usb-ocd-h, olimex-arm-usb-tiny-h, olimex-jtag-tiny, tumpa
    CURRENT: upload_protocol = esptool
    MethodWrapper(["upload"], [".pio/build/esp-wrover-kit/firmware.bin"])
    Auto-detected: /dev/ttyUSB0
    "/usr/bin/python" "/home/zweieuro/.platformio/packages/tool-esptoolpy/esptool.py" --chip esp32 --port "/dev/ttyUSB0" --baud 460800 --before default_reset --after hard_reset write_flash -z --flash_mode dio --flash_freq 40m --flash_size detect 0x1000 /home/zweieuro/Documents/PlatformIO/Projects/master_gate/.pio/build/esp-wrover-kit/bootloader.bin 0x8000 /home/zweieuro/Documents/PlatformIO/Projects/master_gate/.pio/build/esp-wrover-kit/partitions.bin 0xd000 /home/zweieuro/Documents/PlatformIO/Projects/master_gate/.pio/build/esp-wrover-kit/ota_data_initial.bin 0x10000 .pio/build/esp-wrover-kit/firmware.bin

looking through the bin files and their sizes yields:
32K bootloader.bin lands from 0x1000 - 0x8000 seems to be not enough space for the bottloader (28672)
1,6M firmware.bin
8,0K ota_data_initial.bin
4,0K partitions.bin

To fix this i removed all the offset values from the partition table so it can be done automatically and changed
CONFIG_PARTITION_TABLE_OFFSET=0x8000
to
CONFIG_PARTITION_TABLE_OFFSET=0x9000

and now it could upload fine

Well now it runs but i produce:
assertion "it != NULL" failed: file "/home/zweieuro/.platformio/packages/framework-espidf/components/app_update/esp_ota_ops.c", line 542, function: esp_ota_get_running_partition

and crash instantly when using that function, which i need in order to use OTA.
So i am guessing i messed something up with the partition table again ?

Thinking this was caused by the ota partition i ran
pio run --target erase
and then uploaded again, now i am stuck on this infinite loop

I (14) boot: ESP-IDF 3.40200.210118 2nd stage bootloader
I (14) boot: compile time 15:27:36
I (14) boot: chip revision: 3
I (17) boot_comm: chip revision: 3, min. bootloader chip revision: 0
I (24) boot.esp32: SPI Speed      : 40MHz
I (28) boot.esp32: SPI Mode       : DIO
I (33) boot.esp32: SPI Flash Size : 4MB
I (37) boot: Enabling RNG early entropy source...
I (43) boot: Partition Table:
I (46) boot: ## Label            Usage          Type ST Offset   Length
I (54) boot:  0 nvs              WiFi data        01 02 00009000 00004000
I (61) boot:  1 otadata          OTA data         01 00 0000d000 00002000
I (69) boot:  2 phy_init         RF data          01 01 0000f000 00001000
I (76) boot:  3 factory          factory app      00 00 00010000 001db000
I (83) boot:  4 ota_0            OTA app          00 10 001f0000 001db000
I (91) boot:  5 storage          Unknown data     01 82 003cb000 0001e000
I (98) boot: End of partition table
I (103) boot: Defaulting to factory image
E (107) esp_image: image at 0x10000 has invalid magic byte
W (113) esp_image: image at 0x10000 has invalid SPI mode 255
W (120) esp_image: image at 0x10000 has invalid SPI size 15
E (126) boot: Factory app partition is not bootable
E (132) esp_image: image at 0x1f0000 has invalid magic byte
W (138) esp_image: image at 0x1f0000 has invalid SPI mode 255
W (144) esp_image: image at 0x1f0000 has invalid SPI size 15
E (151) boot: OTA app partition slot 0 is not bootable
E (156) boot: No bootable app partitions in the partition table
I (14) boot: ESP-IDF 3.40200.210118 2nd stage bootloader
I (14) boot: compile time 15:27:36
I (14) boot: chip revision: 3
I (17) boot_comm: chip revision: 3, min. bootloader chip revision: 0
I (24) boot.esp32: SPI Speed      : 40MHz
I (28) boot.esp32: SPI Mode       : DIO
I (33) boot.esp32: SPI Flash Size : 4MB
I (37) boot: Enabling RNG early entropy source...
I (43) boot: Partition Table:
I (46) boot: ## Label            Usage          Type ST Offset   Length

which really looks like i broke something

o_O I can’t reproduce those issues. I’ve also done a pio run -t erase, then using the above esp-idf blinky project with the original partition table that you’ve posted, uploading that results in

C:\Users\Max\temp\espidf-blink>pio device monitor -b 115200 --raw
--- Available filters and text transformations: colorize, debug, default, direct, esp32_exception_decoder, hexlify, log2file, nocontrol, printable, send_on_enter, time
--- More details at http://bit.ly/pio-monitor-filters
--- Miniterm on COM40  115200,8,N,1 ---
--- Quit: Ctrl+C | Menu: Ctrl+T | Help: Ctrl+T followed by Ctrl+H ---
ets Jun  8 2016 00:22:57

rst:0x10 (RTCWDT_RTC_RESET),boot:0x13 (SPI_FAST_FLASH_BOOT)
configsip: 0, SPIWP:0xee
clk_drv:0x00,q_drv:0x00,d_drv:0x00,cs0_drv:0x00,hd_drv:0x00,wp_drv:0x00
mode:DIO, clock div:2
load:0x3fff0030,len:4
load:0x3fff0034,len:7272
load:0x40078000,len:13212
load:0x40080400,len:4568
entry 0x400806f4
I (29) boot: ESP-IDF 3.40200.210118 2nd stage bootloader
I (30) boot: compile time 14:14:32
I (30) boot: chip revision: 0
I (33) boot.esp32: SPI Speed      : 40MHz
I (38) boot.esp32: SPI Mode       : DIO
I (42) boot.esp32: SPI Flash Size : 4MB
I (47) boot: Enabling RNG early entropy source...
I (52) boot: Partition Table:
I (56) boot: ## Label            Usage          Type ST Offset   Length
I (63) boot:  0 nvs              WiFi data        01 02 00009000 00004000
I (71) boot:  1 otadata          OTA data         01 00 0000d000 00002000
I (78) boot:  2 phy_init         RF data          01 01 0000f000 00001000
I (85) boot:  3 factory          factory app      00 00 00010000 001db000
I (93) boot:  4 ota_0            OTA app          00 10 001f0000 001db000
I (100) boot:  5 storage          Unknown data     01 82 003cb000 0002a800
I (108) boot: End of partition table
I (112) boot: Defaulting to factory image
I (117) esp_image: segment 0: paddr=0x00010020 vaddr=0x3f400020 size=0x05f78 ( 24440) map
I (135) esp_image: segment 1: paddr=0x00015fa0 vaddr=0x3ffb0000 size=0x020b0 (  8368) load
I (139) esp_image: segment 2: paddr=0x00018058 vaddr=0x40080000 size=0x00404 (  1028) load
I (144) esp_image: segment 3: paddr=0x00018464 vaddr=0x40080404 size=0x07bb4 ( 31668) load
I (167) esp_image: segment 4: paddr=0x00020020 vaddr=0x400d0020 size=0x12e60 ( 77408) map
I (196) esp_image: segment 5: paddr=0x00032e88 vaddr=0x40087fb8 size=0x01f38 (  7992) load
I (206) boot: Loaded app from partition at offset 0x10000
I (206) boot: Disabling RNG early entropy source...
I (208) cpu_start: Pro cpu up.
I (212) cpu_start: Application information:
I (216) cpu_start: Project name:     espidf-blink
I (222) cpu_start: App version:      1
I (226) cpu_start: Compile time:     Jun 15 2021 14:13:59
I (232) cpu_start: ELF file SHA256:  64837967f2f2eff9...
I (238) cpu_start: ESP-IDF:          3.40200.210118
I (244) cpu_start: Starting app cpu, entry point is 0x40081090
I (0) cpu_start: App cpu up.
I (254) heap_init: Initializing. RAM available for dynamic allocation:
I (261) heap_init: At 3FFAE6E0 len 00001920 (6 KiB): DRAM
I (267) heap_init: At 3FFB28E0 len 0002D720 (181 KiB): DRAM
I (274) heap_init: At 3FFE0440 len 00003AE0 (14 KiB): D/IRAM
I (280) heap_init: At 3FFE4350 len 0001BCB0 (111 KiB): D/IRAM
I (286) heap_init: At 40089EF0 len 00016110 (88 KiB): IRAM
I (293) cpu_start: Pro cpu start user code
I (311) spi_flash: detected chip: gd
I (311) spi_flash: flash io: dio
I (311) cpu_start: Starting scheduler on PRO CPU.
I (0) cpu_start: Starting scheduler on APP CPU.
Turning off the LED
Turning on the LED
Turning off the LED

I can’t quite follow: What is of version 2.2.1 vs 2.2.0? I only have PlatformIO and C/C++ extension by Microsoft installed in my VSCode. No ESP-IDF / CMake extension.

Well your fix from before suggests 2.2.0 , i had 2.2.1 installed.
So it seems there was some compatability issue on my system with these versions.

but the problem from the linker only happens if i use pio run from the cli, not from the extension.
meaning they have different pyparse versions, meaning that the extension i got installed from pio packages the pyparse version as 2.2.0

it’s irrelevant to my problem here now though, i have no idea what to do now and hope i didn’t brick my chip.
I posted my issue here:

as a bug report, because i really think the partition table offset really messed something up

Ah you mean pyparsing. I was missing that context.

Do you have multiple PlatformIO cores installed maybe? One via pip and the other one via the normal VSCode PlatformIO extension which runs in a pyenv?

(try checking pio system info in the CLI vs the extension).

Tried that, they seem to be the same, also i fully removed it from my system and used the one from pio extension and visa versa, the problem persists

apparently ma problem is because PIO hard codes the factory app location and i made it realtive so that didn’t work.

now i got a different problem with the nvs, but i will try to fix that too

For further reference, i fixed my problem that the bootloader became bigger not by changing the total partition table offset, since that just created more problems, but moving the nvs partition a little lower 0x1000 and making it smaller by the same amount.
I am not using the nvs too much, so i jsut hope i am not producing some other problem with this change