ESP32-S3 plaformio(INI/JSON/CSV) FLASH and PSRAM implications help

Hi,

I am trying to have a better understanding of the FLASH and PSRAM settings, and its side effects when correctly configured in PIO.
Of course, to be able to setup an adequate platformio.ini, esp32-s3-devkitc-1.json and default_8MB.csv file (probably changing the name of the last two).
My test board is ESP32-S3-DevKitC-1.1 N8R2 and I am able to compile, monitor and debug, all on the native USB interface, using one USB cable.
I bought this kit for testing purposes only, but in my two coming projects (job), I will use “esp32-s3-wroom-1_1u” NxRy, having not decided yet for “Nx” and “Ry”.
So, for now my FLASH is 8MB and my PSRAM is 2MB. I have numbered all my questions and would very much appreciate if I can get answers/comments on them.

Maybe its rude to make so many questions in one post, but because they are very related, I think it makes sense (for future readers too).

Feel free to change or suggest a title change.

If you look at my compile LOG (red box), I have 320K of RAM used at a 5.9%

  • Am I guessing correctly, that this is defined “maximum_ram_size”: 327680 in the JSON file?
  • I imagine this chunk is taken from “Internal SRAM 1” bank as it is the only bank that can provide 320K from its 416K? (technical reference manual)
  • I imagine not more is taken as the RTOS needs storage memory too?
  • Have never seen something indicating the PSRAM size, only DBOARD_HAS_PSRAM, so I imagine it is only to tell the compiler to add/link the code, that enables to allocate under RTOS, memory chunks from it, by using “heap_caps_malloc”?
  • Because PSRAM is used dynamically I assume it is the responsibility of the programmer to check for enough space, but is this checking done on the return value of “heap_caps_malloc” and in some way RTOS will know the real size or might it be possible the “heap_caps_malloc” return a chunk that starts in valid PSRAM offset but ends beyond PSRAM’s end boundary (crash)?
  • Can PSRAM be used statically when the 320K get consumed completely?
  • Where do we have to tell if the PSRAM is Quad or Octal and how?

If you look at my compile LOG (yellow box), I have 3.342.336 bytes of flash (0x330000) that match with these two lines

app0, app, ota_0, 0x10000, 0x330000,
app1, app, ota_1, 0x340000,0x330000,

in the partition file

  • Why has the N8 and no PSRAM config (only one available at PIO), of the most basic esp32-s3-devkitc-1, divided the 8MB of FLASH in these two odd sized chunks leaving 1.703.936 bytes unused?
  • I just recompiled my last ESP project (where a regular ESP32 from a HELTEC kit was used) that had only 1.310.720 of FLASH and now, due to the library and PIO updates, instead of using 99% of FLASH, it reached 100.5% (used 1316949 bytes from 1310720 bytes). This is not a big deal as I can make it fit, and I do not expect to surpass the 0x330000 size in my next projects, but in a hypothetical case, would the compiler use app0 and app1 or would I be required to resize or join them?
  • I imagine, the best way to configure any type of NxRy (FLASH/PSRAM) combination of an S3 processor board, is to create a custom JSON file (referenced in platformio.ini) and on that file reference a custom CSV file, but I am wondering if I need to put something else than DBOARD_HAS_PSRAM in the platformio.ini file?
  • For my 8MB of FLASH “flash_mode”: “qio” is correct being it a Quad SPI, what would I need to use if the FLASH Octal SPI?
  • In case of 16MB/32MB Octal SPI, the fact that it is a 1,8V SPI is already wired in the board or do we need to set a parameter in the configuration files?
  • I recall seeing at some place (not sure if an INI PIO file) where “qio_qio” was set. What does this mean?

Regards,

Thomas

p.s.
This is my compile LOG of a simple blink on the ESP32-S3-DevKitC-1.1 N8R2:
imagen

This is my platformio.ini

[env:esp32-s3-devkitc-1]
platform = espressif32
board = esp32-s3-devkitc-1
framework = arduino
monitor_speed = 115200
upload_port = COM34
monitor_port = COM34
build_flags =
    -DARDUINO_USB_CDC_ON_BOOT=1
    -DBOARD_HAS_PSRAM
monitor_dtr = 1
monitor_rts = 1
lib_deps = adafruit/Adafruit [NeoPixel@^1.12.2](mailto:NeoPixel@%5e1.12.2)

Looking at esp32-s3-devkitc-1.json I find:

    "arduino":{
      "ldscript": "esp32s3_out.ld",
      "partitions": "default_8MB.csv"
    },

And looking at default_8MB.csv I find:

# Name,   Type, SubType, Offset,  Size, Flags
nvs,      data, nvs,     0x9000,  0x5000,
otadata,  data, ota,     0xe000,  0x2000,
app0,     app,  ota_0,   0x10000, 0x330000,
app1,     app,  ota_1,   0x340000,0x330000,
spiffs,   data, spiffs,  0x670000,0x180000,
coredump, data, coredump,0x7F0000,0x10000,
1 Like

Yes that’s correct. It is defined in the corresponding board .json file.

I think this is also correct.

The PSRAM is an external chip (located inside the ESP32-S3 module). It’s size is determined during runtime.

Fortunately the Arduino core helps you because it comes with the settings CONFIG_SPIRAM_USE=1 and CONFIG_SPIRAM_MALLOC_ALWAYSINTERNAL=4096

This allows you simply to use malloc().
All memory chunks >= 4096 bytes are taken from PS RAM automatically.

For details, please see esp32 api-guide

You can verify this by looking at the used address.
Here is a simple sketch to verify this behavior:

#include <Arduino.h>

void setup() {
    Serial.begin(115200);
    void* ptr1 = malloc(1*1024);
    void* ptr2 = malloc(1*1024);
    void* ptr3 = malloc(5*1024);

    Serial.printf("ptr1 = %08x\r\n", ptr1);
    Serial.printf("ptr2 = %08x\r\n", ptr2);
    Serial.printf("ptr3 = %08x\r\n", ptr3);
}

void loop() {
}

Output:

ptr1 = 3fcec7bc
ptr2 = 3fcecbcc
ptr3 = 3d800a0c

The memory taken for ptr3 is in the region where the PSRAM is mapped.

That’s good question which I cannot answer. (Haven’t tested it yet).
Mayby it’s worth to write a simple test sketch for this scenario.

Btw. You can’t allocate the complete 320K from the internal memory.
You only can allocated the maximum largest available block → ESP.getMaxAllocHeap()

There are 2 ways. Set the specific settings in the platformio.ini.
But i think that’s messy. So I decided to create my own board file for my ESP32-S3-Devkit-c1-N16R8. See here

You can use this file as a base for your board.
I think you only need to change the partition layout and maximum size, and the description of course.

Sorry, I have to make a break here (no more time left).
I’ll try to answer your remaining questions later today.

1 Like

The two partitions are used for OTA updates.
If you look at the complete partition table, you will notice that nothing remains unused.

You would need to use a custom partition table and customise it according to your needs.

See my board file for the S3-N16R8 linked above.
Even DBOARD_HAS_PSRAM is not mandatory in the platformio.ini if it is already defined in the .json file.

The quivalent would be “opi”

Flash and PS-RAM running in Quad SPI.


Your post contains a lot of detailed questions on various topics. This makes it difficult to go into detail on the individual points. I hope I have been able to answer your questions to some reasonable depth.

1 Like

Hi,

Very much appreciated and very clear.
Thanks for your time, especially considering the quantity of questions, that even to me were in the “excessive” range.
You answered all that was important for the decisions I have to take in these days, regarding this processor family.

Have a nice day and thanks again!

Thomas

2 Likes

I hope it is not foul to post a clarification “request” after marking a solution for the topic.
I am dealing with ESP32-S3-DevKitC-1-N8R2

and in the JSON file I have

{
  "build": {
    "arduino": {
      "ldscript": "esp32s3_out.ld",
      "partitions": "default_8MB_devkitc_n8r2.csv",
      "memory_type": "qio_qio"
    },
    "core": "esp32",
    "extra_flags": [
      "-DARDUINO_ESP32S3_DEV",
      "-DARDUINO_USB_MODE=1",
      "-DARDUINO_RUNNING_CORE=1",
      "-DARDUINO_EVENT_RUNNING_CORE=1",
      "-DBOARD_HAS_PSRAM"
    ],
    "f_cpu": "240000000L",
    "f_flash": "80000000L",
    "flash_mode": "qio",
    "psram_type": "qio",

"memory_type": "qio_qio" is generating a compile error but if I change it to "memory_type": "qio_opi" as in your JSON example it compiles (despite the inconsistency with "psram_type")

This is the first error reported

Executing task: C:\Users\tmagdahl\.platformio\penv\Scripts\platformio.exe run 

Processing esp32-s3-devkitc-1-n8r2 (platform: espressif32; board: esp32-s3-devkitc-1-n8r2; framework: arduino)
------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------Verbose mode can be enabled via `-v, --verbose` option
CONFIGURATION: https://docs.platformio.org/page/boards/espressif32/esp32-s3-devkitc-1-n8r2.html
PLATFORM: Espressif 32 (6.6.0) > Espressif ESP32-S3-DevKitC-1-N8R2 (8 MB QD FLASH, 2 MB QD PSRAM)
HARDWARE: ESP32S3 240MHz, 320KB RAM, 8MB Flash
DEBUG: Current (esp-builtin) On-board (esp-builtin) External (cmsis-dap, esp-bridge, esp-prog, iot-bus-jtag, jlink, minimodule, olimex-arm-usb-ocd, olimex-arm-usb-ocd-h, olimex-arm-usb-tiny-h, olimex-jtag-tiny, tumpa)
PACKAGES:
 - framework-arduinoespressif32 @ 3.20014.231204 (2.0.14)
 - tool-esptoolpy @ 1.40501.0 (4.5.1)
 - toolchain-riscv32-esp @ 8.4.0+2021r2-patch5
 - toolchain-xtensa-esp32s3 @ 8.4.0+2021r2-patch5
LDF: Library Dependency Finder -> https://bit.ly/configure-pio-ldf
LDF Modes: Finder ~ chain, Compatibility ~ soft
Found 34 compatible libraries
Scanning dependencies...
Dependency Graph
|-- Adafruit NeoPixel @ 1.12.2
Building in release mode      
Compiling .pio\build\esp32-s3-devkitc-1-n8r2\src\main.cpp.o
Compiling .pio\build\esp32-s3-devkitc-1-n8r2\lib817\Adafruit NeoPixel\Adafruit_NeoPixel.cpp.o
Compiling .pio\build\esp32-s3-devkitc-1-n8r2\lib817\Adafruit NeoPixel\esp.c.o
Compiling .pio\build\esp32-s3-devkitc-1-n8r2\lib817\Adafruit NeoPixel\esp8266.c.o
Compiling .pio\build\esp32-s3-devkitc-1-n8r2\lib817\Adafruit NeoPixel\kendyte_k210.c.o
In file included from c:\users\tmagdahl\.platformio\packages\toolchain-xtensa-esp32s3\xtensa-esp32s3-elf\sys-include\sys\reent.h:503,
                 from C:/Users/tmagdahl/.platformio/packages/framework-arduinoespressif32/tools/sdk/esp32s3/include/newlib/platform_include/sys/reent.h:17,
                 from c:\users\tmagdahl\.platformio\packages\toolchain-xtensa-esp32s3\xtensa-esp32s3-elf\sys-include\stdio.h:60,
                 from C:/Users/tmagdahl/.platformio/packages/framework-arduinoespressif32/cores/esp32/Arduino.h:27,
                 from src/main.cpp:1:
C:/Users/tmagdahl/.platformio/packages/framework-arduinoespressif32/tools/sdk/esp32s3/include/newlib/platform_include/assert.h:20:10: fatal error: sdkconfig.h: No such file or directory

*******************************************************************
* Looking for sdkconfig.h dependency? Check our library registry!
*
* CLI  > platformio lib search "header:sdkconfig.h"
* Web  > https://registry.platformio.org/search?q=header:sdkconfig.h
*
*******************************************************************

 #include <sdkconfig.h>
          ^~~~~~~~~~~~~
compilation terminated.

I looked at the include files reported and could not figure out what the problema was.

Any clue where I am messing things up?

This should be "memory_type": "qio_qspi"

So is there a place to find the construction rule (accepted values for prefix and postfix) I could take a look at?

Browse the boards.txt and look for used values

Sometimes there are combinations of different settings like
.build.memory_type={build.boot}_{build.psram_type}

Then you have to take a look at .build.boot= and build.psram_type too

Example:

...
esp32s3.build.boot=qio
...
esp32s3.build.psram_type=qspi
...
esp32s3.build.memory_type={build.boot}_{build.psram_type}

This also would result in build.memory_type=qio_qspi

Fine,
thanks a lot, I hope I do not need to abuse your good will any more (do not know why I have the feeling, I will have to eat my words at some near future :rofl:).

No problem.

Unfortunately, this is also one of the things for which there is little or no proper description. At least I have not yet found a useful description.

Hi,
I generated all the different SPI settings, found on the boards.txt file, by type of setting (column name).
Maybe usefull for somebody.

I have an excel file with the 1900 entries related to SPI, found in that same file, but did not find a way to upload that data other than doing a copy-paste into this text box, that seemed little rude. If somebody is interested send me an email to tmagdahl@dts.cl.
It has been rearranged so that you can filter things on MODE, BOOT, PSRAM and MEMORY.

Regards

MODE BOOT PSRAM MEMORY_TYPE
build.flash_mode build.boot build.psram_type build.memory_type
dio dio opi dio_qspi
dout dout qspi opi_opi
qio opi opi_qspi
qio qio_opi
qout qio_qspi
These are the different values found for each attribute indicated in the column title
Be aware that MEMORY_TYPE is prone to get more combinations in the future as it reflects the combination choosen on new boards
2 Likes

Hi there,
@sivar2311, I have to say reading and lurking in here, YOU have delivered on the Learning part for sure. Thank you very much for the above info, and taking the time to hit it point by point. I gleened more from this thread than the hours I spent picking and pecking through PLIO Wiki and Espressif GiT
You could give a clinic on walking the OP thru the solutions.
GREAT job
Thanks I appreciate it.
GL :slight_smile: PJ
:v:

1 Like

Thank’s for the kind words! :blush:

But the credits for the table goes to @t_magdahl :clap:

1 Like

Thank you so much @sivar2311 for all your help here and thank you @t_magdahl for asking questions!

I am a novice in all this, and this could be expected but once I include the esp32-s3-devkitc-1-n8r2.json in the boards folder of my project directory and reference it in platformio.ini, my output of the build command still shows lower flash and RAM size as seen below:

Building in release mode
Retrieving maximum program size .pio\build\esp32-s3-devkitc-1-n8r2\firmware.elf
Checking size .pio\build\esp32-s3-devkitc-1-n8r2\firmware.elf
Advanced Memory Usage is available via "PlatformIO Home > Project Inspect"
RAM:   [=         ]  13.9% (used 45552 bytes from 327680 bytes)
Flash: [===       ]  26.9% (used 897869 bytes from 3342336 bytes)

This is my platformio.ini:

[env:esp32-s3-devkitc-1-n8r2]
platform = espressif32
board = esp32-s3-devkitc-1-n8r2
framework = arduino
upload_speed = 115200
monitor_speed = 115200
board_build.flash_size = 8MB
build_flags = 
    -DBOARD_HAS_PSRAM
    -mfix-esp32-psram-cache-issue
board_build.embed_files = data/cert/x509_crt_bundle.bin

How do I fix this so that the correct RAM and flash sizes are shown? I want to make sure that platformIO does not think that my ESP32-S3 has smaller flash and RAM than it actually does and fail to upload for bigger-size projects. The board is currently running a small test project, but I will be uploading much bigger projects into it.

Any help would be highly appreciated, thank you very much.

These values are absolutely fine!
RAM is the on chip RAM (not the external SRAM)
FLASH is size of the APP / OTA partition

This line can be removed for ESP32-S3 and is only necessary for ESP32 (without S)

1 Like

Got it! Thank you very much for your help and for creating the .json settings files for all the different ESP32-S3 boards on your GitHub linked above.

1 Like