ESP-IDF unable to mount/see files in PlatformIO SPIFFS image

PlatformIO IDE
,
1

I’m trying to create a SPIFFS-based file server, using the PlatformIO IDE, running the esp32 (ESP-IDF) framework. To my knowledge, I’ve completed all the required steps:

My partitions.csv file contains a partition for spiffs:

# Name,   Type, SubType, Offset,  Size, Flags
nvs,          data,  nvs,          ,        0x6000,
phy_init,   data,  phy,          ,        0x1000,
factory,     app,   factory,     ,        0x200000,
storage,    data, spiffs,        ,        0x100000,

My CMakeLists.txt file contains the instruction spiffs_create_partition_image(spiffs data)

My files to upload are in a directory named data

When I select ‘Build Filesystem Image’ and ‘Upload Filesystem Image’, everything appears correct, and the image is written to the correct memory location (0x207000).

Processing esp32dev (platform: espressif32; framework: espidf; board: esp32dev)
------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------Verbose mode can be enabled via `-v, --verbose` option
CONFIGURATION: https://docs.platformio.org/page/boards/espressif32/esp32dev.html
PLATFORM: Espressif 32 (3.3.2) > Espressif ESP32 Dev Module
HARDWARE: ESP32 240MHz, 320KB RAM, 4MB Flash
DEBUG: Current (esp-prog) External (esp-prog, iot-bus-jtag, jlink, minimodule, olimex-arm-usb-ocd, olimex-arm-usb-ocd-h, olimex-arm-usb-tiny-h, olimex-jtag-tiny, tumpa)
PACKAGES:
 - framework-espidf 3.40300.0 (4.3.0)
 - tool-cmake 3.16.4
 - tool-esptoolpy 1.30100.210531 (3.1.0)
 - tool-idf 1.0.1
 - tool-mconf 1.4060000.20190628 (406.0.0)
 - tool-mkspiffs 2.230.0 (2.30)
 - tool-ninja 1.9.0
 - toolchain-esp32ulp 1.22851.191205 (2.28.51)
 - toolchain-xtensa32 2.80400.210211 (8.4.0)
Reading CMake configuration...
LDF: Library Dependency Finder -> http://bit.ly/configure-pio-ldf
LDF Modes: Finder ~ chain, Compatibility ~ soft
Found 0 compatible libraries
Scanning dependencies...
No dependencies
Building in release mode
Building SPIFFS image from 'data' directory to .pio\build\esp32dev\spiffs.bin
/favicon.ico
/HereIsText.txt
/testimg.png
/upload_script.html
================================================================================= [SUCCESS] Took 14.05 seconds =================================================================================

Processing esp32dev (platform: espressif32; framework: espidf; board: esp32dev)
------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------Verbose mode can be enabled via `-v, --verbose` option
CONFIGURATION: https://docs.platformio.org/page/boards/espressif32/esp32dev.html
PLATFORM: Espressif 32 (3.3.2) > Espressif ESP32 Dev Module
HARDWARE: ESP32 240MHz, 320KB RAM, 4MB Flash
DEBUG: Current (esp-prog) External (esp-prog, iot-bus-jtag, jlink, minimodule, olimex-arm-usb-ocd, olimex-arm-usb-ocd-h, olimex-arm-usb-tiny-h, olimex-jtag-tiny, tumpa)
PACKAGES:
 - framework-espidf 3.40300.0 (4.3.0)
 - tool-cmake 3.16.4
 - tool-esptoolpy 1.30100.210531 (3.1.0)
 - tool-idf 1.0.1
 - tool-mconf 1.4060000.20190628 (406.0.0)
 - tool-mkspiffs 2.230.0 (2.30)
 - tool-ninja 1.9.0
 - toolchain-esp32ulp 1.22851.191205 (2.28.51)
 - toolchain-xtensa32 2.80400.210211 (8.4.0)
Reading CMake configuration...
LDF: Library Dependency Finder -> http://bit.ly/configure-pio-ldf
LDF Modes: Finder ~ chain, Compatibility ~ soft
Found 0 compatible libraries
Scanning dependencies...
No dependencies
Building in release mode
Building SPIFFS image from 'data' directory to .pio\build\esp32dev\spiffs.bin
/favicon.ico
/HereIsText.txt
/testimg.png
/upload_script.html
Looking for upload port...
Auto-detected: COM16
Uploading .pio\build\esp32dev\spiffs.bin
esptool.py v3.1
Serial port COM16
Connecting......
Chip is ESP32-D0WD-V3 (revision 3)
Features: WiFi, BT, Dual Core, 240MHz, VRef calibration in efuse, Coding Scheme None
Crystal is 40MHz
MAC: 94:b9:7e:40:24:b8
Uploading stub...
Running stub...
Stub running...
Changing baud rate to 460800
Changed.
Configuring flash size...
Auto-detected Flash size: 4MB
Flash will be erased from 0x00207000 to 0x00306fff...
Compressed 1048576 bytes to 58234...
Writing at 0x00207000... (25 %)
Writing at 0x0020f666... (50 %)
Writing at 0x002189b9... (75 %)
Writing at 0x0022338b... (100 %)
Wrote 1048576 bytes (58234 compressed) at 0x00207000 in 7.7 seconds (effective 1085.1 kbit/s)...
Hash of data verified.

Leaving...
Hard resetting via RTS pin...
================================================================================= [SUCCESS] Took 23.57 seconds =================================================================================

However, when I attempt to use the following function (TAG is #defined as “file_server”):

esp_err_t init_spiffs(void)
{
    ESP_LOGI(TAG, "Initializing SPIFFS");

    esp_vfs_spiffs_conf_t conf = {
      .base_path = "/spiffs",
      .partition_label = NULL,
      .max_files = 5,   // This decides the maximum number of files that can be created on the storage
      .format_if_mount_failed = true
    };

    esp_err_t ret = esp_vfs_spiffs_register(&conf);
    if (ret != ESP_OK) {
        if (ret == ESP_FAIL) {
            ESP_LOGE(TAG, "Failed to mount or format filesystem");
        } else if (ret == ESP_ERR_NOT_FOUND) {
            ESP_LOGE(TAG, "Failed to find SPIFFS partition");
        } else {
            ESP_LOGE(TAG, "Failed to initialize SPIFFS (%s)", esp_err_to_name(ret));
        }
        return ESP_FAIL;
    }

    size_t total = 0, used = 0;
    ret = esp_spiffs_info(NULL, &total, &used);
    if (ret != ESP_OK) {
        ESP_LOGE(TAG, "Failed to get SPIFFS partition information (%s)", esp_err_to_name(ret));
        return ESP_FAIL;
    }

    ESP_LOGI(TAG, "Partition size: total: %d, used: %d", total, used);

    FILE* f = fopen("/spiffs/HereIsText.txt", "r");
    if(f == NULL){
        ESP_LOGE(TAG, "Failed to open BANANA file");
    }
    fclose(f);
    return ESP_OK;
}

I get the resulting output:

␛[0;32mI (1570) file_server: Initializing SPIFFS␛[0m
␛[0;33mW (1580) SPIFFS: mount failed, -10025. formatting...␛[0m
␛[0;32mI (12630) file_server: Partition size: total: 901953, used: 0␛[0m
␛[0;31mE (12690) file_server: Failed to open BANANA file␛[0m

As shown, SPIFFS fails to mount and formats the memory, after which I am unable to load any of my files. As indicated by used: 0, everything seems to get wiped. Am I missing a step somewhere?

2

Half my problem has now been fixed; I replaced 0x100000 in my partitions.csv with 1M, and this successfully mounted after rebuilding/reuploading. However, it still shows 0 bytes used, even though there should be about 64KB of files in the uploaded image.

Edit, it appears I was mistaken about this, if I do the uploading in a different order it still fails to mount.

3
  1. There’s a pre-prepared example for ESP-IDF + SPIFFS at platform-espressif32/examples/espidf-storage-spiffs at develop · platformio/platform-espressif32 · GitHub, does that work?
  2. Have you set the same partitions.csv table as your partition table in the menuconfig?
  3. Have you set it in the platformio.ini?
4

I tried importing the example and it did indeed work, which I find confusing as my SPIFFS code was virtually copy-pasted from that exact example - I even made a minimum project using just that function. Yes, my partitions.csv table is set in both menuconfig and platformio.ini, and I can see the file image being written to the expected memory location. Partition table and SPIFFS settings in the sdkconfig are identical, only difference is my custom table is just called ‘partitions.csv’

5

A difference between your partition table and the example one are missing explicit offsets for the first 3 partitions, I wonder if that influences anything. But if ESP-IDF was able to boot your application, I guess it was okay…

Can you upload your whole current project for inspection?

6

Just as a .zip file?

7

Whatever works. ZIP on GoogleDrive, repo on Github, …

8

Here’s the minimum example that reproduces the issue, full project has some sensitive stuff: GitHub - ZacDaMan/ESP-IDF-SPIFFS-test

9

Yeah something is definitely wrong with the SPIFFS upload address.

>pio run -t uploadfs
Wrote 1048576 bytes (52274 compressed) at 0x00207000 in 7.2 seconds (effective 1160.7 kbit/s)...

␛[0;32mI (51) boot: Partition Table:␛[0m
␛[0;32mI (55) boot: ## Label            Usage          Type ST Offset   Length␛[0m
␛[0;32mI (62) boot:  0 nvs              WiFi data        01 02 00009000 00006000␛[0m
␛[0;32mI (70) boot:  1 phy_init         RF data          01 01 0000f000 00001000␛[0m
␛[0;32mI (77) boot:  2 factory          factory app      00 00 00010000 00200000␛[0m
␛[0;32mI (84) boot:  3 storage          Unknown data     01 82 00210000 00100000␛[0m
...
␛[0;32mI (10305) example: Initializing SPIFFS␛[0m
␛[0;33mW (10305) SPIFFS: mount failed, -10025. formatting...␛[0m

The firmware thinks SPIFFS is @ 0x00210000, PIO uploads to 0x00207000. Let me see why that is.

If the mounting of the expected SPIFFS partition at the address fails, it does a format. After that there 0 used bytes. This is expected. The mounting fail is the problem.

10

Could it be something related to how factory always gets written to an offset that is a multiple of 0x10000 even if you enter something else? Looks like spiffs might have automatically had the same 0x3000 offset added that factory did

11

There is a logic bug somewhere. When I add a print(spiffs) statement before

I get

 {'name': 'storage', 'type': 'data', 'subtype': 'spiffs', 'offset': 2125824, 'size': '1M', 'flags': ''}

With offset being 0x207000. Since your partition table has left out all offset information, PIO will be accumulating previous partition sizes to get the current offset.

The ESP-IDF firmware has interpreted that partition table as the SPIFFS partition being at 0x00210000.

One of them is wrong in the address calculation. The problem will likely be solved by adding explicit offset info for now, but that doesn’t solve the underlying cause.

1 Like
12

I tried explicitly setting the factory offset to 0x10000 and that seems to have fixed the problem - as you say though it doesn’t fully explain what caused the problem to begin with; I would guess whatever writes the SPIFFS image simply adds up the memory sizes of previous partitions and fails to take into account the offset that gets added if factory isn’t already offset by a multiple of 0x10000?

13

Ah there’s a alignment process? “Factory” must be aligned at a 0x10000 boundary? Do you have a reference to that in ESP-IDF docs?

14

I can’t remember where I found it, but I came across something stating that any partition with type app must be written to an offset that is a multiple of 0x10000, and it will be automatically adjusted if that is not the case. Will try and find the source for that

15

Ah yes here we go: Partition Tables - ESP32 - — ESP-IDF Programming Guide latest documentation

“Partitions of type app have to be placed at offsets aligned to 0x10000 (64K). If you leave the offset field blank, gen_esp32part.py will automatically align the partition. If you specify an unaligned offset for an app partition, the tool will return an error.”

16

I see. I’ll be opening a bug report in Issues · platformio/platform-espressif32 · GitHub to get partition table parsing fixed + aligned with ESP-IDF. The fix in PlatformIO for now is using explicit offset information.

1 Like
17

Issue is open in Partition table parsing wrong · Issue #627 · platformio/platform-espressif32 · GitHub, until then the workaround is valid.

1 Like