ESP32 file creation problem spiffs.bin

simonvassi88 · May 4, 2020, 11:43pm

hello
I noticed a problem when creating my spiffs.bin binary file, it is always the same size whatever its content its size is always the same 1.5M
therefore impossible to do a remote update with a spiffs.bin file

when i open the spiffs.bin file it is actually write half written the other half and clearly readable
so it didn’t properly create
what I do not understand because when I charge it in USB it works very well

would you have an idea to correct this problem ?
thank you in advance !

sorry if my english is not correct thank you google translate lol

mac os

[env:wemos_d1_mini32]

platform = espressif32
board = wemos_d1_mini32
framework = arduino

links = https://raw.githubusercontent.com/gauix4/serveur_solaire/master/spiffs.bin

maxgerhardt · May 6, 2020, 10:17pm

It is correct that the file size is always constant. It gets padded with filler values. Similiar question was already asked before: ESP8266 SIFFS.bin file to large. For ESP 32, it’s the same, see code.

github.com

espressif/arduino-esp32/blob/master/tools/gen_esp32part.py#L240


      
          
              for p in self:
                  if p.type == ptype and p.subtype == subtype:
                      yield p
              return
          
          def find_by_name(self, name):
              for p in self:
                  if p.name == name:
                      return p
              return None
          
          def verify(self):
              # verify each partition individually
              for p in self:
                  p.verify()
          
              # check on duplicate name
              names = [p.name for p in self]
              duplicates = set(n for n in names if names.count(n) > 1)

The size of the SPIFFS file is is chosen by the selected partition table, as documented. The default SPIFFS size is 0x170000 bytes aka crica 1.5MB.

github.com

espressif/arduino-esp32/blob/master/tools/partitions/default.csv

# Name,   Type, SubType, Offset,  Size, Flags
nvs,      data, nvs,     0x9000,  0x5000,
otadata,  data, ota,     0xe000,  0x2000,
app0,     app,  ota_0,   0x10000, 0x140000,
app1,     app,  ota_1,   0x150000,0x140000,
spiffs,   data, spiffs,  0x290000,0x160000,
coredump, data, coredump,0x3F0000,0x10000,

What exact error is occuring? SPIFFS upload shoult still be perfectly fine with OTA upload selected upload_protocol = espota and friends, docs)

Yes it did, see code above.

simonvassi88 · May 8, 2020, 12:39am

@maxgerhardt
a very big thank you to you for your really very clear and detailed answer I will see that
thanks again

for the OTA update I do not do it from IDE but from an HTTP server so if the SPIFFS.bin file is not correct necessarily that it does not work
my binary file and on the github the ESP32 will search for it to do its OTA update
I left the link at the top of the page you can see my binary export file from platfomio

so I managed to do what it takes for my partition I chose the file near configured 8M
board_build.partitions = default_8MB.csv

on the other hand I did not understand how to solve the problem SPIFFS.bin it is still incorrect

again thank you very much for your help!!

cordially

maxgerhardt · May 8, 2020, 10:07am

What exactly tells you that the SPIFFS file is invalid? Some kind of error? By using the mkspiffs tool I can read your SPIFFS.bin perfectly fine.

> mkspiffs.exe -l spiffs.bin
92764   /raphael-2.1.4.min.js
172812  /Chart.min.js
1650    /index.html
0       /testdata.txt
38111   /justgage.js
4021    /script.js
89493   /jquery-3.5.0.min.js

Unpacking it also gives valid data:

> mkspiffs.exe -u spiffs_unpacked spiffs.bin
/raphael-2.1.4.min.js    > ./spiffs_unpacked/raphael-2.1.4.min.js       size: 92764 Bytes
/Chart.min.js    > ./spiffs_unpacked/Chart.min.js       size: 172812 Bytes
/index.html      > ./spiffs_unpacked/index.html size: 1650 Bytes
/testdata.txt    > ./spiffs_unpacked/testdata.txt       size: 0 Bytes
/justgage.js     > ./spiffs_unpacked/justgage.js        size: 38111 Bytes
/script.js       > ./spiffs_unpacked/script.js  size: 4021 Bytes
/jquery-3.5.0.min.js     > ./spiffs_unpacked/jquery-3.5.0.min.js        size: 89493 Bytes

index.html

<!DOCTYPE HTML><!-- crèe par Simon Vassilieff :) -->
...

simonvassi88 · May 8, 2020, 3:25pm

@maxgerhardt
then it would seem that my OTA space is still not correct but I don’t know what its values correspond to (0x10000, 0x330000,)

# Name, Type, SubType, Offset, Size, Flags
nvs, data, nvs, 0x9000, 0x5000,
otadata, data, ota, 0xe000, 0x2000,
app0, app, ota_0, 0x10000, 0x330000,
app1, app, ota_1, 0x340000,0x330000,
spiffs, data, spiffs, 0x670000,0x190000,

I did not find information on this subject I found configuration already made but I would prefer to know how to change my values myself and what its values correspond to in memory

maxgerhardt · May 8, 2020, 3:41pm

Can you upload a PIO project with which the problem can be reproduced? If I don’t see an error message or the code thaht fails it’s really hard to give advice on what might be wrong.

simonvassi88 · May 8, 2020, 6:40pm

@maxgerhardt
HTTP_UPDATE_FAILD Error (-100): To less space

I just use the sketch ESP32httpUpdate.h
whether on the Arduino IDE or platformIO the problem is the same.
I have no problem with all my other ESP8266
the problem concerns only my ESP32

maxgerhardt · May 8, 2020, 7:26pm

Why are you using an 8MB partition table if a WeMos D1 mini 32 only has 4 MB flash?

Also, since you mentioned using ESP32httpUpdate.h which seems to come from this library they are recommending the official, integrated OTA way as well and say the project is deprecated.

The error To less space / HTTP_UE_TOO_LESS_SPACE seems to be coming from here:

github.com

suculent/esp32-http-update/blob/965b0bb7b40fa59f97f2dcaa842d2bb56bef4707/src/ESP32httpUpdate.cpp#L223-L242


      
          switch(code) {
          case HTTP_CODE_OK:  ///< OK (Start Update)
              if(len > 0) {
                  bool startUpdate = true;
                  if(spiffs) {
                      size_t spiffsSize = ((size_t) SPIFFS.totalBytes() - (size_t) SPIFFS.usedBytes());
                      if(len > (int) spiffsSize) {
                          DEBUG_HTTP_UPDATE("[httpUpdate] spiffsSize to low (%d) needed: %d\n", spiffsSize, len);
                          startUpdate = false;
                      }
                  } else {
                      //if(len > (int) ESP.getFreeSketchSpace()) {
                      //    DEBUG_HTTP_UPDATE("[httpUpdate] FreeSketchSpace to low (%d) needed: %d\n", ESP.getFreeSketchSpace(), len);
                      //    startUpdate = false;
                      //}
                  }
          
                  if(!startUpdate) {
                      _lastError = HTTP_UE_TOO_LESS_SPACE;
                      ret = HTTP_UPDATE_FAILED;

So it checks whether the available bytes in the SPIFFS partition is at least as big as the downlaoded file. Then however updating it with a full SPIFFS.bin partition file which will always be bigger than free space in the SPIFFS partition due to overhead makes no sense… It seems to be expecting differently formated update files, though it doesn’t say how to create those. This is also noted in SPIFFS mounting issue · Issue #2 · suculent/esp32-http-update · GitHub.

Backtracking through the code the update from the TCP stream happens here which just calls into the Updater class which in the SPIFFS case just writes it directly into flash.

So my best advices would be:

remove the length check in the if(spiffs) code section
use the official Arduino-ESP32 library (arduino-esp32/libraries/ArduinoOTA at master · espressif/arduino-esp32 · GitHub) instead

simonvassi88 · May 12, 2020, 9:18pm

@maxgerhardt
Hello
I do have a wemos32 I changed the flash memory to 8M
I saw this error but I tested by removing it does not change anything but it is because I did not organize the spiff space well now it all works thanks to you thank you very much

I use this library because it is very simple to use more it is very light unlike other libraries which take more memory

Thank you for your help, it was really useful! A big thank you to you