Thank you for the reply. I understand that it is not a problem of your making, TBH I just saw the previous mention and did not think too much about the actual cause.
Thank you for your ongoing efforts in PIO.
Sorry,
but it seems I’m too stupid.
Have read the whole thread and the the readme at github.
But I don’t understand where to put the “download_fs.py” script.
In my project there is no custom folder. Ok. I created two. One named “customs” and one named “Customs”. Because sometimes you are talking about “custom” and sometimes about “Custom” I’m totaly confused. What is the correct name? Linux filesystems are case sensitive always!
OK. I added the “extra_scripts” statements to platformio.ini. But nothing changed.
When looking at project tasks I did not see any sort of custom like in “usage” as described here:
I took a look at platformio documentation. I found “extra_scripts” but I did not found where to place them :(.
I’m completly lost …
In the root of the project, at the same level as the platformio.ini.
You then add extra_scripts = download_fs.py to your environment in the platformio.ini.
After reloading the project tasks, you should then have a new task under Custom → Download Filesystem.
No custom_.. options are needed in the platformio.ini if you’re fine with the default.
@maxgerhardt Got it. Thank you very much for clarifying.
Since your script does not support LittleFS on ESP32 I had a look at
Tasmota as described by Jason2866
Since I’m not using Tasmota, i removed some lines to get it work with LittleFS
Processing esp32doit-devkit-v1 (platform: espressif32; board: esp32doit-devkit-v1; framework: arduino)
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Verbose mode can be enabled via `-v, --verbose` option
CONFIGURATION: https://docs.platformio.org/page/boards/espressif32/esp32doit-devkit-v1.html
PLATFORM: Espressif 32 (6.4.0) > DOIT ESP32 DEVKIT V1
HARDWARE: ESP32 240MHz, 320KB RAM, 4MB Flash
DEBUG: Current (cmsis-dap) External (cmsis-dap, esp-bridge, esp-prog, iot-bus-jtag, jlink, minimodule, olimex-arm-usb-ocd, olimex-arm-usb-ocd-h, olimex-arm-usb-tiny-h, olimex-jtag-tiny, tumpa)
PACKAGES:
- framework-arduinoespressif32 @ 3.20011.230801 (2.0.11)
- tool-esptoolpy @ 1.40501.0 (4.5.1)
- toolchain-xtensa-esp32 @ 8.4.0+2021r2-patch5
LDF: Library Dependency Finder -> https://bit.ly/configure-pio-ldf
LDF Modes: Finder ~ chain, Compatibility ~ soft
Found 52 compatible libraries
Scanning dependencies...
Dependency Graph
|-- ESPAsync_WiFiManager_Lite @ 1.10.5
|-- ESP Async WebServer @ 1.2.3+sha.8e9c5a6
|-- ESPAsyncDNSServer @ 1.0.0+sha.3c6627d
|-- AsyncMqttClient @ 0.9.0
|-- AsyncElegantOTA @ 2.2.8
|-- WiFi @ 2.0.0
Building in release mode
command_download_fs(["downloadfs"], [])
Entrypoint
Executing flash download command to read partition table.
['/data/PlatformIO/penv/bin/python', '/data/PlatformIO/packages/tool-esptoolpy/esptool.py', '--chip', 'esp32', '--port', '/dev/ttyUSB0', '--baud', '921600', '--before', 'default_reset', '--after', 'hard_reset', 'read_flash', '0x8000', '0x1000', '/data/PlatformIO/Projects/RainWaterControl.ESP32/partition_table_from_flash.bin']
esptool.py v4.5.1
Serial port /dev/ttyUSB0
Connecting....
Chip is ESP32-D0WDQ6 (revision v1.0)
Features: WiFi, BT, Dual Core, 240MHz, VRef calibration in efuse, Coding Scheme None
Crystal is 40MHz
MAC: 3c:71:bf:4d:c1:ac
Uploading stub...
Running stub...
Stub running...
Changing baud rate to 921600
Changed.
4096 (100 %)
4096 (100 %)
Read 4096 bytes at 0x00008000 in 0.1 seconds (629.5 kbit/s)...
Hard resetting via RTS pin...
Partition data:
type: 0x82
address: 0x290000
size: 0x160000
Retrieving filesystem info for esp32.
Partition file: /data/PlatformIO/packages/framework-arduinoespressif32/tools/partitions/default.csv
Executing flash download command to read partition table.
['/data/PlatformIO/penv/bin/python', '/data/PlatformIO/packages/tool-esptoolpy/esptool.py', '--chip', 'esp32', '--port', '/dev/ttyUSB0', '--baud', '921600', '--before', 'default_reset', '--after', 'hard_reset', 'read_flash', '0x8000', '0x1000', '/data/PlatformIO/Projects/RainWaterControl.ESP32/partition_table_from_flash.bin']
esptool.py v4.5.1
Serial port /dev/ttyUSB0
Connecting....
Chip is ESP32-D0WDQ6 (revision v1.0)
Features: WiFi, BT, Dual Core, 240MHz, VRef calibration in efuse, Coding Scheme None
Crystal is 40MHz
MAC: 3c:71:bf:4d:c1:ac
Uploading stub...
Running stub...
Stub running...
Changing baud rate to 921600
Changed.
4096 (100 %)
4096 (100 %)
Read 4096 bytes at 0x00008000 in 0.1 seconds (641.4 kbit/s)...
Hard resetting via RTS pin...
Partition data:
type: 0x82
address: 0x290000
size: 0x160000
Parsed FS info: FS type FSType.SPIFFS Start 0x290000 Len 1441792 Page size 256 Block size 4096 Tool: /data/PlatformIO/packages/tool-mklittlefs/mklittlefs
FS type FSType.SPIFFS Start 0x290000 Len 1441792 Page size 256 Block size 4096 Tool: /data/PlatformIO/packages/tool-mklittlefs/mklittlefs
Executing flash download command.
['/data/PlatformIO/penv/bin/python', '/data/PlatformIO/packages/tool-esptoolpy/esptool.py', '--chip', 'esp32', '--port', '/dev/ttyUSB0', '--baud', '921600', '--before', 'default_reset', '--after', 'hard_reset', 'read_flash', '0x290000', '0x160000', '/data/PlatformIO/Projects/RainWaterControl.ESP32/.pio/build/esp32doit-devkit-v1/downloaded_fs_0x290000_0x160000.bin']
esptool.py v4.5.1
Serial port /dev/ttyUSB0
Connecting.....
Chip is ESP32-D0WDQ6 (revision v1.0)
Features: WiFi, BT, Dual Core, 240MHz, VRef calibration in efuse, Coding Scheme None
Crystal is 40MHz
MAC: 3c:71:bf:4d:c1:ac
Uploading stub...
Running stub...
Stub running...
Changing baud rate to 921600
Changed.
1441792 (100 %)
1441792 (100 %)
Read 1441792 bytes at 0x00290000 in 18.0 seconds (642.4 kbit/s)...
Hard resetting via RTS pin...
Executing extraction command: "/data/PlatformIO/packages/tool-mklittlefs/mklittlefs" -b 4096 -p 256 --unpack "unpacked_fs" "/data/PlatformIO/Projects/RainWaterControl.ESP32/.pio/build/esp32doit-devkit-v1/downloaded_fs_0x290000_0x160000.bin"
lfs error:3409: Unsupported name_max (255 > 32)
drd.dat > ./unpacked_fs/drd.dat size: 0 Bytes
mrd.dat > ./unpacked_fs/mrd.dat size: 4 Bytes
wm_config.bak > ./unpacked_fs/wm_config.bak size: 236 Bytes
wm_config.dat > ./unpacked_fs/wm_config.dat size: 236 Bytes
wm_cp.bak > ./unpacked_fs/wm_cp.bak size: 4 Bytes
wm_cp.dat > ./unpacked_fs/wm_cp.dat size: 4 Bytes
wm_cred.bak > ./unpacked_fs/wm_cred.bak size: 210 Bytes
wm_cred.dat > ./unpacked_fs/wm_cred.dat size: 210 Bytes
Unpacked filesystem.
Extracted 8 file(s) from filesystem.
Oh that’s great, I wanted to get the changes tha Jason did with reading the partition table. I just PRed your fork back, looking great!
@maxgerhardt thanks for your great work on this download utility!
Should I expect this to work with LittleFS and the esp32-s3? So far I’m getting an error but before trying to troubleshoot I wanted to confirm whether this failure is expected for LittleFS with esp32-s3.
Here’s the error output (using PlatformIO with VS Code):
* Executing task: C:\Users\dgerm\.platformio\penv\Scripts\platformio.exe run --target downloadfs --environment um_pros3
Processing um_pros3 (platform: espressif32; board: um_pros3; framework: arduino)
----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Verbose mode can be enabled via `-v, --verbose` option
CONFIGURATION: https://docs.platformio.org/page/boards/espressif32/um_pros3.html
PLATFORM: Espressif 32 (6.1.0) > Unexpected Maker PROS3
HARDWARE: ESP32S3 240MHz, 320KB RAM, 16MB Flash
DEBUG: Current (cmsis-dap) External (cmsis-dap, esp-bridge, esp-builtin, esp-prog, iot-bus-jtag, jlink, minimodule, olimex-arm-usb-ocd, olimex-arm-usb-ocd-h, olimex-arm-usb-tiny-h, olimex-jtag-tiny, tumpa)
PACKAGES:
- framework-arduinoespressif32 @ 3.20007.0 (2.0.7)
- tool-esptoolpy @ 1.40500.0 (4.5.0)
- toolchain-riscv32-esp @ 8.4.0+2021r2-patch5
- toolchain-xtensa-esp32s3 @ 8.4.0+2021r2-patch5
LDF: Library Dependency Finder -> https://bit.ly/configure-pio-ldf
LDF Modes: Finder ~ chain, Compatibility ~ soft
Found 39 compatible libraries
Scanning dependencies...
Dependency Graph
|-- genieArduino @ 1.5.3
|-- UMS3 Helper @ 1.0.1
|-- MAX31850 DallasTemp @ 1.1.6
|-- OneWire @ 2.3.8
|-- LittleFS @ 2.0.0
|-- SureStep
|-- WelderForm
Building in release mode
command_download_fs(["downloadfs"], [])
Entrypoint
Auto-detected: COM15
Executing flash download command to read partition table.
['C:\\Users\\dgerm\\.platformio\\penv\\Scripts\\python.exe', 'C:\\Users\\dgerm\\.platformio\\packages\\tool-esptoolpy\\esptool.py', '--chip', 'esp32s3', '--port', 'COM15', '--baud', '460800', '--before', 'default_reset', '--after', 'hard_reset', 'read_flash', '0x8000', '0x1000', 'C:\\Users\\dgerm\\Documents\\Git\\Tongrun\\fusion-welder\\Firmware\\FusionWelding\\partition_table_from_flash.bin']
esptool.py v4.5
Serial port COM15
Connecting...
Chip is ESP32-S3 (revision v0.2)
Features: WiFi, BLE
Crystal is 40MHz
MAC: ec:da:3b:41:b4:38
Uploading stub...
Running stub...
Stub running...
Changing baud rate to 460800
Changed.
4096 (100 %)
4096 (100 %)
Read 4096 bytes at 0x00008000 in 0.4 seconds (88.4 kbit/s)...
Hard resetting via RTS pin...
Partition data:
Retrieving filesystem info for esp32s3.
Partition file: C:\Users\dgerm\.platformio\packages\framework-arduinoespressif32\tools\partitions\default_16MB.csv
Executing flash download command to read partition table.
['C:\\Users\\dgerm\\.platformio\\penv\\Scripts\\python.exe', 'C:\\Users\\dgerm\\.platformio\\packages\\tool-esptoolpy\\esptool.py', '--chip', 'esp32s3', '--port', 'COM15', '--baud', '460800', '--before', 'default_reset', '--after', 'hard_reset', 'read_flash', '0x8000', '0x1000', 'C:\\Users\\dgerm\\Documents\\Git\\Tongrun\\fusion-welder\\Firmware\\FusionWelding\\partition_table_from_flash.bin']
esptool.py v4.5
Serial port COM15
Connecting...
Chip is ESP32-S3 (revision v0.2)
Features: WiFi, BLE
Crystal is 40MHz
MAC: ec:da:3b:41:b4:38
Uploading stub...
Running stub...
Stub running...
Changing baud rate to 460800
Changed.
4096 (100 %)
4096 (100 %)
Read 4096 bytes at 0x00008000 in 0.4 seconds (91.8 kbit/s)...
Hard resetting via RTS pin...
Partition data:
*** [downloadfs] KeyError : 'SPIFFS_START'
Traceback (most recent call last):
File "C:\Users\dgerm\.platformio\packages\tool-scons\scons-local-4.7.0\SCons\Action.py", line 1441, in execute
result = self.execfunction(target=target, source=rsources, env=env)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\dgerm\Documents\Git\Tongrun\fusion-welder\Firmware\FusionWelding\download_fs.py", line 311, in command_download_fs
info = get_fs_type_start_and_length()
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\dgerm\Documents\Git\Tongrun\fusion-welder\Firmware\FusionWelding\download_fs.py", line 222, in get_fs_type_start_and_length
return SPIFFSInfo(env["SPIFFS_START"], env["SPIFFS_SIZE"], env["SPIFFS_PAGE"], env["SPIFFS_BLOCK"])
~~~^^^^^^^^^^^^^^^^
File "C:\Users\dgerm\.platformio\packages\tool-scons\scons-local-4.7.0\SCons\Environment.py", line 587, in __getitem__
return self._dict[key]
~~~~~~~~~~^^^^^
KeyError: 'SPIFFS_START'
LittleFS should have been correctly detected, but I guess the script is now outdated with respect to the espressif32 platform. I’ll look into it.
Thank you! Let me know if you need any additional info or help from me.
@maxgerhardt Maybe you can reuse parts from here
The base was your script. It does work perfectly for all esp32x and the esp8266 with LittleFS. Don’t know if SPIFFS is still working. I have removed SPIFFS support in my framework builds…
To get it working I had to compile my own mklittlefs for esp32. Is installed and used when using my Platformio espressif32 fork.
script is updated. give it a try.
Thank you! I won’t have access to my test setup for about a week but I’ll give it a try as soon as I can.
Thank you so much for updating the script, Jason. I apologize for the long delay. We tested it and now there’s a different “FS_START” error. Please let me know what else I can do to help troubleshoot.
Executing task: C:\Users\barre\.platformio\penv\Scripts\platformio.exe run --target downloadfs --environment um_pros3
Processing um_pros3 (platform: espressif32; board: um_pros3; framework: arduino)
--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Verbose mode can be enabled via `-v, --verbose` option
CONFIGURATION: https://docs.platformio.org/page/boards/espressif32/um_pros3.html
PLATFORM: Espressif 32 (6.1.0) > Unexpected Maker PROS3
HARDWARE: ESP32S3 240MHz, 320KB RAM, 16MB Flash
DEBUG: Current (cmsis-dap) External (cmsis-dap, esp-bridge, esp-builtin, esp-prog, iot-bus-jtag, jlink, minimodule, olimex-arm-usb-ocd, olimex-arm-usb-ocd-h, olimex-arm-usb-tiny-h, olimex-jtag-tiny, tumpa)
PACKAGES:
- framework-arduinoespressif32 @ 3.20007.0 (2.0.7)
- tool-esptoolpy @ 1.40500.0 (4.5.0)
- toolchain-riscv32-esp @ 8.4.0+2021r2-patch5
- toolchain-xtensa-esp32s3 @ 8.4.0+2021r2-patch5
LDF: Library Dependency Finder -> https://bit.ly/configure-pio-ldf
LDF Modes: Finder ~ chain, Compatibility ~ soft
Found 47 compatible libraries
Scanning dependencies...
Dependency Graph
|-- genieArduino @ 1.5.3
|-- UMS3 Helper @ 1.0.1
|-- MAX31850 DallasTemp @ 1.1.6
|-- OneWire @ 2.3.8
|-- EspSoftwareSerial @ 8.2.0
|-- ghostl @ 1.0.1+sha.52a206b
|-- RTClib @ 2.1.4
|-- Adafruit BusIO @ 1.16.0
|-- LabelPrinter
|-- LittleFS @ 2.0.0
|-- SPI @ 2.0.0
|-- SureStep
|-- WelderForm
Building in release mode
command_download_fs(["downloadfs"], [])
Retrieving filesystem info for esp32s3.
Auto-detected: COM18
esptool.py v4.5
Serial port COM18
Connecting...
Chip is ESP32-S3 (revision v0.2)
Features: WiFi, BLE
Crystal is 40MHz
MAC: ec:da:3b:41:b4:50
Uploading stub...
Running stub...
Stub running...
A fatal error occurred: Corrupt data, expected 0x1000 bytes but received 0x7d2 bytes
*** [downloadfs] KeyError : 'FS_START'
Traceback (most recent call last):
File "C:\Users\barre\.platformio\packages\tool-scons\scons-local-4.7.0\SCons\Action.py", line 1441, in execute
result = self.execfunction(target=target, source=rsources, env=env)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\barre\ImpactGit\welder\Firmware\Welding\download_fs.py", line 262, in command_download_fs
info = get_fs_type_start_and_length()
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\barre\ImpactGit\welder\Firmware\Welding\download_fs.py", line 182, in get_fs_type_start_and_length
return FS_Info(env["FS_START"], env["FS_SIZE"], env["FS_PAGE"], env["FS_BLOCK"])
~~~^^^^^^^^^^^^
File "C:\Users\barre\.platformio\packages\tool-scons\scons-local-4.7.0\SCons\Environment.py", line 587, in __getitem__
return self._dict[key]
~~~~~~~~~~^^^^^
KeyError: 'FS_START'
reduce speed for esptool.py You encountered a transmission error.
@maxgerhardt @jason2866
I work with dgermana and have tested the updated script and there still appears to be an issue. I tried manually decreasing the download speed all the way down to 19200 but still encountered the same error messages.
I tried to look into the KeyError, so I used command pio run -t envdump from the PlatformIO CLI terminal and didn’t see any values named FS_START, FS_SIZE, FS_PAGE, or FS_BLOCK.
Is this a problem with the script? Or is there additional configuration I need to add to my platformio.ini? We added the line board_build.filesystem = littlefs but that’s it.
Thanks in advance!
Yes you have to add
extra_scripts = download_fs.py
Thanks, Jason. We have that line in our platformio.ini as well. Here’s our complete platformio.ini file:
; PlatformIO Project Configuration File
;
; Build options: build flags, source filter
; Upload options: custom upload port, speed and extra flags
; Library options: dependencies, extra library storages
; Advanced options: extra scripting
;
; Please visit documentation for the other options and examples
; https://docs.platformio.org/page/projectconf.html
[env:um_pros3]
platform = espressif32
board = um_pros3
framework = arduino
lib_deps =
4dsystems/genieArduino@^1.5.3
unexpectedmaker/UMS3 Helper@^1.0.1
adafruit/MAX31850 DallasTemp@^1.1.6
paulstoffregen/OneWire@^2.3.8
plerup/EspSoftwareSerial@^8.2.0
https://github.com/dok-net/ghostl
adafruit/RTClib@^2.1.4
adafruit/Adafruit BusIO@^1.16.0
adafruit/Adafruit INA228 Library@^1.0.0
build_flags =
-DARDUINO_USB_MODE=1 ; Enable ESP32-S3 to automatically reboot after flashing new firmware
; -DCORE_DEBUG_LEVEL=2 ; Print ESP-LOG messages to serial (warning level)
-DCORE_DEBUG_LEVEL=3 ; Print ESP-LOG messages to serial (info level)
;-DCORE_DEBUG_LEVEL=4 ; Print ESP-LOG messages to serial (debug level)
upload_flags = --no-stub
monitor_filters = esp32_exception_decoder
board_build.filesystem = littlefs
; Adds a file system download script to the "Custom" menu in the PlatformIO Project Tasks
extra_scripts = download_fs.py
custom_unpack_directory = data_downloaded ; The directory to unpack the file system to
As long as you get this it can’t work
A fatal error occurred: Corrupt data, expected 0x1000 bytes but received 0x7d2 bytes
There is the info where to find the LittleFS partition. Since the data is not successfully fetched there is no way. There is something really wrong.
Can you download the whole firmware with esptool.py? If not you have to find the reason why.
Why do you disable the stub in your setup?