How to define LittleFS partition, build image and flash it on ESP32?

viktak · May 8, 2020, 1:40pm

Hi,
where exactly do you put these scripts (LittleFSBuilder.py and mklittlefs.py)?

Sorry about the silly question, but I haven’t worked with scripts in Platformio…

Thank you!!!

pfeerick · May 9, 2020, 8:25am

Anywhere it can be accessed! Keep in mind you need to know where it goes when you add the extra_scripts line in your platformio… so you probably want it in the root of the project folder, or in a extras subfolder… etc.

viktak · May 9, 2020, 12:49pm

Thanks for the quick reply. However, I still cannot get it to work…:

So, I put the two files in the root, and trying to upload the FS, I get the following error message:

Processing esp12e (platform: espressif8266; framework: arduino; board: esp12e)
---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

Verbose mode can be enabled via `-v, --verbose` option
CONFIGURATION: https://docs.platformio.org/page/boards/espressif8266/esp12e.html
PLATFORM: Espressif 8266 2.5.1 > Espressif ESP8266 ESP-12E
HARDWARE: ESP8266 80MHz, 80KB RAM, 4MB Flash
PACKAGES:
 - framework-arduinoespressif8266 3.20701.0 (2.7.1)
 - tool-esptool 1.413.0 (4.13)
 - tool-esptoolpy 1.20800.0 (2.8.0)
 - tool-mkspiffs 1.200.0 (2.0)
 - toolchain-xtensa 2.40802.200502 (4.8.2)
LDF: Library Dependency Finder -> http://bit.ly/configure-pio-ldf
LDF Modes: Finder ~ chain, Compatibility ~ soft
Found 36 compatible libraries
Scanning dependencies...
Dependency Graph
|-- <ArduinoJson> 6.15.1
|-- <Time> 1.6
|-- <Ticker> 1.0
|-- <Timezone> 1.2.4
|   |-- <Time> 1.6
|-- <PCF8574_ESP> 1.0.10
|   |-- <Wire> 1.0
|-- <EspSoftwareSerial> 6.8.1
|-- <TinyGPSPlus> 1.0.2
|-- <AccelStepper> 1.59
|-- <ESP8266WiFi> 1.0
|-- <ArduinoOTA> 1.0
|   |-- <ESP8266WiFi> 1.0
|   |-- <ESP8266mDNS> 1.2
|   |   |-- <ESP8266WiFi> 1.0
|-- <EEPROM> 1.0
|-- <ESP8266mDNS> 1.2
|   |-- <ESP8266WiFi> 1.0
|-- <ESP8266WebServer> 1.0
|   |-- <ESP8266WiFi> 1.0
|-- <LittleFS(esp8266)> 0.1.0
|-- <PubSubClient> 1.99.1
|-- <SD(esp8266)> 2.0.0
|   |-- <SDFS> 0.1.0
|   |   |-- <ESP8266SdFat> 1.1.0
|   |   |   |-- <SPI> 1.0
|   |   |-- <SPI> 1.0
|-- <SPI> 1.0
|-- <Wire> 1.0
Building in release mode
Building SPIFFS image from 'data' directory to .pio\build\esp12e\spiffs.bin
Python was not found but can be installed from the Microsoft Store: https://go.microsoft.com/fwlink?linkID=2082640scons: *** [.pio\build\esp12e\spiffs.bin] Error 9009
========================================================================================= [FAILED] Took 1.52 seconds =========================================================================================
The terminal process terminated with exit code: 1

Of course, Python IS installed on my system… It is also in my (user) path variable:

python-env

Also, in Visual Studio Code, in the terminal I can access it normally:

Any idea what I might be missing? This is driving me crazy…

BTW: I am using a Windows 10 Pro, if it makes any difference…

maxgerhardt · May 9, 2020, 12:58pm

What’s the exact content of your platformio.ini?

viktak · May 9, 2020, 1:20pm

platformio.ini:

[env:esp12e]
platform = espressif8266
framework = arduino
board = esp12e

extra_scripts = LittleFSBuilder.py

lib_deps =
            ArduinoJson
            Time
            Ticker
            Timezone
            PCF8574_ESP
            EspSoftwareSerial
            1655                #TinyGPSPlus
            AccelStepper

upload_protocol = espota
upload_port = 192.168.1.201

And this is how the project folder looks:
project-struct

maxgerhardt · May 9, 2020, 2:36pm

But does it also work via python3? If the content of your LittleFSBuild.py is the same from the post above,

Import("env")
env.Replace( MKSPIFFSTOOL='python3" "' + env.get("PROJECT_DIR") + '/mklittlefs.py' )

the script will invoke python3, not python… Though I don’t know why the double quotes are exactly this way, you might have to delete them so that it results in python3 <project_dir>/mklittefs.py (or put the whole path to the script in quotes in case the path contains spaces)

Yuriy · May 10, 2020, 1:24am

I tried the proposed option, but the resulting image (I guess) is not compatible with LittleFSv2 in Arduino.
What worked for me is a similar substitution:

Import (“env”)
env.Replace (MKSPIFFSTOOL = env.get (“PROJECT_DIR”) + ‘/mklittlefs.exe’)

but with - it’s parameters compatible with mkspiffstool so it’s just work

viktak · May 10, 2020, 8:16am

I changed python3 to python in the script as you suggested, and now it seems to be running OK, but I get an error message saying that there is not enough space…:

> Executing task in folder stepper-test: C:\Users\Viktor\.platformio\penv\Scripts\platformio.exe run --target uploadfs <


Processing esp12e (platform: espressif8266; framework: arduino; board: esp12e)
---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

Verbose mode can be enabled via `-v, --verbose` option
CONFIGURATION: https://docs.platformio.org/page/boards/espressif8266/esp12e.html
PLATFORM: Espressif 8266 2.5.1 > Espressif ESP8266 ESP-12E
HARDWARE: ESP8266 80MHz, 80KB RAM, 4MB Flash
PACKAGES:
 - framework-arduinoespressif8266 3.20701.0 (2.7.1)
 - tool-esptool 1.413.0 (4.13)
 - tool-esptoolpy 1.20800.0 (2.8.0)
 - tool-mkspiffs 1.200.0 (2.0)
 - toolchain-xtensa 2.40802.200502 (4.8.2)
LDF: Library Dependency Finder -> http://bit.ly/configure-pio-ldf
LDF Modes: Finder ~ chain, Compatibility ~ soft
Found 36 compatible libraries
Scanning dependencies...
Dependency Graph
|-- <ArduinoJson> 6.15.1
|-- <Time> 1.6
|-- <Ticker> 1.0
|-- <Timezone> 1.2.4
|   |-- <Time> 1.6
|-- <PCF8574_ESP> 1.0.10
|   |-- <Wire> 1.0
|-- <EspSoftwareSerial> 6.8.1
|-- <TinyGPSPlus> 1.0.2
|-- <AccelStepper> 1.59
|-- <ESP8266WiFi> 1.0
|-- <ArduinoOTA> 1.0
|   |-- <ESP8266WiFi> 1.0
|   |-- <ESP8266mDNS> 1.2
|   |   |-- <ESP8266WiFi> 1.0
|-- <EEPROM> 1.0
|-- <ESP8266mDNS> 1.2
|   |-- <ESP8266WiFi> 1.0
|-- <ESP8266WebServer> 1.0
|   |-- <ESP8266WiFi> 1.0
|-- <LittleFS(esp8266)> 0.1.0
|-- <PubSubClient> 1.99.1
|-- <SD(esp8266)> 2.0.0
|   |-- <SDFS> 0.1.0
|   |   |-- <ESP8266SdFat> 1.1.0
|   |   |   |-- <SPI> 1.0
|   |   |-- <SPI> 1.0
|-- <SPI> 1.0
|-- <Wire> 1.0
Building in release mode
Building SPIFFS image from 'data' directory to .pio\build\esp12e\spiffs.bin
Building LittleFS image…
{'-c': 'data', '-p': '256', '-b': '8192', '-s': '1024000'}
Adding badrequest.html
Adding favicon.ico
Adding generalsettings.html
Adding index.html
Adding login.html
Adding networksettings.html
Adding pageheader.html
Adding sensors.html
Adding status.html
Adding tools.html
Uploading .pio\build\esp12e\spiffs.bin
11:11:49 [DEBUG]: Options: {'esp_ip': '192.168.1.201', 'host_ip': '0.0.0.0', 'esp_port': 8266, 'host_port': 57875, 'auth': '', 'image': '.pio\\build\\esp12e\\spiffs.bin', 'spiffs': True, 'debug': True, 'progress': True}
11:11:49 [INFO]: Starting on 0.0.0.0:57875
11:11:49 [INFO]: Upload size: 12582912
11:11:49 [INFO]: Sending invitation to: 192.168.1.201
11:11:49 [ERROR]: Bad Answer: ERR: ERROR[4]: Not Enough Space


*** [uploadfs] Error 1
========================================================================================= [FAILED] Took 3.35 seconds =========================================================================================
The terminal process terminated with exit code: 1

Note, that before the conversion to LittleFS the files fit without a problem. The total size of files is around 30 kBytes.

viktak · May 10, 2020, 8:46am

I tried your proposal and it finally went through without a problem. However, none of the web pages (in the data folder) get displayed. This leads me to believe that I haven’t made sufficient changes in my code (for the ESP).

As I read that LittleFS is “drop-in compatible” with SPIFFS, so far I only changed all occurrences of SPIFFS to LittleFS, like this:
fs::File f = LittleFS.open("/pageheader.html", "r");

Is there anything else I need to be aware of?

Worth noting, that any existing code that creates/modifies/reads files on the fly work. The problem is only with the file system I upload.

viktak · May 10, 2020, 11:53am

This is the log of the succeeded upload:

> Executing task in folder stepper-test: C:\Users\Viktor\.platformio\penv\Scripts\platformio.exe run --target uploadfs <

Processing esp12e (platform: espressif8266; framework: arduino; board: esp12e)
---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

Verbose mode can be enabled via `-v, --verbose` option
CONFIGURATION: https://docs.platformio.org/page/boards/espressif8266/esp12e.html
PLATFORM: Espressif 8266 2.5.1 > Espressif ESP8266 ESP-12E
HARDWARE: ESP8266 80MHz, 80KB RAM, 4MB Flash
PACKAGES:
 - framework-arduinoespressif8266 3.20701.0 (2.7.1)
 - tool-esptool 1.413.0 (4.13)
 - tool-esptoolpy 1.20800.0 (2.8.0)
 - tool-mkspiffs 1.200.0 (2.0)
 - toolchain-xtensa 2.40802.200502 (4.8.2)
LDF: Library Dependency Finder -> http://bit.ly/configure-pio-ldf
LDF Modes: Finder ~ chain, Compatibility ~ soft
Found 36 compatible libraries
Scanning dependencies...
Dependency Graph
|-- <ArduinoJson> 6.15.1
|-- <Time> 1.6
|-- <Ticker> 1.0
|-- <Timezone> 1.2.4
|   |-- <Time> 1.6
|-- <PCF8574_ESP> 1.0.10
|   |-- <Wire> 1.0
|-- <EspSoftwareSerial> 6.8.1
|-- <TinyGPSPlus> 1.0.2
|-- <AccelStepper> 1.59
|-- <ESP8266WiFi> 1.0
|-- <ArduinoOTA> 1.0
|   |-- <ESP8266WiFi> 1.0
|   |-- <ESP8266mDNS> 1.2
|   |   |-- <ESP8266WiFi> 1.0
|-- <EEPROM> 1.0
|-- <ESP8266mDNS> 1.2
|   |-- <ESP8266WiFi> 1.0
|-- <ESP8266WebServer> 1.0
|   |-- <ESP8266WiFi> 1.0
|-- <LittleFS(esp8266)> 0.1.0
|-- <PubSubClient> 1.99.1
|-- <SD(esp8266)> 2.0.0
|   |-- <SDFS> 0.1.0
|   |   |-- <ESP8266SdFat> 1.1.0
|   |   |   |-- <SPI> 1.0
|   |   |-- <SPI> 1.0
|-- <SPI> 1.0
|-- <Wire> 1.0
Building in release mode
Building SPIFFS image from 'data' directory to .pio\build\esp12e\spiffs.bin
/badrequest.html
/favicon.ico
/generalsettings.html
/index.html
/login.html
/networksettings.html
/pageheader.html
/sensors.html
/status.html
/tools.html
Uploading .pio\build\esp12e\spiffs.bin
14:51:36 [DEBUG]: Options: {'esp_ip': '192.168.1.201', 'host_ip': '0.0.0.0', 'esp_port': 8266, 'host_port': 10654, 'auth': '', 'image': '.pio\\build\\esp12e\\spiffs.bin', 'spiffs': True, 'debug': True, 'progress': True}
14:51:36 [INFO]: Starting on 0.0.0.0:10654
14:51:36 [INFO]: Upload size: 1024000
14:51:36 [INFO]: Sending invitation to: 192.168.1.201
14:51:36 [INFO]: Waiting for device...

Uploading: [                                                            ] 0% 
.
.
.
.

Uploading: [============================================================] 99%
Uploading: [============================================================] 100% Done...


14:51:54 [INFO]: Waiting for result...
14:51:55 [INFO]: Result: OK
======================================================================================== [SUCCESS] Took 20.28 seconds ========================================================================================

Yuriy · May 10, 2020, 6:14pm

Try this (platformio.ini)
platform = GitHub - platformio/platform-espressif8266: Espressif 8266: development platform for PlatformIO
platform_packages = GitHub - esp8266/Arduino: ESP8266 core for Arduino
And i diid’t check ota updates… only by wire, but why it’s should differ… Your log looks right.

viktak · May 11, 2020, 9:07am

ok, so finally I managed to get it to work. Probably last night i was tired, as now, when I tried it it worked for the first try.

At the moment this is the only line added to my platformio.ini:

extra_scripts = ../_common/tools/LittleFS/LittleFSBuilder.py

and LittleFSBuilder.py looks like this:

Import("env")
env.Replace (MKSPIFFSTOOL = "../_common/tools/LittleFs/mklittlefs.exe")

This works.

However, I would still like to find out and understand why the python version doesn’t work:

Import("env")
env.Replace( MKSPIFFSTOOL='python" "' + '../_common/tools/LittleFs/mklittlefs.py' )

It still gives me the error:

12:06:32 [DEBUG]: Options: {'esp_ip': '192.168.1.53', 'host_ip': '0.0.0.0', 'esp_port': 8266, 'host_port': 56868, 'auth': '', 'image': '.pio\\build\\esp12e\\spiffs.bin', 'spiffs': True, 'debug': True, 'progress': True}
12:06:32 [INFO]: Starting on 0.0.0.0:56868
12:06:32 [INFO]: Upload size: 12582912
12:06:32 [INFO]: Sending invitation to: 192.168.1.53
12:06:35 [ERROR]: Bad Answer: ERR: ERROR[4]: Not Enough Space

Thank you for all of you sticking with me!

maxgerhardt · May 11, 2020, 11:41am

If you copied it from above the script does…

fs = LittleFS(block_size=4096, block_count=3072)                                    # Open LittleFS, create a 12MB partition

a 12MB partition. What’s the size of your ESP8266 flash? Keep in mind that this thread is with regards to the ESP32 originally.

viktak · May 11, 2020, 12:33pm

Well, this makes sense…

True, I am using an ESP-12F module…

Do you know what block size and count should I use for the ESP12-F?

I reduced the block_count to 250 to make the image the same size as the other method creates, it uploads fine, but the pages are not loadable… (I don’t get any error messages, simply nothing is served to the client). Again, the file system seems to be OK, because my software can read and write a new file.

maxgerhardt · May 11, 2020, 1:17pm

I’m not an expert on how littlefs-python · PyPI creates the LittleFS exactly and what kind of debugging should be undertaken to find out what’s going on behind the curtain. As a general debugging step I’d recommend trying to unpack / list the contents of the generated spiffs.bin (actually LittelFS) with the tools given above, like mklittlefs. You should also try to mount and enumerate the LittleFS filesystem (there’s got to be some functions which tells you how big the mounted LittleFS partition is and a directory/file crawler enumerating all files).

For reproduction I’d also recommend upload your project (e.g. to github) so that other people can reproduce your project and problem 1:1.

viktak · May 12, 2020, 7:52am

OK, I will continue to investigate, although not right now as I got it to work (with the EXE version) and I have other urgent things to do. Will report back if/when I will know more.

Thanks again for your input!

whitelionatx · May 21, 2020, 9:06pm

I think my solution for LittleFsV2 is easy to implement.
You need:

python installed
these two files: (download)
one line in your project - > platformio.ini

install python and test if its useable form your VS terminal window.
drop the two files one directory above your project directory.
This is my project directory:
“E:\Arduino und Derivate\PlatformIO_Projects\Projects\Sensor Typ B”
This is where the upload files are located (similar to SPIFFS upload directory):
“E:\Arduino und Derivate\PlatformIO_Projects\Projects\Sensor Typ B\data”
This is where you place the files (LittleFSBuilder.py and mklittlefs.exe):
"E:\Arduino und Derivate\PlatformIO_Projects\Projects"
After you have done this the only thing you need to do is add the following line to your platformio.ini
extra_scripts = ../LittleFSBuilder.py
Adding this line means that LittleFsV2 file system will be used. If you dont add it The file system will be default (SPIFFS)
Steps 1. and 2. are only initial needed. (At least if your projects have a path structure like mine. If not, feel free to mod the python script)

dakkar · May 24, 2020, 8:34am

but imho this should be implemented by platformIO itself:
On the one hand LillteFS is the official successor of SPIFFS. Even vscode marks it as deprecated.
On the other hand der is this fuddling around with custom scripts, posted “somewhere on the internet”

It feels wrong

whitelionatx · May 24, 2020, 12:19pm

You talk about a world that would be beautiful, but does not exist like this. Everybody who is more intensively engaged in programming and all that goes with it, knows these conditions well enough. I myself would have preferred to save the time instead of struggling with the search for a solution. - be glad that you don’t have these problems 20 years before and that there is “somebody from the internet”, who is able to chew a solution for you. - I know other times

maxgerhardt · May 24, 2020, 12:47pm

That’s why our developers are already looking into it. ESP8266 NodeMCV1 LittleFS workflow? - #3 by valeros