PlatformIO Community

[Question] ESP32: Compress files in Data to GZip before upload possible to SPIFFS?

Hey,

I developed a huge Webpage project for a ESP32 with Bootstrap 4 and ESPAsyncWebServer which supports serving .gz files. As I write the page I have to manually GZIP the Javascript, CSS and HTML files before I upload them to the ESP32 SPIFFS Datasystem.

So my question is: Is there a way to implement a script which starts the 7ZIP Program and uploads the files after 7ZIP is ready?

Thanks :slight_smile:

Environment: Win10, VS Code (V1.30.2) with PlattformIO (V3.6.3), ESP32_DevKit_V4

Look at http://docs.platformio.org/en/latest/projectconf/advanced_scripting.html, section “Before/Pre and After/Post actions”, especially the lines

# custom action before building SPIFFS image. For example, compress HTML, etc.
env.AddPreAction("$BUILD_DIR/spiffs.bin", callback...)
1 Like

I imagine that “extra_script.py” should be created at the same level as “platformio.ini”, then
‘extra_scripts = extra_script.py’ line added to platformio.ini.
Then, what?

Let’s say, I’m on W10, using platformio as an addon to VSCode.
My ‘data’ folder contains two files: favicon.ico and index.html.
I want .html file to be gzipped every time before uploading ‘data’ contents to the SPIFFS image.
Can anybody provide a simple example of how to do that? Thanks in advance.

Then you write the python code you want to execute. For example, using the subprocess module, you can call into gzip. Then move all files that you don’t want to include in the SPIFFS binary out of the folder, and with with the env.AddPostAction() register a callback which will move the old files back in and remove the old gzip file.

2 Likes

Currently, I have this code inside the script. Didn’t try gzip, yet, just dealing with moving, for now:

Import("env", "projenv")

import os
# import gzip
import shutil


# # Packing files
# with open('path/to/file') as src, gzip.open('path/to/file.gz', 'wb') as dst:        
#     dst.writelines(src)

# Moving files
def index_html_move():
    shutil.move("/long_path/test1/file.foo", "/long_path/test2/file.foo")

def index_html_move_next():
    shutil.move("/long_path/test2/file.foo", "/long_path/test3/file.foo")

env.AddPreAction("$BUILD_DIR/spiffs.bin", index_html_move())
env.AddPostAction("$BUILD_DIR/spiffs.bin", index_html_move_next())

I have two issues:

  1. Platformio cannot recognize “%BUILD_DIR” or “%PROJECT_DIR” as a part of the file’s path, so I need to provide full path instead (“c:\projects…” substituted as “long_path” in the example).
  2. Though I can see that file.foo is moved to “test3” folder, while being inside “test1” folder, initially, I get the following error:

*** [.pio\build\esp32doit-devkit-v1\spiffs.bin] AttributeError : ‘NoneType’ object has no attribute ‘genstring’

And there’s no uploading of the spiffs image, in the end.

1 Like

You don’t want to execute that at this point, you want to pass the function name

env.AddPreAction("$BUILD_DIR/spiffs.bin", index_html_move)
env.AddPostAction("$BUILD_DIR/spiffs.bin", index_html_move_next)
1 Like
index_html_move([".pio\build\esp32doit-devkit-v1\spiffs.bin"], ["data"])
*** [.pio\build\esp32doit-devkit-v1\spiffs.bin] TypeError : index_html_move() takes no arguments (3 given)
Traceback (most recent call last):
  File "C:\...\.platformio\packages\tool-scons\script\..\engine\SCons\Action.py", line 1054, in execute
    result = self.execfunction(target=target, source=rsources, env=env)
TypeError: index_html_move() takes no arguments (3 given)

As can be seen in the documentation, the callback function needs to accept the 3 parameters

def before_upload(source, target, env):
     # code..
1 Like

You’re a great help, Max. It’s working now. I just need to figure out now how to feed relative path to it ($PROJECT_DIR) instead of a full path.
Also need to experiment with gzip part.

BTW, do you know how to make AsyncWebServer serve gzipped html file to the browser?
I’m also struggling with this part of my project.

When I use this code, my browser is displaying gzip as plain text:

  server.on("/", HTTP_GET, [](AsyncWebServerRequest *request){
    AsyncWebServerResponse *response = request->beginResponse(200, "text/html");
    response->addHeader("Content-Encoding", "gzip");
    request->send(SPIFFS, "/index.html.gz", "text/html", false);
  });

Do the values of the source and target variables contain fully-expanded paths? Then using os.path.basename() and friends can help.

Could you try instead

  server.on("/", HTTP_GET, [](AsyncWebServerRequest *request){
    AsyncWebServerResponse *response = request->beginResponse(SPIFFS, "/index.html.gz", "text/html");
    response->addHeader("Content-Encoding", "gzip");
    request->send(response);
  });
1 Like

Your variant looks more logical and, what’s important, it actually works.

I will check this tomorrow, as well as gzipping everything. Thank you for your help.

2 Likes

It’s weird that there’s no ready-to-use solution posted anywhere, so for anyone coming here from google search (as i did), here’s what i came up with:

prep_data_folder.py:

Import('env', 'projenv')

import os
import gzip
import shutil
import glob

def prepare_www_files(source, target, env):
    #WARNING -  this script will DELETE your 'data' dir and recreate an empty one to copy/gzip files from 'data_src'
    #           so make sure to edit your files in 'data_src' folder as changes madt to files in 'data' woll be LOST
    #           
    #           If 'data_src' dir doesn't exist, and 'data' dir is found, the script will autimatically
    #           rename 'data' to 'data_src


    #add filetypes (extensions only) to be gzipped before uploading. Everything else will be copied directly
    filetypes_to_gzip = ['js', 'html']

    
    print('[COPY/GZIP DATA FILES]')

    data_dir = env.get('PROJECTDATA_DIR')
    data_src_dir = os.path.join(env.get('PROJECT_DIR'), 'data_src')

    if(os.path.exists(data_dir) and not os.path.exists(data_src_dir) ):
        print('  "data" dir exists, "data_src" not found.')
        print('  renaming "' + data_dir + '" to "' + data_src_dir + '"')
        os.rename(data_dir, data_src_dir)

    if(os.path.exists(data_dir)):
        print('  Deleting data dir ' + data_dir)
        shutil.rmtree(data_dir)

    print('  Re-creating empty data dir ' + data_dir)
    os.mkdir(data_dir)

    files_to_gzip = []
    for extension in filetypes_to_gzip:
        files_to_gzip.extend(glob.glob(os.path.join(data_src_dir, '*.' + extension)))
    
    print('  files to gzip: ' + str(files_to_gzip))

    all_files = glob.glob(os.path.join(data_src_dir, '*.*'))
    files_to_copy = list(set(all_files) - set(files_to_gzip))

    print('  files to copy: ' + str(files_to_copy))

    for file in files_to_copy:
        print('  Copying file: ' + file + ' to data dir')
        shutil.copy(file, data_dir)
    
    for file in files_to_gzip:
        print('  GZipping file: ' + file + ' to data dir')
        with open(file) as src, gzip.open(os.path.join(data_dir, os.path.basename(file) + '.gz'), 'wb') as dst:        
            dst.writelines(src)

    print('[/COPY/GZIP DATA FILES]')
    
env.AddPreAction('$BUILD_DIR/spiffs.bin', prepare_www_files)

And add this line to your build environment in platformio.ini:

extra_scripts = post:prep_data_folder.py

It’s probably not the most optimal way to do it, but it does the job.
It works by gzipping selected file types from ‘data_src’ folder to data and directly copying everything else before SPIFFS image is built from ‘data’ folder.

Downside of doing it this way is that you need to edit your files in ‘data_src’, not the default ‘data’ location, as ‘data’ is emptied each time you run upload filesystem task.

If you don’t have the ‘data_src’ folder, but you have ‘data’, the script will automatically rename ‘data’ to ‘data_src’, so you won’t lose any of your files. Just remember to reopen files from the new location after this renaming.

You can customize which file types get gzipped by adding their extensions to filetypes_to_gzip array. Every other file will be directly copied from ‘data_src’ to ‘data’ without modification.

1 Like

Nice work! :slight_smile:

There is this article about gzipping files before upload, but it uses gulp, which to my mind isn’t has compact and portable as using gzip /w python. On the other hand, it can do a lot more than just gzip, but I don’t think ‘minifying’ the code first would make it much, if any, smaller when gzipped…