Hey,
I developed a huge Webpage project for a ESP32 with Bootstrap 4 and ESPAsyncWebServer which supports serving .gz files. As I write the page I have to manually GZIP the Javascript, CSS and HTML files before I upload them to the ESP32 SPIFFS Datasystem.
So my question is: Is there a way to implement a script which starts the 7ZIP Program and uploads the files after 7ZIP is ready?
Thanks 
Environment: Win10, VS Code (V1.30.2) with PlattformIO (V3.6.3), ESP32_DevKit_V4
Look at Redirecting..., section “Before/Pre and After/Post actions”, especially the lines
# custom action before building SPIFFS image. For example, compress HTML, etc.
env.AddPreAction("$BUILD_DIR/spiffs.bin", callback...)
I imagine that “extra_script.py” should be created at the same level as “platformio.ini”, then
‘extra_scripts = extra_script.py’ line added to platformio.ini.
Then, what?
Let’s say, I’m on W10, using platformio as an addon to VSCode.
My ‘data’ folder contains two files: favicon.ico and index.html.
I want .html file to be gzipped every time before uploading ‘data’ contents to the SPIFFS image.
Can anybody provide a simple example of how to do that? Thanks in advance.
Then you write the python code you want to execute. For example, using the subprocess module, you can call into gzip. Then move all files that you don’t want to include in the SPIFFS binary out of the folder, and with with the env.AddPostAction() register a callback which will move the old files back in and remove the old gzip file.
Currently, I have this code inside the script. Didn’t try gzip, yet, just dealing with moving, for now:
Import("env", "projenv")
import os
# import gzip
import shutil
# # Packing files
# with open('path/to/file') as src, gzip.open('path/to/file.gz', 'wb') as dst:
# dst.writelines(src)
# Moving files
def index_html_move():
shutil.move("/long_path/test1/file.foo", "/long_path/test2/file.foo")
def index_html_move_next():
shutil.move("/long_path/test2/file.foo", "/long_path/test3/file.foo")
env.AddPreAction("$BUILD_DIR/spiffs.bin", index_html_move())
env.AddPostAction("$BUILD_DIR/spiffs.bin", index_html_move_next())
I have two issues:
*** [.pio\build\esp32doit-devkit-v1\spiffs.bin] AttributeError : ‘NoneType’ object has no attribute ‘genstring’
And there’s no uploading of the spiffs image, in the end.
You don’t want to execute that at this point, you want to pass the function name
env.AddPreAction("$BUILD_DIR/spiffs.bin", index_html_move)
env.AddPostAction("$BUILD_DIR/spiffs.bin", index_html_move_next)
index_html_move([".pio\build\esp32doit-devkit-v1\spiffs.bin"], ["data"])
*** [.pio\build\esp32doit-devkit-v1\spiffs.bin] TypeError : index_html_move() takes no arguments (3 given)
Traceback (most recent call last):
File "C:\...\.platformio\packages\tool-scons\script\..\engine\SCons\Action.py", line 1054, in execute
result = self.execfunction(target=target, source=rsources, env=env)
TypeError: index_html_move() takes no arguments (3 given)As can be seen in the documentation, the callback function needs to accept the 3 parameters
def before_upload(source, target, env):
# code..
You’re a great help, Max. It’s working now. I just need to figure out now how to feed relative path to it ($PROJECT_DIR) instead of a full path.
Also need to experiment with gzip part.
BTW, do you know how to make AsyncWebServer serve gzipped html file to the browser?
I’m also struggling with this part of my project.
When I use this code, my browser is displaying gzip as plain text:
server.on("/", HTTP_GET, [](AsyncWebServerRequest *request){
AsyncWebServerResponse *response = request->beginResponse(200, "text/html");
response->addHeader("Content-Encoding", "gzip");
request->send(SPIFFS, "/index.html.gz", "text/html", false);
});Do the values of the source and target variables contain fully-expanded paths? Then using os.path.basename() and friends can help.
Could you try instead
server.on("/", HTTP_GET, [](AsyncWebServerRequest *request){
AsyncWebServerResponse *response = request->beginResponse(SPIFFS, "/index.html.gz", "text/html");
response->addHeader("Content-Encoding", "gzip");
request->send(response);
});
Your variant looks more logical and, what’s important, it actually works.
I will check this tomorrow, as well as gzipping everything. Thank you for your help.
It’s weird that there’s no ready-to-use solution posted anywhere, so for anyone coming here from google search (as i did), here’s what i came up with:
prep_data_folder.py:
Import('env', 'projenv')
import os
import gzip
import shutil
import glob
def prepare_www_files(source, target, env):
#WARNING - this script will DELETE your 'data' dir and recreate an empty one to copy/gzip files from 'data_src'
# so make sure to edit your files in 'data_src' folder as changes madt to files in 'data' woll be LOST
#
# If 'data_src' dir doesn't exist, and 'data' dir is found, the script will autimatically
# rename 'data' to 'data_src
#add filetypes (extensions only) to be gzipped before uploading. Everything else will be copied directly
filetypes_to_gzip = ['js', 'html']
print('[COPY/GZIP DATA FILES]')
data_dir = env.get('PROJECTDATA_DIR')
data_src_dir = os.path.join(env.get('PROJECT_DIR'), 'data_src')
if(os.path.exists(data_dir) and not os.path.exists(data_src_dir) ):
print(' "data" dir exists, "data_src" not found.')
print(' renaming "' + data_dir + '" to "' + data_src_dir + '"')
os.rename(data_dir, data_src_dir)
if(os.path.exists(data_dir)):
print(' Deleting data dir ' + data_dir)
shutil.rmtree(data_dir)
print(' Re-creating empty data dir ' + data_dir)
os.mkdir(data_dir)
files_to_gzip = []
for extension in filetypes_to_gzip:
files_to_gzip.extend(glob.glob(os.path.join(data_src_dir, '*.' + extension)))
print(' files to gzip: ' + str(files_to_gzip))
all_files = glob.glob(os.path.join(data_src_dir, '*.*'))
files_to_copy = list(set(all_files) - set(files_to_gzip))
print(' files to copy: ' + str(files_to_copy))
for file in files_to_copy:
print(' Copying file: ' + file + ' to data dir')
shutil.copy(file, data_dir)
for file in files_to_gzip:
print(' GZipping file: ' + file + ' to data dir')
with open(file) as src, gzip.open(os.path.join(data_dir, os.path.basename(file) + '.gz'), 'wb') as dst:
dst.writelines(src)
print('[/COPY/GZIP DATA FILES]')
env.AddPreAction('$BUILD_DIR/spiffs.bin', prepare_www_files)And add this line to your build environment in platformio.ini:
extra_scripts = post:prep_data_folder.pyIt’s probably not the most optimal way to do it, but it does the job.
It works by gzipping selected file types from ‘data_src’ folder to data and directly copying everything else before SPIFFS image is built from ‘data’ folder.
Downside of doing it this way is that you need to edit your files in ‘data_src’, not the default ‘data’ location, as ‘data’ is emptied each time you run upload filesystem task.
If you don’t have the ‘data_src’ folder, but you have ‘data’, the script will automatically rename ‘data’ to ‘data_src’, so you won’t lose any of your files. Just remember to reopen files from the new location after this renaming.
You can customize which file types get gzipped by adding their extensions to filetypes_to_gzip array. Every other file will be directly copied from ‘data_src’ to ‘data’ without modification.
Nice work!
There is this article about gzipping files before upload, but it uses gulp, which to my mind isn’t has compact and portable as using gzip /w python. On the other hand, it can do a lot more than just gzip, but I don’t think ‘minifying’ the code first would make it much, if any, smaller when gzipped…
Thank you for this helpful script. It gave me a good headstart on doing the exact same thing. But I did not want to delete and create data folders. I just wanted all the files which need gzip-ing to be found in a dedicated gzip-directory of my SPIFFS.
Since the static webserver directory should not deliver these files I added an underscore to those files the webserver should not find/deliver directly. I added handler-methods in the code to deliver the gzipped files separately (not via static file server).
I found one thing very important to know, you could run into issues that PlatformIO build task does not find the extrascript, even though it is in the src-directory. I fixed that by adding following entry to the ini-file:
extra_scripts =
post:src/LMBuildExtensionSPIFFS.py
This makes the Python file reachable. Otherwise you might see a weird warning which sounds like this and totally does not give a clue what happened:
warning: Calling missing SConscript without error is deprecated.
Transition by adding must_exist=0 to SConscript calls.
Missing SConscript 'LMBuildExtensionSPIFFS.py'
File "/Users/blazr/.platformio/penv/lib/python3.8/site-packages/platformio/builder/main.py", line 164, in <module>
So for everyone hitting this roadblock, this is just a not found file, add the correct relative path from your root project directory and it works.
I changed your code a bit. I exchanged especially the gzip-part because I had issues with it actually working. This worked for me.
Find the changed file here:
# SCRIPT TO GZIP CRITICAL FILES FOR ACCELERATED WEBSERVING
# see also https://community.platformio.org/t/question-esp32-compress-files-in-data-to-gzip-before-upload-possible-to-spiffs/6274/10
#
Import( 'env', 'projenv' )
import os
import gzip
import shutil
import glob
# HELPER TO GZIP A FILE
def gzip_file( src_path, dst_path ):
with open( src_path, 'rb' ) as src, gzip.open( dst_path, 'wb' ) as dst:
for chunk in iter( lambda: src.read(4096), b"" ):
dst.write( chunk )
# GZIP DEFINED FILES FROM 'data' DIR to 'data/gzip/' DIR
def gzip_webfiles( source, target, env ):
# FILETYPES / SUFFIXES WHICh NEED TO BE GZIPPED
source_file_prefix = '_'
filetypes_to_gzip = [ 'css', 'html' ]
print( '\nGZIP: INITIATED GZIP FOR SPIFFS...\n' )
GZIP_DIR_NAME = 'gzip'
data_dir_path = env.get( 'PROJECTDATA_DIR' )
gzip_dir_path = os.path.join( env.get( 'PROJECTDATA_DIR' ), GZIP_DIR_NAME )
# CHECK DATA DIR
if not os.path.exists( data_dir_path ):
print( 'GZIP: DATA DIRECTORY MISSING AT PATH: ' + data_dir_path )
print( 'GZIP: PLEASE CREATE THE DIRECTORY FIRST (ABORTING)' )
print( 'GZIP: FAILURE / ABORTED' )
return
# CHECK GZIP DIR
if not os.path.exists( gzip_dir_path ):
print( 'GZIP: GZIP DIRECTORY MISSING AT PATH: ' + gzip_dir_path )
print( 'GZIP: TRYING TO CREATE IT...' )
try:
os.mkdir( gzip_dir_path )
except Exception as e:
print( 'GZIP: FAILED TO CREATE DIRECTORY: ' + gzip_dir_path )
# print( 'GZIP: EXCEPTION... ' + str( e ) )
print( 'GZIP: PLEASE CREATE THE DIRECTORY FIRST (ABORTING)' )
print( 'GZIP: FAILURE / ABORTED' )
return
# DETERMINE FILES TO COMPRESS
files_to_gzip = []
for extension in filetypes_to_gzip:
match_str = source_file_prefix + '*.'
files_to_gzip.extend( glob.glob( os.path.join( data_dir_path, match_str + extension ) ) )
# print( 'GZIP: GZIPPING FILES... {}\n'.format( files_to_gzip ) )
# COMPRESS AND MOVE FILES
was_error = False
try:
for source_file_path in files_to_gzip:
print( 'GZIP: ZIPPING... ' + source_file_path )
base_file_path = source_file_path.replace( source_file_prefix, '' )
target_file_path = os.path.join( gzip_dir_path, os.path.basename( base_file_path ) + '.gz' )
# CHECK IF FILE ALREADY EXISTS
if os.path.exists( target_file_path ):
print( 'GZIP: REMOVING... ' + target_file_path )
os.remove( target_file_path )
# print( 'GZIP: GZIPPING FILE...\n' + source_file_path + ' TO...\n' + target_file_path + "\n\n" )
print( 'GZIP: GZIPPED... ' + target_file_path + "\n" )
gzip_file( source_file_path, target_file_path )
except IOError as e:
was_error = True
print( 'GZIP: FAILED TO COMPRESS FILE: ' + source_file_path )
# print( 'GZIP: EXCEPTION... {}'.format( e ) )
if was_error:
print( 'GZIP: FAILURE/INCOMPLETE.\n' )
else:
print( 'GZIP: SUCCESS/COMPRESSED.\n' )
# IMPORTANT, this needs to be added to call the routine
env.AddPreAction( '$BUILD_DIR/spiffs.bin', gzip_webfiles )
This will give you detailed feedback on the console while building. I tested my script against my build environment BUILD_ESP32 like this from the PlatformIO CLI Core Console:
platformio run -e BUILD_ESP32 --target buildfs
Thanks again for your work. 
