ESP8266 OTA upload VS compressed OTA upload

Hi all,
what are the pros/cons of compressed OTA upload?

Are there any other pros over the obvious less bytes to transfer?

… but the real question is…

how can I enable the compressed OTA upload via platformio?

At the moment, because it’s not ‘natively’ supported by PlatformIO… reading the docs on the new OTA compression support it looks like two things need to be done… a flag set (ATOMIC_FS_UPDATE, which can easily be done via build_flags. I think the catch will be the only other change… that the file be compressed before it’s uploaded.

gzip -9 sketch.bin  # Maximum compression, output sketch.bin.gz
<Upload the resultant sketch.bin.gz>

For this, I think some crafty extra_scripting will be needed, which compresses the file pre-upload, and then either over-rides the filename for the upload, or cheekily deletes the old file, and renames the compressed one to the expected filename.

hope to see some complete example soon, sincerely I’m not able to do it right now without
any docs on it.

I finally decided to play with this, and came up with the following. Before you use it, ensure you have flashed your target device with espressif8266@2.5.1 or esp8266/Arduino v2.7.1 at least once, to ensure that the target device will understand what to do with a compressed OTA payload.

Add extra_scripts = compressed_ota.py to any environment in your platformio.ini where you want to have compressed OTA updates active, and put the following in a file called compressed_ota.py in the root of your project dir (i.e. the same folder the platformio.ini is in):

Updated 6/June/2020 to include @srchip’s platform friendly seperator tip, and link to github gist with the code.

import gzip
import shutil
import os
Import("env")


def compressFirmware(source, target, env):
    """ Compress ESP8266 firmware using gzip for 'compressed OTA upload' """
    SOURCE_FILE = env.subst("$BUILD_DIR") + os.sep + env.subst("$PROGNAME") + ".bin"
    if not os.path.exists(SOURCE_FILE+'.bak'):
        print("Compressing firmware for upload...")
        shutil.move(SOURCE_FILE, SOURCE_FILE + '.bak')
        with open(SOURCE_FILE + '.bak', 'rb') as f_in:
            with gzip.open(SOURCE_FILE, 'wb') as f_out:
                shutil.copyfileobj(f_in, f_out)

    if os.path.exists(SOURCE_FILE+'.bak'):
        ORG_FIRMWARE_SIZE = os.stat(SOURCE_FILE + '.bak').st_size
        GZ_FIRMWARE_SIZE = os.stat(SOURCE_FILE).st_size

        print("Compression reduced firmware size by {:.0f}% (was {} bytes, now {} bytes)".format((GZ_FIRMWARE_SIZE / ORG_FIRMWARE_SIZE) * 100, ORG_FIRMWARE_SIZE, GZ_FIRMWARE_SIZE))

env.AddPreAction("upload", compressFirmware)

Then give it a try, and look for the two (three, actually, since PIO mentions the extra_scripts hook) extra lines related to compression in the output when you try your next upload.

RAM:   [====      ]  39.3% (used 32180 bytes from 81920 bytes)
Flash: [====      ]  38.8% (used 405328 bytes from 1044464 bytes)
compressFirmware(["upload"], [".pio\build\d1_mini-OTA\firmware.bin"])
Compressing firmware for upload...
Compression reduced firmware size by 70% (was 409488 bytes, now 287750 bytes)
Configuring upload protocol...

Please kick the tires, and see what breaks. It is working fine for me one a Wemos D1 Mini, on Windows 10. Since I’m using the python gzip library, it should work everywhere Python does, but there also isn’t any error checking… :open_mouth:

2 Likes

love it, using it right now on my small projects.
THANKS Pferrick!!!

1 Like

Hope that this feature will be added in platformio soon, thanks!

1 Like

Awesome work!! I’d suggest a tiny upgrade to make it work in other OSes as well.

Change this to

SOURCE_FILE = env.subst("$BUILD_DIR") + os.sep + env.subst("$PROGNAME") + ".bin"
1 Like

Nice! Thanks for that! :slight_smile:

@pfeerick , do you think that this could work on ESP32 too?

No idea… there’s one way to fine out! :stuck_out_tongue:

If I get a chance I’ll look at the ESP32 in the next few days… haven’t done anything in that space for a little while…

ordered an ESP32, I will try it as soon as it will arrive :slight_smile:
first who try inform the other xD

1 Like

Works for ESP8266 but not for ESP32 crashes after 30%. :slightly_frowning_face:
Please share if you were successful with ESP32.

Esp32 is a mess. I don’t know if IDF is better but the Arduino support for ESP32 is way worse than esp8266, this why I choose esp8266 over esp32 for most of my projects.

Sincerely it is very difficult to justify the needs of the additional features that comes with esp32 for most of the projects.

Esp32 TCP stack is a mess, this means that esp8266 is often significantly faster than ESP32 in a lot of “connected situations”.

For me esp32 is a no go until Espressif decide to really support the Arduino platform.

Sorry for the ot but just a suggestion, stop wasting time with esp32 if you don’t really need the additional features it brings.

There are more cons than pros with esp32.

2 Likes