Running a post-build-script from library?

sivar2311 · June 26, 2022, 6:59pm

Hi!

Is it possible to run a post-build script from a library?
In the library.json I already added "build":{"extraScript:":"versioning.py"}.

The script runs before the build process and creates the files versioning.txt in the project folder and version.h in the src folder.

I would also like to add another script that runs after the build process and copies the resulting firmware.bin to a folder firmware and creates another file firmware.nfo.

Is this possible at all?

Note: I am not a python expert

maxgerhardt · June 26, 2022, 7:26pm

You can create a post action on the firmware.bin file as shown in the docs and execute arbitrary logic.

You might need to use the newly available projenv environment (and only this environment, not others) for this, as showcased in Force RTTI from library? - #23 by cziter15.

sivar2311 · June 26, 2022, 7:58pm

Hi @maxgerhardt

I have read the doc, but unfortunately I understand only half of it

I have once created a simple version of the script:

Import("env", "projenv")

def copy_firmware(source, target, env):
    print("test copy_firmware() is called")

env.AddPostAction("buildprog", copy_firmware)
print("test: script gets called")

Import("env", "projenv") results in *** Import of non-existent variable ''projenv''

Removing “projenv” fixed the error, but the function copy_firmware() is never called.

This is the output from the build process:

> Executing task in folder Firmware_Builder: C:\Users\Boris\.platformio\penv\Scripts\platformio.exe run --environment ESP32 <

Processing ESP32 (platform: espressif32; board: esp32dev; framework: arduino)
-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------Verbose mode can be enabled via `-v, --verbose` option
CONFIGURATION: https://docs.platformio.org/page/boards/espressif32/esp32dev.html
PLATFORM: Espressif 32 (4.4.0) > Espressif ESP32 Dev Module
HARDWARE: ESP32 240MHz, 320KB RAM, 4MB Flash
DEBUG: Current (cmsis-dap) External (cmsis-dap, esp-prog, iot-bus-jtag, jlink, minimodule, olimex-arm-usb-ocd, olimex-arm-usb-ocd-h, olimex-arm-usb-tiny-h, olimex-jtag-tiny, tumpa)
PACKAGES:
 - framework-arduinoespressif32 @ 3.20003.220619 (2.0.3)
 - tool-esptoolpy @ 1.30300.0 (3.3.0)
 - toolchain-xtensa-esp32 @ 8.4.0+2021r2-patch3
LDF: Library Dependency Finder -> https://bit.ly/configure-pio-ldf
LDF Modes: Finder ~ chain, Compatibility ~ soft
test: script gets called
Found 40 compatible libraries
Scanning dependencies...
Dependency Graph
|-- SimpleHTTPUpdater @ 0.1.0+sha.2c6fbc6
|   |-- HTTPClient @ 2.0.0
|   |   |-- WiFi @ 2.0.0
|   |   |-- WiFiClientSecure @ 2.0.0
|   |   |   |-- WiFi @ 2.0.0
|   |-- Update @ 2.0.0
|-- WiFi @ 2.0.0
Building in release mode
Compiling .pio\build\ESP32\src\main.cpp.o
Linking .pio\build\ESP32\firmware.elf
Retrieving maximum program size .pio\build\ESP32\firmware.elf
Checking size .pio\build\ESP32\firmware.elf
Advanced Memory Usage is available via "PlatformIO Home > Project Inspect"
RAM:   [=         ]  12.1% (used 39776 bytes from 327680 bytes)
Flash: [=======   ]  65.3% (used 855945 bytes from 1310720 bytes)
Building .pio\build\ESP32\firmware.bin
esptool.py v3.3
Creating esp32 image...
Merged 25 ELF sections
Successfully created esp32 image.

sivar2311 · June 26, 2022, 8:11pm

Addition:

Putting extra_scripts = versioning.py in platformio.ini works:

... shortened ...
Dependency Graph
|-- SimpleHTTPUpdater @ 0.1.0+sha.2c6fbc6
|   |-- HTTPClient @ 2.0.0
|   |   |-- WiFi @ 2.0.0
|   |   |-- WiFiClientSecure @ 2.0.0     
|   |   |   |-- WiFi @ 2.0.0
|   |-- Update @ 2.0.0
|-- WiFi @ 2.0.0
Building in release mode
test: script gets called
... shortened ...
Linking .pio\build\ESP32\firmware.elf
Retrieving maximum program size .pio\build\ESP32\firmware.elf
Checking size .pio\build\ESP32\firmware.elf
Advanced Memory Usage is available via "PlatformIO Home > Project Inspect"
RAM:   [=         ]  12.1% (used 39776 bytes from 327680 bytes)
Flash: [=======   ]  65.3% (used 855945 bytes from 1310720 bytes)
Building .pio\build\ESP32\firmware.bin
esptool.py v3.3
Creating esp32 image...
Merged 25 ELF sections
Successfully created esp32 image.
copy_firmware(["buildprog"], [".pio\build\ESP32\firmware.bin"])
test: copy_firmware() gets called

But my goal is to avoid manual changes to platformio.ini.

maxgerhardt · June 26, 2022, 8:12pm

This feature is only available in the dev version for now, hasn’t yet been published into a new release. CLI + pio upgrade --dev to upgrade, then retry with projenv.

sivar2311 · June 26, 2022, 8:19pm

Since it’s a library, I would gladly do without the developer version. This could cause problems for the users of the library who do not have the developer version installed.

Is the execution of a post-build script dependent on the “projenv”?

maxgerhardt · June 26, 2022, 9:04pm

Hm yes actually this doesn’t seem to be possible from a library’s extraScript at all, I tested this with

Import('env', 'projenv')

def copy_firmware(source, target, env):
    print("test copy_firmware() is called")

def post_bin_action(source, target, env):
    print("test post_bin_action() is called")

def post_progsize(source, target, env):
    print("test post_progsize() is called")

for e in (env, projenv, DefaultEnvironment()):
    e.AddPostAction("buildprog", copy_firmware)
    e.AddPostAction("checkprogsize", post_progsize)
    # for Uno projects: firmware.hex
    e.AddPostAction("$BUILD_DIR/${PROGNAME}.hex", post_bin_action)
    # for other projects
    e.AddPostAction("$BUILD_DIR/${PROGNAME}.bin", post_bin_action)
print("test: script gets called")

and non but the test: script gets called shows up – maybe these post actions are not available in library scripts? Could you confirm that @ivankravets? Test project is at GitHub - maxgerhardt/pio-script-hooks-actions

(PS: Some library authors restor to just storing the python script in the library and then tell users to reference it in the platformio.ini, like in here).

sivar2311 · June 28, 2022, 6:14am

@maxgerhardt Thanks for testing and confirming.
I will wait for the feedback from @ivankravets

ivankravets · June 28, 2022, 4:46pm

Hi there,

This is a historical issue. It is a bug from the PlatformIO’s perspective but from the SCons (which we use internally for the building) this is a feature. See

github.com

SCons/scons/blob/master/SCons/Environment.py#L1960


      
              """
              try:
                  kwbd = kw['BUILDERS']
              except KeyError:
                  pass
              else:
                  kwbd = BuilderDict(kwbd,self)
                  del kw['BUILDERS']
                  self.__setitem__('BUILDERS', kwbd)
              kw = copy_non_reserved_keywords(kw)
              self._update(semi_deepcopy(kw))
              self.scanner_map_delete(kw)
          
          
def ReplaceIxes(self, path, old_prefix, old_suffix, new_prefix, new_suffix):
              """
              Replace old_prefix with new_prefix and old_suffix with new_suffix.
          
          
    env - Environment used to interpolate variables.
              path - the path that will be modified.
              old_prefix - construction variable for the old prefix.
              old_suffix - construction variable for the old suffix.

So, if the target is not ready yet, action will not be assigned.

Nevertheless, I agree that we should fix that. So, I played a little bit and wrapped SCons handlers to catch these cases when targets are not ready. See

Please re-test with PIO Core 6.1-dev. We increased PIO Core minor version. A lot of changes and new features. Does it work for you?

sivar2311 · June 30, 2022, 5:34pm

Thank you very much for the quick answer and implementation. Unfortunately, I am away for a few days. I will test it when I return and report back.

maxgerhardt · June 30, 2022, 7:46pm

With 6.1.0b1 and the same test code, only in combination with environment = DefaultEnvironment() (not env or projenv), it triggers the e.AddPostAction("$BUILD_DIR/${PROGNAME}.hex", post_bin_action) action. Post actions for buildprog and checkprogsize are not triggered. Registering these callbacks only in env, only in projenv or in both makes no AddPostAction() callback trigger – strange.

ivankravets · July 1, 2022, 5:15pm

Thanks! Fixed! Please re-test with pio upgrade --dev. Should finally work.

maxgerhardt · July 1, 2022, 6:41pm

After that, all actions trigger when using e = DefaultEnvironment()

avr-g++ -o .pio\build\uno\firmware.elf -mmcu=atmega328p -Os -Wl,--gc-sections -flto -fuse-linker-plugin .pio\build\uno\src\main.cpp.o -L.pio\build\uno -Wl,--start-group .pio\build\uno\libfc0\libTestLib.a .pio\build\uno\libFrameworkArduinoVariant.a .pio\build\uno\libFrameworkArduino.a -lm -Wl,--end-group
MethodWrapper(["checkprogsize"], [".pio\build\uno\firmware.elf"])
Advanced Memory Usage is available via "PlatformIO Home > Project Inspect"
RAM:   [          ]   0.4% (used 9 bytes from 2048 bytes)
Flash: [          ]   1.4% (used 444 bytes from 32256 bytes)
[...]
post_progsize(["checkprogsize"], [".pio\build\uno\firmware.elf"])
>>>>>>>>test post_progsize() is called<<<<<<<<<<<<
avr-objcopy -O ihex -R .eeprom .pio\build\uno\firmware.elf .pio\build\uno\firmware.hex
post_bin_action([".pio\build\uno\firmware.hex"], [".pio\build\uno\firmware.elf"])
>>>>>>>>test post_bin_action() is called<<<<<<<<<<<<
copy_firmware(["buildprog"], [".pio\build\uno\firmware.hex"])
>>>>>>>>test copy_firmware() is called<<<<<<<<<<<<

But still none of the callbacks fire when using env and projenv – maybe I’m misunderstanding these environments?

ivankravets · July 2, 2022, 12:39pm

This is correct behavior. There are no targets in these envs on which you plan to assign actions. These targets are declared in the main/global env where “progenv” and “env” (lib env) are cloned/isolated environments. We have this info in the docs Construction Environments — PlatformIO latest documentation

sivar2311 · July 10, 2022, 6:07am

Sorry for the late feedback.
I have tested the latest dev version and 6.1.0.
In both versions it works with DefaultEnvironment() (as the note in the documentation describes).