Can't get extra_script Scons Commands to run at all, must be doing something fundamentally wrong

I’ve read-up on SCons and created a really straightforward extra_script, but whatever I do I can’t get either of the Commands to actually run!:

import os

Import("env")

release_dir = "releases"


def create_release(target, source, env):
    print("target=%s, source=%s" % (target[0], source[0]))

    firmware_version = env.GetProjectOption("custom_build_number")
    tag = env['PIOENV']

    release_file = env.Command(
        os.path.join(release_dir, tag + "_" + source[0].name),
        source[0],
        action=[
            Mkdir(release_dir),
            Copy("$TARGET", "$SOURCE"),
            sayHi
        ])

    print("release_file[0] is %s" % release_file[0])

    release_descriptor = env.Command(
        os.path.join(release_dir, tag + ".json"),
        release_file,
        sayHi)


def sayHi():
    print("Hi")

script_action = env.AddCustomTarget(
    "create_release",
    ["$BUILD_DIR/${PROGNAME}.bin"],
    create_release,
    title="Create Release",
    description="Deploy to release folder and create descriptor file",
    always_build=True,
)

The extra_script is being found just fine, and the print calls are running as the script is parsed, because the output (snipped) is:

rebuilding `checkprogsize' because AlwaysBuild() is specified
Retrieving maximum program size .pio/build/bench_test/firmware_2.4.elf
Checking size .pio/build/bench_test/firmware_2.4.elf
Advanced Memory Usage is available via "PlatformIO Home > Project Inspect"
RAM:   [=         ]  11.5% (used 37836 bytes from 327680 bytes)
Flash: [=======   ]  71.0% (used 1488093 bytes from 2097152 bytes)
rebuilding `create_release' because AlwaysBuild() is specified
create_release(["create_release"], [".pio/build/bench_test/firmware_2.4.bin"])
target=create_release, source=.pio/build/bench_test/firmware_2.4.bin
release_file[0] is releases/bench_test_firmware_2.4.bin

But the directory is never created, the file is never copied and - as you can see above - the sayHi() method doesn’t ever get called.

Where am I going wrong? I’ve tried setting Default, Alias, Depends and any number of other things - but my SCons commands are never run, the build just finishes :frowning: Please help!

(I should add that I’ve got a pre: script appending the _2.4 to the PROGNAME as per one of the examples in the docs, and the file does get compiled just fine)

Does anyone have any insight or experience they could share on this one please? I can use vanilla python scripts, but I find that I have to have AlwaysBuild=True otherwise SCons seems to struggle to know when a rebuild of the job is required.

I suspect that being able to use proper SCons Commands etc would also allow me to hook into them from other targets/ actions… env.Depends() doesn’t seem to work with my custom actions or targets

@valeros if you get a moment I’d be really grateful for any pointers you could offer - it looks like you committed a lot of the core code that interacts with scons. I’ve been through it but can’t tell why my extra_script code is running differently to your core code.

Thank you!

Hi @mrfellows ! I guess it doesn’t work because in your action function create_release you’re just creating additional targets (via env.Command) that are basically orphans as they’re not connected to anything and hence not visible in the dependency tree. You possibly can manually add that new targets to the dependency tree via the env.Depends command (e.g. env.Depends("create_release", release_descriptor)). But why bother, if you can immediately execute any commands via env.Execute or even using generic Python code (e.g. via shutil or subprocess).

Interesting, and thank you @valeros. I would have thought that was implicit in the env.addCustomTarget call (especially given that always_build=True) that when I call that target via pio run -t create_release I would want to trigger the scons steps but I see what you mean.

Where would I put the env.Depends("create_release", release_descriptor) command? I’ve included it at the end of the create_release function, but the commands still aren’t triggered. I know the script is being loaded because the two debug prints run - but the commands don’t trigger to either sayHi or create the subdir.

The below should be a minimal reproducible example - just add this file to extra_scripts in platformio.ini.

import os

Import("env")

release_dir = "releases"


def create_release(target, source, env):
    print("target=%s, source=%s" % (target[0], source[0]))

    tag = env['PIOENV']

    release_file = env.Command(
        os.path.join(release_dir, tag + "_" + source[0].name),
        source[0],
        action=[
            Mkdir(release_dir),
            Copy("$TARGET", "$SOURCE"),
            sayHi
        ])

    print("release_file[0] is %s" % release_file[0])

    release_descriptor = env.Command(
        os.path.join(release_dir, tag + ".json"),
        release_file,
        sayHi)

    env.Depends("create_release", release_descriptor)



def sayHi():
    print("Hi")

script_action = env.AddCustomTarget(
    "create_release",
    ["$BUILD_DIR/${PROGNAME}.bin"],
    create_release,
    title="Create Release",
    description="Deploy to release folder and create descriptor file",
    always_build=True,
)

Because this is just a minimal example showing the issue. I’ve got a few different tasks that I’d like to include as extra_scripts that depend on each other but that should only run when they’re stale or when a predecessor has rebuilt. The power of Scons looks like it fits the bill perfectly, and you’ve already built it into Platformio so I would love to crack this so I can take advantage of it :slight_smile:

Hmm, could you please share here your dependency tree? Just set an additional SCons flag via an env variable export SCONSFLAGS="--tree=derived" and run your target again.

I’ve created an empty project with just a main.cpp and this extra script to try and cut-down the noise in the dependency tree but there’s still all the arduino files. I’ve pasted everything including and after the two prints that happen when the extra_script is being parsed/ loaded:

create_release(["create_release"], [".pio/build/esp32dev/firmware.bin"])
target=create_release, source=.pio/build/esp32dev/firmware.bin
release_file[0] is releases/esp32dev_firmware.bin
+-create_release
  +-.pio/build/esp32dev/firmware.bin
  | +-.pio/build/esp32dev/firmware.elf
  | | +-.pio/build/esp32dev/src/main.cpp.o
  | | +-.pio/build/esp32dev/bootloader.bin
  | | +-.pio/build/esp32dev/partitions.bin
  | | +-.pio/build/esp32dev/libFrameworkArduino.a
  | |   +-.pio/build/esp32dev/FrameworkArduino/Esp.cpp.o
  | |   +-.pio/build/esp32dev/FrameworkArduino/FirmwareMSC.cpp.o
  | |   +-.pio/build/esp32dev/FrameworkArduino/FunctionalInterrupt.cpp.o
  | |   +-.pio/build/esp32dev/FrameworkArduino/HWCDC.cpp.o
  | |   +-.pio/build/esp32dev/FrameworkArduino/HardwareSerial.cpp.o
  | |   +-.pio/build/esp32dev/FrameworkArduino/IPAddress.cpp.o
  | |   +-.pio/build/esp32dev/FrameworkArduino/IPv6Address.cpp.o
  | |   +-.pio/build/esp32dev/FrameworkArduino/MD5Builder.cpp.o
  | |   +-.pio/build/esp32dev/FrameworkArduino/Print.cpp.o
  | |   +-.pio/build/esp32dev/FrameworkArduino/Stream.cpp.o
  | |   +-.pio/build/esp32dev/FrameworkArduino/StreamString.cpp.o
  | |   +-.pio/build/esp32dev/FrameworkArduino/Tone.cpp.o
  | |   +-.pio/build/esp32dev/FrameworkArduino/USB.cpp.o
  | |   +-.pio/build/esp32dev/FrameworkArduino/USBCDC.cpp.o
  | |   +-.pio/build/esp32dev/FrameworkArduino/USBMSC.cpp.o
  | |   +-.pio/build/esp32dev/FrameworkArduino/WMath.cpp.o
  | |   +-.pio/build/esp32dev/FrameworkArduino/WString.cpp.o
  | |   +-.pio/build/esp32dev/FrameworkArduino/base64.cpp.o
  | |   +-.pio/build/esp32dev/FrameworkArduino/cbuf.cpp.o
  | |   +-.pio/build/esp32dev/FrameworkArduino/esp32-hal-adc.c.o
  | |   +-.pio/build/esp32dev/FrameworkArduino/esp32-hal-bt.c.o
  | |   +-.pio/build/esp32dev/FrameworkArduino/esp32-hal-cpu.c.o
  | |   +-.pio/build/esp32dev/FrameworkArduino/esp32-hal-dac.c.o
  | |   +-.pio/build/esp32dev/FrameworkArduino/esp32-hal-gpio.c.o
  | |   +-.pio/build/esp32dev/FrameworkArduino/esp32-hal-i2c-slave.c.o
  | |   +-.pio/build/esp32dev/FrameworkArduino/esp32-hal-i2c.c.o
  | |   +-.pio/build/esp32dev/FrameworkArduino/esp32-hal-ledc.c.o
  | |   +-.pio/build/esp32dev/FrameworkArduino/esp32-hal-matrix.c.o
  | |   +-.pio/build/esp32dev/FrameworkArduino/esp32-hal-misc.c.o
  | |   +-.pio/build/esp32dev/FrameworkArduino/esp32-hal-psram.c.o
  | |   +-.pio/build/esp32dev/FrameworkArduino/esp32-hal-rgb-led.c.o
  | |   +-.pio/build/esp32dev/FrameworkArduino/esp32-hal-rmt.c.o
  | |   +-.pio/build/esp32dev/FrameworkArduino/esp32-hal-sigmadelta.c.o
  | |   +-.pio/build/esp32dev/FrameworkArduino/esp32-hal-spi.c.o
  | |   +-.pio/build/esp32dev/FrameworkArduino/esp32-hal-time.c.o
  | |   +-.pio/build/esp32dev/FrameworkArduino/esp32-hal-timer.c.o
  | |   +-.pio/build/esp32dev/FrameworkArduino/esp32-hal-tinyusb.c.o
  | |   +-.pio/build/esp32dev/FrameworkArduino/esp32-hal-touch.c.o
  | |   +-.pio/build/esp32dev/FrameworkArduino/esp32-hal-uart.c.o
  | |   +-.pio/build/esp32dev/FrameworkArduino/firmware_msc_fat.c.o
  | |   +-.pio/build/esp32dev/FrameworkArduino/libb64/cdecode.c.o
  | |   +-.pio/build/esp32dev/FrameworkArduino/libb64/cencode.c.o
  | |   +-.pio/build/esp32dev/FrameworkArduino/main.cpp.o
  | |   +-.pio/build/esp32dev/FrameworkArduino/stdlib_noniso.c.o
  | |   +-.pio/build/esp32dev/FrameworkArduino/wiring_pulse.c.o
  | |   +-.pio/build/esp32dev/FrameworkArduino/wiring_shift.c.o
  | +-checkprogsize
  |   +-.pio/build/esp32dev/firmware.elf
  |     +-.pio/build/esp32dev/src/main.cpp.o
  |     +-.pio/build/esp32dev/bootloader.bin
  |     +-.pio/build/esp32dev/partitions.bin
  |     +-.pio/build/esp32dev/libFrameworkArduino.a
  |       +-.pio/build/esp32dev/FrameworkArduino/Esp.cpp.o
  |       +-.pio/build/esp32dev/FrameworkArduino/FirmwareMSC.cpp.o
  |       +-.pio/build/esp32dev/FrameworkArduino/FunctionalInterrupt.cpp.o
  |       +-.pio/build/esp32dev/FrameworkArduino/HWCDC.cpp.o
  |       +-.pio/build/esp32dev/FrameworkArduino/HardwareSerial.cpp.o
  |       +-.pio/build/esp32dev/FrameworkArduino/IPAddress.cpp.o
  |       +-.pio/build/esp32dev/FrameworkArduino/IPv6Address.cpp.o
  |       +-.pio/build/esp32dev/FrameworkArduino/MD5Builder.cpp.o
  |       +-.pio/build/esp32dev/FrameworkArduino/Print.cpp.o
  |       +-.pio/build/esp32dev/FrameworkArduino/Stream.cpp.o
  |       +-.pio/build/esp32dev/FrameworkArduino/StreamString.cpp.o
  |       +-.pio/build/esp32dev/FrameworkArduino/Tone.cpp.o
  |       +-.pio/build/esp32dev/FrameworkArduino/USB.cpp.o
  |       +-.pio/build/esp32dev/FrameworkArduino/USBCDC.cpp.o
  |       +-.pio/build/esp32dev/FrameworkArduino/USBMSC.cpp.o
  |       +-.pio/build/esp32dev/FrameworkArduino/WMath.cpp.o
  |       +-.pio/build/esp32dev/FrameworkArduino/WString.cpp.o
  |       +-.pio/build/esp32dev/FrameworkArduino/base64.cpp.o
  |       +-.pio/build/esp32dev/FrameworkArduino/cbuf.cpp.o
  |       +-.pio/build/esp32dev/FrameworkArduino/esp32-hal-adc.c.o
  |       +-.pio/build/esp32dev/FrameworkArduino/esp32-hal-bt.c.o
  |       +-.pio/build/esp32dev/FrameworkArduino/esp32-hal-cpu.c.o
  |       +-.pio/build/esp32dev/FrameworkArduino/esp32-hal-dac.c.o
  |       +-.pio/build/esp32dev/FrameworkArduino/esp32-hal-gpio.c.o
  |       +-.pio/build/esp32dev/FrameworkArduino/esp32-hal-i2c-slave.c.o
  |       +-.pio/build/esp32dev/FrameworkArduino/esp32-hal-i2c.c.o
  |       +-.pio/build/esp32dev/FrameworkArduino/esp32-hal-ledc.c.o
  |       +-.pio/build/esp32dev/FrameworkArduino/esp32-hal-matrix.c.o
  |       +-.pio/build/esp32dev/FrameworkArduino/esp32-hal-misc.c.o
  |       +-.pio/build/esp32dev/FrameworkArduino/esp32-hal-psram.c.o
  |       +-.pio/build/esp32dev/FrameworkArduino/esp32-hal-rgb-led.c.o
  |       +-.pio/build/esp32dev/FrameworkArduino/esp32-hal-rmt.c.o
  |       +-.pio/build/esp32dev/FrameworkArduino/esp32-hal-sigmadelta.c.o
  |       +-.pio/build/esp32dev/FrameworkArduino/esp32-hal-spi.c.o
  |       +-.pio/build/esp32dev/FrameworkArduino/esp32-hal-time.c.o
  |       +-.pio/build/esp32dev/FrameworkArduino/esp32-hal-timer.c.o
  |       +-.pio/build/esp32dev/FrameworkArduino/esp32-hal-tinyusb.c.o
  |       +-.pio/build/esp32dev/FrameworkArduino/esp32-hal-touch.c.o
  |       +-.pio/build/esp32dev/FrameworkArduino/esp32-hal-uart.c.o
  |       +-.pio/build/esp32dev/FrameworkArduino/firmware_msc_fat.c.o
  |       +-.pio/build/esp32dev/FrameworkArduino/libb64/cdecode.c.o
  |       +-.pio/build/esp32dev/FrameworkArduino/libb64/cencode.c.o
  |       +-.pio/build/esp32dev/FrameworkArduino/main.cpp.o
  |       +-.pio/build/esp32dev/FrameworkArduino/stdlib_noniso.c.o
  |       +-.pio/build/esp32dev/FrameworkArduino/wiring_pulse.c.o
  |       +-.pio/build/esp32dev/FrameworkArduino/wiring_shift.c.o
  +-releases/esp32dev.json
    +-releases/esp32dev_firmware.bin
      +-.pio/build/esp32dev/firmware.bin
        +-.pio/build/esp32dev/firmware.elf
        | +-.pio/build/esp32dev/src/main.cpp.o
        | +-.pio/build/esp32dev/bootloader.bin
        | +-.pio/build/esp32dev/partitions.bin
        | +-.pio/build/esp32dev/libFrameworkArduino.a
        |   +-.pio/build/esp32dev/FrameworkArduino/Esp.cpp.o
        |   +-.pio/build/esp32dev/FrameworkArduino/FirmwareMSC.cpp.o
        |   +-.pio/build/esp32dev/FrameworkArduino/FunctionalInterrupt.cpp.o
        |   +-.pio/build/esp32dev/FrameworkArduino/HWCDC.cpp.o
        |   +-.pio/build/esp32dev/FrameworkArduino/HardwareSerial.cpp.o
        |   +-.pio/build/esp32dev/FrameworkArduino/IPAddress.cpp.o
        |   +-.pio/build/esp32dev/FrameworkArduino/IPv6Address.cpp.o
        |   +-.pio/build/esp32dev/FrameworkArduino/MD5Builder.cpp.o
        |   +-.pio/build/esp32dev/FrameworkArduino/Print.cpp.o
        |   +-.pio/build/esp32dev/FrameworkArduino/Stream.cpp.o
        |   +-.pio/build/esp32dev/FrameworkArduino/StreamString.cpp.o
        |   +-.pio/build/esp32dev/FrameworkArduino/Tone.cpp.o
        |   +-.pio/build/esp32dev/FrameworkArduino/USB.cpp.o
        |   +-.pio/build/esp32dev/FrameworkArduino/USBCDC.cpp.o
        |   +-.pio/build/esp32dev/FrameworkArduino/USBMSC.cpp.o
        |   +-.pio/build/esp32dev/FrameworkArduino/WMath.cpp.o
        |   +-.pio/build/esp32dev/FrameworkArduino/WString.cpp.o
        |   +-.pio/build/esp32dev/FrameworkArduino/base64.cpp.o
        |   +-.pio/build/esp32dev/FrameworkArduino/cbuf.cpp.o
        |   +-.pio/build/esp32dev/FrameworkArduino/esp32-hal-adc.c.o
        |   +-.pio/build/esp32dev/FrameworkArduino/esp32-hal-bt.c.o
        |   +-.pio/build/esp32dev/FrameworkArduino/esp32-hal-cpu.c.o
        |   +-.pio/build/esp32dev/FrameworkArduino/esp32-hal-dac.c.o
        |   +-.pio/build/esp32dev/FrameworkArduino/esp32-hal-gpio.c.o
        |   +-.pio/build/esp32dev/FrameworkArduino/esp32-hal-i2c-slave.c.o
        |   +-.pio/build/esp32dev/FrameworkArduino/esp32-hal-i2c.c.o
        |   +-.pio/build/esp32dev/FrameworkArduino/esp32-hal-ledc.c.o
        |   +-.pio/build/esp32dev/FrameworkArduino/esp32-hal-matrix.c.o
        |   +-.pio/build/esp32dev/FrameworkArduino/esp32-hal-misc.c.o
        |   +-.pio/build/esp32dev/FrameworkArduino/esp32-hal-psram.c.o
        |   +-.pio/build/esp32dev/FrameworkArduino/esp32-hal-rgb-led.c.o
        |   +-.pio/build/esp32dev/FrameworkArduino/esp32-hal-rmt.c.o
        |   +-.pio/build/esp32dev/FrameworkArduino/esp32-hal-sigmadelta.c.o
        |   +-.pio/build/esp32dev/FrameworkArduino/esp32-hal-spi.c.o
        |   +-.pio/build/esp32dev/FrameworkArduino/esp32-hal-time.c.o
        |   +-.pio/build/esp32dev/FrameworkArduino/esp32-hal-timer.c.o
        |   +-.pio/build/esp32dev/FrameworkArduino/esp32-hal-tinyusb.c.o
        |   +-.pio/build/esp32dev/FrameworkArduino/esp32-hal-touch.c.o
        |   +-.pio/build/esp32dev/FrameworkArduino/esp32-hal-uart.c.o
        |   +-.pio/build/esp32dev/FrameworkArduino/firmware_msc_fat.c.o
        |   +-.pio/build/esp32dev/FrameworkArduino/libb64/cdecode.c.o
        |   +-.pio/build/esp32dev/FrameworkArduino/libb64/cencode.c.o
        |   +-.pio/build/esp32dev/FrameworkArduino/main.cpp.o
        |   +-.pio/build/esp32dev/FrameworkArduino/stdlib_noniso.c.o
        |   +-.pio/build/esp32dev/FrameworkArduino/wiring_pulse.c.o
        |   +-.pio/build/esp32dev/FrameworkArduino/wiring_shift.c.o
        +-checkprogsize
          +-.pio/build/esp32dev/firmware.elf
            +-.pio/build/esp32dev/src/main.cpp.o
            +-.pio/build/esp32dev/bootloader.bin
            +-.pio/build/esp32dev/partitions.bin
            +-.pio/build/esp32dev/libFrameworkArduino.a
              +-.pio/build/esp32dev/FrameworkArduino/Esp.cpp.o
              +-.pio/build/esp32dev/FrameworkArduino/FirmwareMSC.cpp.o
              +-.pio/build/esp32dev/FrameworkArduino/FunctionalInterrupt.cpp.o
              +-.pio/build/esp32dev/FrameworkArduino/HWCDC.cpp.o
              +-.pio/build/esp32dev/FrameworkArduino/HardwareSerial.cpp.o
              +-.pio/build/esp32dev/FrameworkArduino/IPAddress.cpp.o
              +-.pio/build/esp32dev/FrameworkArduino/IPv6Address.cpp.o
              +-.pio/build/esp32dev/FrameworkArduino/MD5Builder.cpp.o
              +-.pio/build/esp32dev/FrameworkArduino/Print.cpp.o
              +-.pio/build/esp32dev/FrameworkArduino/Stream.cpp.o
              +-.pio/build/esp32dev/FrameworkArduino/StreamString.cpp.o
              +-.pio/build/esp32dev/FrameworkArduino/Tone.cpp.o
              +-.pio/build/esp32dev/FrameworkArduino/USB.cpp.o
              +-.pio/build/esp32dev/FrameworkArduino/USBCDC.cpp.o
              +-.pio/build/esp32dev/FrameworkArduino/USBMSC.cpp.o
              +-.pio/build/esp32dev/FrameworkArduino/WMath.cpp.o
              +-.pio/build/esp32dev/FrameworkArduino/WString.cpp.o
              +-.pio/build/esp32dev/FrameworkArduino/base64.cpp.o
              +-.pio/build/esp32dev/FrameworkArduino/cbuf.cpp.o
              +-.pio/build/esp32dev/FrameworkArduino/esp32-hal-adc.c.o
              +-.pio/build/esp32dev/FrameworkArduino/esp32-hal-bt.c.o
              +-.pio/build/esp32dev/FrameworkArduino/esp32-hal-cpu.c.o
              +-.pio/build/esp32dev/FrameworkArduino/esp32-hal-dac.c.o
              +-.pio/build/esp32dev/FrameworkArduino/esp32-hal-gpio.c.o
              +-.pio/build/esp32dev/FrameworkArduino/esp32-hal-i2c-slave.c.o
              +-.pio/build/esp32dev/FrameworkArduino/esp32-hal-i2c.c.o
              +-.pio/build/esp32dev/FrameworkArduino/esp32-hal-ledc.c.o
              +-.pio/build/esp32dev/FrameworkArduino/esp32-hal-matrix.c.o
              +-.pio/build/esp32dev/FrameworkArduino/esp32-hal-misc.c.o
              +-.pio/build/esp32dev/FrameworkArduino/esp32-hal-psram.c.o
              +-.pio/build/esp32dev/FrameworkArduino/esp32-hal-rgb-led.c.o
              +-.pio/build/esp32dev/FrameworkArduino/esp32-hal-rmt.c.o
              +-.pio/build/esp32dev/FrameworkArduino/esp32-hal-sigmadelta.c.o
              +-.pio/build/esp32dev/FrameworkArduino/esp32-hal-spi.c.o
              +-.pio/build/esp32dev/FrameworkArduino/esp32-hal-time.c.o
              +-.pio/build/esp32dev/FrameworkArduino/esp32-hal-timer.c.o
              +-.pio/build/esp32dev/FrameworkArduino/esp32-hal-tinyusb.c.o
              +-.pio/build/esp32dev/FrameworkArduino/esp32-hal-touch.c.o
              +-.pio/build/esp32dev/FrameworkArduino/esp32-hal-uart.c.o
              +-.pio/build/esp32dev/FrameworkArduino/firmware_msc_fat.c.o
              +-.pio/build/esp32dev/FrameworkArduino/libb64/cdecode.c.o
              +-.pio/build/esp32dev/FrameworkArduino/libb64/cencode.c.o
              +-.pio/build/esp32dev/FrameworkArduino/main.cpp.o
              +-.pio/build/esp32dev/FrameworkArduino/stdlib_noniso.c.o
              +-.pio/build/esp32dev/FrameworkArduino/wiring_pulse.c.o
              +-.pio/build/esp32dev/FrameworkArduino/wiring_shift.c.o

Could you try the following?

env.Depends("$BUILD_DIR/${PROGNAME}.bin", release_descriptor)

Frankly speaking, I don’t understand why you need here chained commands and SCons “internals”. I would replace “env.Command” logic with native Python’s os/shutil.

Thanks for taking a look Ivan.

Replacing the env.Depends command so that the binary file creation depends on release_decriptor also doesn’t help. The directory isn’t created, and “Hi” isn’t printed:

Scanning dependencies...
No dependencies
Building in release mode
Building .pio/build/esp32dev/bootloader.bin
Generating partitions .pio/build/esp32dev/partitions.bin
esptool.py v4.2.1
Creating esp32 image...
Merged 1 ELF section
Successfully created esp32 image.
Retrieving maximum program size .pio/build/esp32dev/firmware.elf
Checking size .pio/build/esp32dev/firmware.elf
Advanced Memory Usage is available via "PlatformIO Home > Project Inspect"
RAM:   [=         ]   5.0% (used 16440 bytes from 327680 bytes)
Flash: [==        ]  18.6% (used 244065 bytes from 1310720 bytes)
Building .pio/build/esp32dev/firmware.bin
esptool.py v4.2.1
Creating esp32 image...
Merged 2 ELF sections
Successfully created esp32 image.
create_release(["create_release"], [".pio/build/esp32dev/firmware.bin"])
target=create_release, source=.pio/build/esp32dev/firmware.bin
release_file[0] is releases/esp32dev_firmware.bin

It appears from the the terminal output that the binary is created before the extra_script is parsed, so I guess env.Depends is being parsed too late?

Because there are a series of tasks that form a dependency chain that I’m trying to manage, some of which are slow and each of which should only be re-run if/ when something higher up the chain has run successfully. According to the SCons docs I believe that it will also handle the “fingerprints” of my custom items in the dependency chain and only rebuild the downstream tasks if necessary. For example:

  • AFTER the binary is rebuilt, patch it with the project name, version, build-time etc (see here - it’s not done by an arduino build so must be post-processed so the ESP functions work)
  • AFTER the binary has been patched, the tests can be run
  • Also AFTER the binary has been patched, create a release descriptor and copy the file to the release directory
  • AFTER BOTH the tests have passed AND the release descriptor has been created, upload the binary and the release descriptor to the staging server so it can be distributed to test devices via OTA

It’s a fairly basic build process that is straightforward in Ant, Make or Maven and I the sort of thing you use SCons for internally within PlatformIO from what I can see? And you’ve envisaged end-users doing the same I think - I’ve been using the PIO docs on Advanced Scripting as my reference, combined with the SCons docs, to build the various extra_scripts.

I expect there’s something very basic that I’m doing wrong. But I can’t see what it is! You can clone the files for the minimal test example above from git@gitlab.com:8w/pio-scons.git and see what I mean :slight_smile:

TLDR: SCons tasks created inside the Action (the third param to env.AddCustomTarget()) don’t get triggered. Instead, the final task in the chain should be added as a dependency for the custom target (i.e. the second parameter). That way, they will be triggered correctly when needed - and only rebuilt as required by a change in an upstream dependency of the tasks.

I think I’ve cracked it, by cutting back until I could replicate @flytrex-vadim’s post here.

I believe the “issue” is that that SCons tasks created within an Action are created too late, and are never triggered. Instead, they should be created and added as a dependency of a target (either a custom one or a built-in one).

So, for example, when you run the my_target target, the code below:

  1. Creates the releases directory and copies the binary into it
  2. Creates an empty file also in that directory

Importantly, it only runs those two steps if an upstream step changes. And Scons is clever enough to track the state too, so if you run pio run and it creates a binary with a different checksum, then it will know that next time you pio run -t run my_target its tasks are stale and should be rebuilt, even though the binary doesn’t change.

import os

Import("env")

release_dir = os.path.join(env.GetLaunchDir(), "releases")


def build_task_chain(upstream_artefact):
    print(">>> Parsing tasks <<<")

    tag = env['PIOENV']

    release_file = env.Command(
        target=os.path.join(release_dir, tag + "_" + os.path.basename(upstream_artefact)),
        source=upstream_artefact,
        action=[
            Mkdir(release_dir),
            Copy("$TARGET", "$SOURCE"),
            sayHi
        ])

    release_descriptor = env.Command(
        target=os.path.join(release_dir, tag + ".json"),
        source=release_file,
        action=[
            Touch("$TARGET"),
            sayHi
        ])

    return release_descriptor


def sayHi(target, source, env):
    print(">>> Hi!  Finished creating %s from %s <<<" % (target[0], source[0]))


last_task_in_chain = build_task_chain("$BUILD_DIR/${PROGNAME}.bin");

script_action = env.AddCustomTarget(
    "my_target",
    [last_task_in_chain],
    []
)

1 Like

Sorry for the late response. Yes, this is exactly what has come to my mind later, and I wanted to share. SCons does not calls the target callbacks while building SCons’ dependency tree.

Thanks for sharing the solution. I’m sure our community will find it useful.

1 Like

I am trying to copy my firmware.hex file after a build. I’m using the solution above, and I see the “>>> Parsing tasks <<<” message, but nothing else. I changed “my_target” to “teensy31”.


[env:teensy31]
platform = teensy
board = teensy31
build_flags = 
	${env.build_flags}
	-DT32BKM=1
	-DTEENSYBKM=1
	-DSERIALCONS=Serial
lib_deps = 
	${env.lib_deps}
	PersistT32
	eepromAnything 
extra_scripts = post:copy_1.py  

I’m not a Python programmer. I don’t understand how $TARGET and $SOURCE are instantiated.

Hi Brian

SOURCE and TARGET are populated with the contents of the first and second parameters to the env.Command method and passed to each of the items in the action array.

If you’re not into python and don’t have a specific reason to get into SCons you’d be better-off using a simple function containing a shutil.copy2 call, triggered via a post action (Pre & Post Actions — PlatformIO latest documentation) :slight_smile:

1 Like

Thank you for the help.