PlatformIO Community

How do I run a script before compiling a project?

Before compiling the project, I need to run the generate.sh or generate.bat script each time, which first compresses the file data and then forms a C array from it and saves it as a header file.
The problem is that depending on the situation (release / debug), the script performs different actions.
How can I implement the correct execution of my script just before compiling the project via extra_skript.py?
The current implementation of the script does not work.
My script takes too long to execute, resulting in errors.

Building in release mode
Compiling .pio/build/release/esp-idf/main/ClickEncoder.c.o
Compiling .pio/build/release/esp-idf/main/addon.c.o
Compiling .pio/build/release/esp-idf/main/addonucg.c.o
Compiling .pio/build/release/esp-idf/main/bt_x01.c.o
Compiling .pio/build/release/esp-idf/main/cencode.c.o
Compiling .pio/build/release/esp-idf/main/custom.c.o
Compiling .pio/build/release/esp-idf/main/eeprom.c.o
Compiling .pio/build/release/esp-idf/main/gpios.c.o
Compiling .pio/build/release/esp-idf/main/interface.c.o
Compiling .pio/build/release/esp-idf/main/irnec.c.o
before_build([".pio/build/release/esp-idf/main/main.c.o"], ["main/main.c"])
before_build WORK
Requesting mini-me of webpage/script.js. . .
Compiling .pio/build/release/esp-idf/main/ota.c.o
Compiling .pio/build/release/esp-idf/main/servers.c.o
Compiling .pio/build/release/esp-idf/main/telnet.c.o
Compiling .pio/build/release/esp-idf/main/vs1053.c.o
Compiling .pio/build/release/esp-idf/main/vs1053b-patches.c.o
Compiling .pio/build/release/esp-idf/main/webclient.c.o
Compiling .pio/build/release/esp-idf/main/webserver.c.o
Compiling .pio/build/release/esp-idf/main/websocket.c.o
In file included from main/webserver.c:10:
include/serv-fs.h:23:10: fatal error: release_page.h: No such file or directory

My platfomio.ini:

; PlatformIO Project Configuration File
;
;   Build options: build flags, source filter
;   Upload options: custom upload port, speed and extra flags
;   Library options: dependencies, extra library storages
;   Advanced options: extra scripting
;
; Please visit documentation for the other options and examples
; https://docs.platformio.org/page/projectconf.html

[platformio]
src_dir = main
lib_dir = components
description = ESP32-Radiola
default_envs = release

[env]
platform = https://github.com/platformio/platform-espressif32.git
framework = espidf
board = esp-wrover-kit
monitor_speed = 115200

board_build.partitions = partitions.csv

monitor_flags = 
	--echo
	--raw

extra_scripts = extra_script.py

[common]
build_flags =
	-DCONFIG_SPIRAM_CACHE_WORKAROUND
	-Iinclude
	-Icomponents/audio_player/src
	-Icomponents/tda7313
	-Icomponents/fifo/src
	-Icomponents/ucglib/src
	-Icomponents/xpt2046
	-DCONFIG_RELEASE

[env:release]
build_type = release

build_flags = 
	-D FILENAME=ESP32Radiola-release
	${common.build_flags}

[env:debug]
build_type = debug

monitor_filters = esp32_exception_decoder

build_flags =
	-D FILENAME=ESP32Radiola-debug
	${common.build_flags}
	-UCONFIG_RELEASE

My extra_script.py:

# -*- coding: utf-8 -*-
import sys
from importlib import reload
# from os.path import isfile, join
from sys import platform
import requests
import subprocess
Import("env")

filename = ""
my_flags = env.ParseFlags(env['BUILD_FLAGS'])
#
for x in my_flags.get("CPPDEFINES"):
    # print(x)
    if isinstance(x, list):
        # grab as key, value
        k, v = x
        if k == "FILENAME":
            filename = v
            # no need to iterate further
            break
env.Replace(PROGNAME="%s" % str(filename))
# print(env.Dump())
# print(env.subst("$PIOENV"))
# print(env.subst("$TARGET"))
# exit(0)

BUILD_DIR = env.subst("$BUILD_DIR")


def before_build(source, target, env):
    print("before_build WORK")
    #exit(0)
    PROJECT_INCLUDE_DIR = env.subst("$PROJECT_INCLUDE_DIR")
    mode = env.subst("$PIOENV")
    if mode == 'release':
        try:
            js_file = 'webpage/script.js'
        except:
            print("Missing input file")
            sys.exit()
        # Grab the file contents
        with open(js_file, 'r') as c:
            js = c.read()
        # Pack it, ship it
        payload = {'input': js}
        url = 'https://javascript-minifier.com/raw'
        print("Requesting mini-me of {}. . .".format(c.name))
        r = requests.post(url, payload)
        # Write out minified version
        minified = js_file.rstrip('.js')+'.min.js'
        with open(minified, 'w') as m:
            m.write(r.text)
        print("Minification complete. See {}".format(m.name))
    if platform == "linux":
        # linux
        subprocess.call("./generate.sh " +
                        PROJECT_INCLUDE_DIR + ' ' + mode, shell=True)
    elif platform == "darwin":
        # OS X
        pass
    elif platform == "win32":
        # Windows...
        subprocess.call('./generate.bat' +
                        PROJECT_INCLUDE_DIR + ' ' + mode, shell=True)


env.AddPreAction("${BUILD_DIR}/esp-idf/main/main.c.o", before_build)

I execute the command pio run --list-targets

Environment    Group     Name         Title                        Description
-------------  --------  -----------  ---------------------------  -----------------------------------------------------
release        Advanced  compiledb    Compilation Database         Generate compilation database `compile_commands.json`
release        Generic   clean        Clean
release        Platform  buildfs      Build Filesystem Image
release        Platform  erase        Erase Flash
release        Platform  menuconfig   Run Menuconfig
release        Platform  size         Program Size                 Calculate program size
release        Platform  upload       Upload
release        Platform  uploadfs     Upload Filesystem Image
release        Platform  uploadfsota  Upload Filesystem Image OTA

debug          Advanced  compiledb    Compilation Database         Generate compilation database `compile_commands.json`
debug          Generic   clean        Clean
debug          Platform  buildfs      Build Filesystem Image
debug          Platform  erase        Erase Flash
debug          Platform  menuconfig   Run Menuconfig
debug          Platform  size         Program Size                 Calculate program size
debug          Platform  upload       Upload
debug          Platform  uploadfs     Upload Filesystem Image
debug          Platform  uploadfsota  Upload Filesystem Image OTA

Why isn’t buildprog listed?
Although there is a buildprog on the documentation page.
If in extra_script.py line env.AddPreAction("${BUILD_DIR}/esp-idf/main/main.c.o", before_build) is replaced by env.AddPreAction("buildprog", before_build), then my script is still launched AFTER the project is built.

Archiving .pio/build/release/lib718/libaudio_player.a
Indexing .pio/build/release/lib718/libaudio_player.a
Archiving .pio/build/release/lib47d/libtda7313.a
Indexing .pio/build/release/lib47d/libtda7313.a
Archiving .pio/build/release/lib350/libucglib.a
Indexing .pio/build/release/lib350/libucglib.a
Linking .pio/build/release/firmware.elf
Building .pio/build/release/firmware.bin
esptool.py v2.6
before_build(["buildprog"], [".pio/build/release/firmware.bin"])
before_build WORK
*** [buildprog] Explicit exit, status 0

Where is the logic?

Solution by example from another project that builds a website with npm before the build is started:

The script does not rely on any PreAction or anything, it just runs directly the function.

Thanks for the example provided. But alas for me it is useless. I need to execute the script or function ONLY before COMPILING the project. In your example, build_web () is executed on any event.

Ah okay. But if the file release_page.h is only needed in webserver.c, then why not hook its pre-action with env.AddPreAction("${BUILD_DIR}/esp-idf/main/webserver.c.o", before_build)? Building shouldn’t continue then until the script function has finished executing.

It’s strange. I’ve tried your the version earlier.
The compilation of the project did not stop, it continued in parallel with the execution of the script. I assumed that the compilation would stop before the script finished, I interrupted the compilation with the certainty that it would not work.

.........
Compiling .pio/build/release/esp-idf/main/webclient.c.o
before_build([".pio/build/release/esp-idf/main/webserver.c.o"], ["main/webserver.c"])
before_build WORKING...
Requesting mini-me of webpage/script.js. . .
Compiling .pio/build/release/esp-idf/main/websocket.c.o
Generating LD script .pio/build/release/esp32_out.ld
Generating partitions .pio/build/release/partitions.bin
Compiling .pio/build/release/esp-idf/app_trace/app_trace.c.o
Compiling .pio/build/release/esp-idf/app_trace/app_trace_util.c.o
Compiling .pio/build/release/esp-idf/app_trace/host_file_io.c.o
Compiling .pio/build/release/esp-idf/app_trace/gcov/gcov_rtio.c.o
Compiling .pio/build/release/esp-idf/app_update/esp_ota_ops.c.o
Compiling .pio/build/release/esp-idf/app_update/esp_app_desc.c.o
Archiving .pio/build/release/esp-idf/app_trace/libapp_trace.a
Indexing .pio/build/release/esp-idf/app_trace/libapp_trace.a
Compiling .pio/build/release/esp-idf/main/webserver.c.o
..........

Applied Your method again.
There were no errors and the project was put together, probably, correctly.
I will check.
Thanks for forcing me to repeat what I went through :slight_smile:
Now My need to think about how to solve one more problem. The before_build function will run every time, regardless of the actual change in the generated files. Probably need to check the checksums of the source files.
Thanks again!

There is definitely multi-threading involved; If you have 16 threads in your CPU, the compilation of 16 files is kicked off in parallel. So it can be possible that webserver.c is starting to compile in parallel with other files, but the actual C compilation of the file should only be started after the prebuid action has been run – otherwise the core has a bug. You should clean and repeat compilation a few times to see whether there is some kind of race-condition involved though when multithreading.

Multithreading can be controlled with the -j option in a pio run command by the way, for debugging. So -j1 should make is singlethreaded.

>pio run --help
Usage: pio run [OPTIONS]

Options:
  -e, --environment TEXT
  -t, --target TEXT
  --upload-port TEXT
  -d, --project-dir PATH
  -c, --project-conf FILE
  -j, --jobs INTEGER       Allow N jobs at once. Default is a number of CPUs
                           in a system (N=16)
  -s, --silent
  -v, --verbose
  --disable-auto-clean
  --list-targets
  -h, --help               Show this message and exit.

I have modest: only 4 cores :slight_smile:
I will check. Thanks again!