Opening Serial Monitor runs PIO extra_tasks

PlatformIO IDE
1

Hi All.

First off a big thanks for all of the work you guys do. This resource has been invaluable in helping resolve many issues. However, I have struck a weird issue that I cannot seem to get my head around, let alone resolve.

I am developing for the ESP32 on platformIO. I am using some extra_scripts to manage version info.

extra_scripts = 
	pre:user_actions_pre.py
	post:user_actions_post.py

user_actions_pre.py is called using the ‘pre:’ hook. This task reads current version data from a file, increments the version and then updates the file. It also deletes the old compiled binaries in the releases folder. Lastly it updates the build environment variables so that the version data is available to the compiler. This works as expected.

The second task post:user_actions_post.py runs after the compiler has finished and moves the created binary into the releases folder, renaming it with the current build version as well as creating a second combined binary that can be used with a firmware uploader. This also works as expected.

However. I have found that on occasion the first task is getting called a second time, after the compiler has finished, resulting in the newly created binaries getting deleted from the releases folder.

I proved that the user_actions_pre.py file gets called again as commenting out the lines that delete the binaries stops the behaviour.

I initially tried the solution given here - env.AddPreAction() help which did resolve the user_actions_pre.py task getting called immediately after compilation, but I’ve just discovered that it now gets called when I open the serial monitor.

Ideally I only want to run the user_actions_pre.py task once, before compilation, which I assumed would be taken care of by the ‘pre:’ hook, which to a degree works, however it is then called again and I am unsure how.

I will keep digging to see if I can find an appropriate manner to programatically inhibit that part of the code, but I thought I’d reach out and try to understand why this is happening.

Any pointers / advice warmly welcomed.

/DM

Full files below for reference. Note I am building using the [env:esp32dev] environment.

Full project is available here - GitHub - DeeEmm/DIY-Flow-Bench at 205-post-compile-configuration

platformIO

; PlatformIO Project Configuration File
;
;   Build options: build flags, source filter
;   Upload options: custom upload port, speed and extra flags
;   Library options: dependencies, extra library storages
;   Advanced options: extra scripting
;
; Please visit documentation for the other options and examples
; https://docs.platformio.org/page/projectconf.html

[platformio]
src_dir = ESP32/DIY-Flow-Bench
data_dir = ESP32/diy-flow-bench/data
lib_dir = lib
libdeps_dir = libdeps
default_envs = esp32dev

[env]
platform = espressif32 ;@3.5.0
monitor_filters = esp32_exception_decoder
framework = arduino
board_build.partitions = partitions-default.csv
monitor_speed = 115200
board_build.f_cpu = 240000000L
board_build.f_flash = 80000000L
board_build.flash_mode = qio
build_flags = ${common.build_flags}
build_src_filter = 
	+<*.h> +<*.s> +<*.S> +<*.cpp> +<*.c> +<*.ino> +<src/> 
	-<.git/> -<data/> -<test/> -<tests/> -<include/> -<mafData/>
extra_scripts = 
	pre:user_actions_pre.py
	post:user_actions_post.py

	


[common_env_data]
lib_deps_builtin = 
	DNSServer
	EEPROM
	ESPmDNS
	FS
	Preferences
	SD
	SPIFFS
	Update
	WebServer
	WiFi
	WiFiClientSecure


[common]
build_flags = 
	-Wno-unused-variable
	-Wno-unused-function
	"-D TEMPLATE_PLACEHOLDER='~'"
	"-D ARDUINO_LOOP_STACK_SIZE=28160"
	; "-D xQueueCreate=256"
	-D LAST_BUILD_TIME=$UNIX_TIME
	;'-D MAJOR_VERSION="2"'
	;'-D MINOR_VERSION="0"'
	;'-D BUILD_NUMBER="UNDEFINED"'
	;'-D RELEASE="V.2.0-RC.8"'
	-D CORE_DEBUG_LEVEL=0


; General ESP32 build environment. Should work for most ESP32's
[env:esp32dev]
build_type = release
board = esp32dev
upload_protocol = esptool
; upload_speed = 921600
upload_speed = 460800
lib_ldf_mode = chain
lib_deps = 
	bblanchon/ArduinoJson@^6.19.4
	esphome/AsyncTCP-esphome
    esphome/ESPAsyncWebServer-esphome
	https://github.com/terryjmyers/ADS1115-Lite.git
	https://github.com/fabyte/Tiny_BME280_Arduino_Library.git
	; https://github.com/DeeEmm/BME680.git
	majicdesigns/MD_REncoder@^1.0.1
lib_ignore = 
	


; Build environment for esp-wrover-kit with onboard JTAG debugger only
[env:esp-wrover-kit]
build_type = debug
board = esp-wrover-kit
upload_speed = 921600
debug_tool = ftdi
debug_load_mode = modified
debug_init_break = tbreak loop
debug_speed = 500
lib_ldf_mode = chain
lib_deps = 
	bblanchon/ArduinoJson@^6.19.4
	esphome/AsyncTCP-esphome
    esphome/ESPAsyncWebServer-esphome
	https://github.com/terryjmyers/ADS1115-Lite.git
	https://github.com/fabyte/Tiny_BME280_Arduino_Library.git
	https://github.com/DeeEmm/BME680.git
lib_ignore = 
	SPI
extra_scripts =


; Build environment for M5Stack-Core2 
[env:m5stack-core2]
board = m5stack-core2
upload_protocol = esptool
upload_speed = 460800
lib_ldf_mode = chain
lib_deps = 
	bblanchon/ArduinoJson@^6.19.4
	esphome/AsyncTCP-esphome
	esphome/ESPAsyncWebServer-esphome
	https://github.com/terryjmyers/ADS1115-Lite.git
	https://github.com/fabyte/Tiny_BME280_Arduino_Library.git
	majicdesigns/MD_REncoder@^1.0.1
	;m5stack/M5Unified@^0.1.17
	;M5GFX
	lbernstone/UncleRus@^1.0.1
	M5_ADS1100
	M5Unit-PbHub
	bsec2
	BME68x Sensor library
lib_ignore = 
extra_scripts =

user_actions_pre.py

import json
import sys
import os
import datetime
import re
import shutil
from SCons.Script import Import

Import("env")

# Check build 
def is_pio_build():
    from SCons.Script import DefaultEnvironment
    env = DefaultEnvironment()
    if "IsCleanTarget" in dir(env) and env.IsCleanTarget(): return False
    return not env.IsIntegrationDump()



if is_pio_build :

    print("Pre-Build tasks")
    print("Loading version.json")

    release_path = env.subst("$PROJECT_DIR/ESP32/DIY-Flow-Bench/release/")

    # read json file into var
    file_path = 'ESP32/DIY-Flow-Bench/version.json'
    with open(file_path) as file_data:
        json_data = json.load(file_data)

    # increment build and update version.json

    # get current date
    dtnow = datetime.datetime.now()
    year = dtnow.strftime("%y")
    month = dtnow.strftime("%m")
    date = dtnow.strftime("%d")

    # read build No from json
    build_num = json_data['BUILD_NUMBER']
    release = json_data['RELEASE']
    # print(build_num  + "\n")

    # get current details and delete files
    old_merged_file = os.path.join(release_path, f"{release}_{build_num}_install.bin")
    old_update_file = os.path.join(release_path, f"{release}_{build_num}_update.bin")
    try:
        os.remove(old_merged_file)
        os.remove(old_update_file)
    except OSError:
        print("Error occurred while deleting files.")

    # print(old_merged_file)

    # get build date info from version.json
    bn_year = build_num[0:2]
    bn_month =  build_num[2:4]
    bn_date =  build_num[4:6]
    bn_inc = build_num[6:10]
    # print(bn_year  + "\n")
    # print(bn_month  + "\n")
    # print(bn_date  + "\n")
    # print(bn_inc  + "\n")

    # check build date incremental count
    if bn_year == year and bn_month == month and bn_date == date:
        # we are still on same day lets increment existing build number
        incremental = int(build_num[6:10])
        incremental += 1    
    else:
        # it's a new day start from 0001
        incremental = 1

    ## add preceding zeroes if required.
    inc_str = str(incremental).zfill(4)
    print("incremental build #: " + inc_str)

    # create build number
    json_data['BUILD_NUMBER'] = year + month + date + inc_str

    # update version.json file
    print("Updating version.json")
    with open(file_path, 'w') as x:
        json.dump(json_data, x, indent=2)

    # Iterate through JSON vars and add them to the build environment
    # Build var template items get updated as part of build
    print("Adding version data to build environment...")
    for key, value in json_data.items():
        env.Append(CPPDEFINES=[f'{key}=\\"{value}\\"'])
        print(f'{key}="{value}"')

user_actions_post.py

import json
import sys
import os
import datetime
import re
import shutil
from SCons.Script import Import

Import("env")


# Source - https://github.com/platformio/platform-espressif32/issues/1078
# Also ...
# https://github.com/arendst/Tasmota/blob/development/pio-tools/post_esp32.py
# https://github.com/dewenni/ESP_Buderus_KM271/blob/0225e70472b4f6b9568f0901c83c3081cf0be644/platformio_release.py

APP_BIN = "$BUILD_DIR/${PROGNAME}.bin"
MERGED_BIN = "$BUILD_DIR/${PROGNAME}_boot.bin"
BOARD_CONFIG = env.BoardConfig()


def extract_release():
    config_path = env.subst("$PROJECT_DIR/ESP32/DIY-Flow-Bench/version.json")
    with open(config_path, "r") as file:
        content = file.read()
        match = re.search(r'"RELEASE":\s+"(.+)"', content)
        if match:
            return match.group(1)
        else:
            return None


def extract_build():
    config_path = env.subst("$PROJECT_DIR/ESP32/DIY-Flow-Bench/version.json")
    with open(config_path, "r") as file:
        content = file.read()
        match = re.search(r'"BUILD_NUMBER":\s+"(.+)"', content)
        print(match)
        if match:
            return match.group(1)
        else:
            return None
        
        

def extract_json_val(jsonKey):
    config_path = env.subst("$PROJECT_DIR/ESP32/DIY-Flow-Bench/version.json")
    with open(config_path, "r") as file:
        content = file.read()
        pattern = r'"' + jsonKey + '":\s+"(.+)"'
        match = re.search(pattern, content)
        # print(match)
        if match:
            return match.group(1)
        else:
            return None



# DEPRECATED 
# def delete_files_in_directory(directory_path):
#    try:
#      files = os.listdir(directory_path)
#      for file in files:
#        file_path = os.path.join(directory_path, file)
#        if os.path.isfile(file_path):
#          os.remove(file_path)
#      print("All files deleted successfully.")
#    except OSError:
#      print("Error occurred while deleting files.")


# DEPRECATED - Search for wildcard filenames  
# def del_wildcard(wildcard):
#    try:
#      print(wildcard)
#      files = os.listdir(wildcard)
#      for file in files:
#        file_path = os.path.join(wildcard, file)
#        if os.path.isfile(file_path):
#         index = file.find(wildcard)
#         if index > -1:
#            os.remove(file_path)
#      print(file_path + "deleted successfully.")
#    except OSError:
#      print("Error occurred while deleting files.")

     


def after_build(source, target, env):
    
    print("Post-Build tasks")
    print("Creating merged binary...")

    release_path = env.subst("$PROJECT_DIR/ESP32/DIY-Flow-Bench/release/")
    project_path = env.subst("$PROJECT_DIR/ESP32/DIY-Flow-Bench/")
    bootloader_path = ".pio/build/esp32dev/bootloader.bin"
    partitions_path = ".pio/build/esp32dev/partitions.bin"
    firmware_path = ".pio/build/esp32dev/firmware.bin"

    build = extract_json_val("BUILD_NUMBER")
    release = extract_json_val("RELEASE")

    merged_file = os.path.join(release_path, f"{release}_{build}_install.bin")
    update_file = os.path.join(release_path, f"{release}_{build}_update.bin")

    releases_directory =  os.path.join(project_path, f"release/")
    data_directory =  os.path.join(project_path, f"data/")

    release = extract_json_val("RELEASE")
    print(release)

    # Run esptool to merge images into a single binary
    env.Execute(
        " ".join(
            [
                '"%s"' % "$PYTHONEXE",
                "$OBJCOPY",
                "--chip",
                BOARD_CONFIG.get("build.mcu", "esp32"),
                "merge_bin",
                "--fill-flash-size",
                BOARD_CONFIG.get("upload.flash_size", "4MB"),
                "-o",
                '"%s"' % merged_file,
                "0x1000",
                bootloader_path,
                "0x8000",
                partitions_path,
                "0x10000",
                firmware_path
            ]
        )
    )

    # env.Execute(f'esptool.py --chip ESP32 merge_bin -o "%s" % {merged_file} --flash_mode dio --flash_size 4MB 0x1000 {bootloader_path} 0x8000 {partitions_path} 0x10000 {firmware_path}')

    # Create the update.bin file
    shutil.copy(".pio/build/esp32dev/firmware.bin", update_file)

env.AddPostAction(APP_BIN , after_build)
2

OK. I finally had a bit of time to do some reading and digging.

I could not find any specific connection between the monitor task and extra_scripts within the documentation. But I did find a clue…

The serial monitor executes as a task…

Executing task: platformio device monitor

I looked through the documentation to see if the monitor task called extra_scripts and found the following…

https://docs.platformio.org/en/stable/core/userguide/device/cmd_monitor.html#filters

Filters

PlatformIO allows you to apply multiple filters to the device monitor INPUT & OUTPUT streams using the pio device monitor --filter command or monitor_filters option.

As I’m using the esp32_exception_decoder which is listed as a built in filter I did a quick test by commenting it out.

The behaviour stops. I am able to open the serial monitor without the extra_scripts being run.

So the question is now whether this is intended behaviour (I suspect not) and how I can inhibit the behaviour or code around it.

Any help on this very much appreciated.

T.I.A.

/DM

3

So I have a working hack, but it’s not nice.

INO_FILE = ".pio/build/esp32dev/src/API.cpp.o"
def before_build(source, target, env):

    ...user_actions_pre code

    env.AddPreAction(INO_FILE, before_build)

It’s a bit of a relay race as it’s triggered by the first file in the schedule.

However I cannot find a definitive hook that is called before the build.

env.AddPreAction("buildprog", callback...)

…is only triggered after ALL files are built and so is a bit of an oxymoron

Anyone have this working properly?

/DM

4

Any devs able to comment?

Pre Task Hooks

Are there any specific hooks that will allow task to run BEFORE the compiler?

I do not see any listed in the documentation. Also read through sCon documentation and cannot see anything usable there either.

Seems like a pretty trivial thing to have a pre_task hook that is triggered prior to the compiler execution. Even the pre/prior language used - ‘env.AddPreAction’ implies this is default behaviour.

i.e. it runs BEFORE the hook.

env.AddPreAction(PRE_TASK_HOOK, before_build)

The hack I implemented above is pretty nasty as it is essentially a race between the compiler and the pre:task - the pre:task needs to to complete and send the new version data to the environment before the compiler gets to the file where the version data is required else the compiler crashes.

Surely I am missing something here. There cannot be a ‘PreAction’ without any way to reliably trigger it?

Monitor Task

I did not expect extra_scripts to be triggered by the monitor. It is not intuitive behaviour. I can understand that the monitor environment would need to access the build configuration, but not that it is treated in the same way that the compiler is with regards to running extra_scripts. This does not make immediate sense.

With hindsight I suspect that this is desired behaviour and an error on my part but neither the documentation, nor the environment itself indicates this.

If I run…

pio run --list-targets

monitor is not included as a target

but there is a hint in the documentation…

–list-targets

List available project targets. It’s also possible to list targets per project environment using pio run --environment option.

There are also built-in system targets:

  • Device

Some clarification on how the monitor task is treated would be good as the documentation is very ambiguous.

Reading through the documentation the only clue is the aforementioned environment variable for monitor tasks

monitor_filters = esp32_exception_decoder

This is the only clue that the monitor task accesses the build environment. Even then in the documentation it is referred to as a filter and implies that each filter has it’s own dedicated custom filter file.

As such the implication is that as it can access it’s own dedicated custom task it would not need to access extra_scripts. As surely one of those processes then becomes redundant

This is of course an assumption on my part as a result of not understanding the relationship, but that is only as a direct result of not being able to find the information in the documentation.

Of course, everything I have written above is also an assumption.

Can anyone clarify the relationship?

T.I.A

/DM

5

OK tying the pre hook to a file is folly.

Works okay if I clean before every build, but that is just wasting a whole bunch of time, which is super frustrating.

@maxgerhardt I note that in your solution for the following thread - How do I run a script before compiling a project? - #6 by maxgerhardt

Ah okay. But if the file release_page.h is only needed in webserver.c, then why not hook its pre-action with env.AddPreAction("${BUILD_DIR}/esp-idf/main/webserver.c.o", before_build)? Building shouldn’t continue then until the script function has finished executing.

It appears to imply that the hook is tied to the file mentioned.

However the file is called multiple times in the project so this solution does not work for me as the multithreaded nature means that one of the other files continues compilation and then crashes out.

The suggestion of changing to a singlethread again is not a viable solution as…

Any solution that creates a different problem is not a solution

What am I missing.

What hook can I used to achieve my goal? (bearing in mind that this is still trying to hack preventing the serial monitor from running extra_scripts).

Would really appreciate some input here.

Kinda feel like I’m pissing into the wind

/DM

6

Managed to get this working with a suggestion by @ivankravets in the following post

The solution is to use the ‘IsIntegrationDump’ method to test whether the script is being called from an external application.

# Do not run a script when external applications, such as IDEs,
# dump integration data. Otherwise, input() will block the process
# waiting for the user input
if env.IsIntegrationDump():
    # stop the current script execution
    Return()

The IsIntegrationDump() test was also in the solution provided by @maxgerhardt above, but did not work for me.

So in my case I could not find a generic / system hook that could be used to trigger the AddPreAction() method. So it was impossible to fire it at the correct time.

The solution was to call the pre_action_script in platformio as normal

extra_scripts = 
	pre:user_actions_pre.py

But then NOT to use the preAction method within user_actions_pre.py to trigger your code, but to just place your code directly in the file…

# stop the current script execution if called from external script
if env.IsIntegrationDump():
    Return()

print("Pre-Build tasks")
print("Loading version.json")

release_path = env.subst("$PROJECT_DIR/ESP32/DIY-Flow-Bench/release/")

# read json file into var

...etc

This way your code will run as soon as the file gets called, which is pretty much directly after the dependencies are loaded.

The test for IsIntegrationDump() prevents code execution should the file get called in other circumstances, which for me was when the esp32_exception_decoder was loaded. Others also note that their pre:user_scripts also get called by other non-system tasks as well.

So my revised and complete code is now this…

import json
import sys
import os
import datetime
import re
import shutil
from SCons.Script import Import

Import("env")


# stop the current script execution if called from external script
if env.IsIntegrationDump():
    Return()

print("Pre-Build tasks")
print("Loading version.json")

release_path = env.subst("$PROJECT_DIR/ESP32/DIY-Flow-Bench/release/")

# read json file into var
file_path = 'ESP32/DIY-Flow-Bench/version.json'
with open(file_path) as file_data:
    json_data = json.load(file_data)

# increment build and update version.json

# get current date
dtnow = datetime.datetime.now()
year = dtnow.strftime("%y")
month = dtnow.strftime("%m")
date = dtnow.strftime("%d")

# read build No from json
build_num = json_data['BUILD_NUMBER']
release = json_data['RELEASE']
# print(build_num  + "\n")

# get current details and delete files
old_merged_file = os.path.join(release_path, f"{release}_{build_num}_install.bin")
old_update_file = os.path.join(release_path, f"{release}_{build_num}_update.bin")
try:
    os.remove(old_merged_file)
    os.remove(old_update_file)
except OSError:
    print("Error occurred while deleting files.")

# print(old_merged_file)

# get build date info from version.json
bn_year = build_num[0:2]
bn_month =  build_num[2:4]
bn_date =  build_num[4:6]
bn_inc = build_num[6:10]
# print(bn_year  + "\n")
# print(bn_month  + "\n")
# print(bn_date  + "\n")
# print(bn_inc  + "\n")

# check build date incremental count
if bn_year == year and bn_month == month and bn_date == date:
    # we are still on same day lets increment existing build number
    incremental = int(build_num[6:10])
    incremental += 1    
else:
    # it's a new day start from 0001
    incremental = 1

## add preceding zeroes if required.
inc_str = str(incremental).zfill(4)
print("incremental build #: " + inc_str)

# create build number
json_data['BUILD_NUMBER'] = year + month + date + inc_str

# update version.json file
print("Updating version.json")
with open(file_path, 'w') as x:
    json.dump(json_data, x, indent=2)

# Iterate through JSON vars and add them to the build environment
# Build var template items get updated as part of build
print("Adding version data to build environment...")
for key, value in json_data.items():
    env.Append(CPPDEFINES=[f'{key}=\\"{value}\\"'])
    print(f'{key}="{value}"')

I really hope that this helps someone else out as I spent a few days scratching my head over this one.

I also note that the platformio documentation has been updated to reflect this.

/DM

7

Marlin also uses this to determine if this is an actual build run

8

Thanks Max. I tried this logic based on your reply in the thread linked previously, but for some reason it did not fire. I am unsure why.

The only thing that I can think of is that perhaps the indentation was incorrect.

Python is very new to me and I am unused to languages like Python and YAML where indentation forms part of the syntax.