PlatformIO Community

Trouble with Custom Firmware Naming

I have referenced the Custom firmware/program name section of the Advanced Scripting doc. I’ve got a start on things, but some of it is not clicking. I’m trying to concatenate the firmware name from:

  • Framework
  • Board
  • Pre-processor flag
  • Git tag

So far I am using this platformio.ini:

[env:uno]
platform = atmelavr
board = uno
framework = arduino
build_flags = !python git_rev.py
extra_scripts = pre:name_firmware.py

My git_rev.py is as follows:

import subprocess
tagcmd = "git describe --tags --abbrev=0"
version = subprocess.check_output(tagcmd).decode().strip()
revcmd = "git log --pretty=format:'%h' -n 1"
commit = subprocess.check_output(revcmd).decode().strip()
print "-DPIO_SRC_TAG=%s" % version
print "-DPIO_SRC_REV=%s" % commit

And my name_firmware.py is as follows (so far):

import subprocess
Import("env")
my_flags = env.ParseFlags(env['BUILD_FLAGS'])
defines = {k: v for (k, v) in my_flags.get("CPPDEFINES")}
env.Replace(PROGNAME="myfile-%s-%s-%s-%s" % (
    defines.get("PIOFRAMEWORK"),
    defines.get("PIOENV"),
    "PreProcDef",
    defines.get("PIO_SRC_TAG")))

So far, the compiled file is named myfile-None-None-PreProcDef-1.2.3.hex

As you can see I get the PIO_SRC_TAG from my Git tag, but I am not getting the framework (expecting “arduino”) and I am not getting the board (expecting “uno”). Hopefully someone can help me with those two?

Also, is there any way to get a pre-processor definition? Where I currently have “PreProcDef” I’d like to get a #define out of one of my config.h file. For instance I have a define like this:

#define STATIC_CONFIG SHIELD_TWI

I’d like to just get the “TWI” portion if possible, but I can always do a case statement if I have to get the whole SHIELD_TWI part… Hopefully there’s a snazzy way to do that without grep/sed hell.

Desired end result would be: myfile-arduino-uno-twi-1.2.3.hex

Banging my head on this but I’m pretty sure it’s possible. Any assistance would be highly appreciated!

Thank you for reading.

Stored in env not the defines.

Import("env")

framework = env["PIOFRAMEWORK"][0]
env_name = str(env["PIOENV"])

print("Framework: %s Environment: %s" % (framework, env_name))

Gives for an mbed-nucleol476rg project

Framework: mbed Environment: nucleo_l476rg

Evaluating the #defines from a python script might not be the best way to do it. It might make sense to move this into the build_flags and then makek different environments based on the configurations. But If you don’t, parsing the .h file might be an option, though an unclean one. Another option would be to invoke gcc with -E on the file, see here, and parse its output.

1 Like

Thank you so much Max! I’m completely new to PlatformIO so while the answer makes absolute sense now, it eluded me when I was staring at it previously.

I understand what you mean about the #defines but I had to hope. Moving that to build_flags would be a pretty large change in the way things are constructed now, but it’s an option.

Thank you again for your help.

Best way to start is by doing print(env.Dump()) and looking at what the environment has to offer. See http://docs.platformio.org/en/latest/projectconf/advanced_scripting.html#construction-environments

Either the macros in by including your configuration header file or the compiler injects it via a -D MACRO=XY flag, but in the end you still have the same macro and can do all the things on it, like #ifdef.. etc.

I ended up getting the last bit I wanted via python:

shields = []
linenum = 0
define = "#define STATIC_CONFIG".lower()
with open (str(env["PROJECTINCLUDE_DIR"]) + '/Config.h', 'rt') as config:
    for line in config:
        linenum += 1
        if line.lower().startswith(define):
            shields.append(line.rstrip('\n'))
for variant in shields:
    shield = variant.split(" ")[-1].split("_")[-1].lower()

Not glamorous, but it works.

Thanks again!

1 Like

Hello!
I would also like to do custom firmware name based on name defined in a config.h file. I have searched the internet and tried to read the documentation about Custom firmware/program name, but no luck with getting it up and running. Don’t have much experience in python and platformio’s custom scripts.
Would you mind posting the full python code and other relevant code so that I (and probably others) could learn and replicate from it?
Thank you in advance!

Sure, and welcome to the community!

As @maxgerhardt said below, getting something from a header file is sub-optimal. I ended up getting the name from the Git environment. You need to include the following in your platformio.ini file:

build_flags = !python tools\git_rev.py
extra_scripts = pre:python tools\name_firmware.py

This assumes you will create a .\tools directory in your project of course. Create two files in your ./tools directory.

tools\git_rev.py:

import subprocess

# Get Git project name
projcmd = "git rev-parse --show-toplevel"
project = subprocess.check_output(projcmd, shell=True).decode().strip()
project = project.split("/")
project = project[len(project)-1]

# Get 0.0.0 version from latest Git tag
tagcmd = "git describe --tags --abbrev=0"
version = subprocess.check_output(tagcmd, shell=True).decode().strip()

# Get latest commit short from Git
revcmd = "git log --pretty=format:'%h' -n 1"
commit = subprocess.check_output(revcmd, shell=True).decode().strip()

# Get branch name from Git
branchcmd = "git rev-parse --abbrev-ref HEAD"
branch = subprocess.check_output(branchcmd, shell=True).decode().strip()

# Make all available for use in the macros
print("-DPIO_SRC_NAM={0}".format(project))
print("-DPIO_SRC_TAG={0}".format(version))
print("-DPIO_SRC_REV={0}".format(commit))
print("-DPIO_SRC_BRH={0}".format(branch))

This script will run before your code compiles and create defines in your environment for:

  • PIO_SRC_NAM - The name of your project according to Git
  • PIO_SRC_TAG - Your current Git tag
  • PIO_SRC_REV - Your current commit has (short)
  • PIO_SRC_BRH - Yout current Git branch

(Incidentally, these can be used in your code, that’s extra credit I’ll leave to you.)

And tools\name_firmware.py:

import subprocess
Import("env")

my_flags = env.ParseFlags(env['BUILD_FLAGS'])
defines = {k: v for (k, v) in my_flags.get("CPPDEFINES")}

env.Replace(PROGNAME="%s-%s-%s-%s-%s" % (
    defines.get("PIO_SRC_NAM"),
    str(env["BOARD"]),
    defines.get("PIO_SRC_TAG"),
    defines.get("PIO_SRC_REV"),
    defines.get("PIO_SRC_BRH")))

This will give you a firmware name similar to: myproject-d1_mini-1.0.0-5ba5cd3-master.bin where myproject is from PIO_SRC_NAM, d1_mini is from the board definition in your platformio.ini file, 1.0.01 is from PIO_SRC_TAG, 5ba5cd3 is from PIO_SRC_REV, and master is from PIO_SRC_BRH.

From that description, you should be able to handle darned near everything you’d want to do.

If you really want to get something from a header file, this is a sample of the python in the tools\name_firmware.py`:

# Parse out defined shield from Config.h
shields = []
linenum = 0
define = "#define BREWPI_STATIC_CONFIG".lower()
with open (str(env["PROJECTSRC_DIR"]) + '\Config.h', 'rt') as config:
    for line in config:
        linenum += 1
        if line.lower().startswith(define):
            shields.append(line.rstrip('\n'))
for variant in shields:
    shield = variant.split(" ")[-1].split("_")[-1].lower()

In this case shield will contain the define argument from a line like this in src/Config.h:

#define BREWPI_STATIC_CONFIG BREWPI_SHIELD_I2C

The shield = variant.split(" ")[-1].split("_")[-1].lower() logic splits out by the underscores and gives me i2c as the value of shield.

It would be easier to not get the firmware name from the config.h but instead set it via a build_flag in the platformio.ini (whicht the config.h can then also include or use directly). Otherwise you would have to run the C pre-processor on your header file or somehow extract the macro value from it, which is painful… If you do it via a build_flag = -D .. you’re back to the this minimal example shown in the documentation. If you have problems with that, then we would need to see the full project.

Thank you both for such a fast response! What a great community!
Special thanks to @LBussy for making the examples with comments and in full detail! Really appreciate it! I will bookmark this for future reference as it seems very inspiring.
I didn’t specify earlier that currently I would only need a very simple approach, because I am developing locally and not using git (not yet, but slowly learning my way). Just developing for some Home Automation Wemos boards with different settings and same code-base.
I managed to get @maxgerhardt suggestion working for file name automation and now trying to figure out how to get the definition made in platformio.ini to include or use directly in .h files. I guess I should start tweaking the example from Override package files ?

You would only use that when you want to override and patch some package, that is e.g. the framework-arduinoavr package. I don’t think you need that.

If in the platformio.ini you specify build_flags = -D MY_VERSION=123 then every source file (.h, .cpp) sees this global macro definition, so you can use it directly in the code as if you would have written #define MY_VERSION 123 at the top of the file. See documentation. Does that make it clearer?

And btw, yes it’s possible that you hardcode the version name in a config.h file, to extract it in python you can e.g. read-out out the file and throw a regular expression at it to find the value, if you must keep the hardcoded value in the config.h instead of a -D build flag.

1 Like

oh wow! I totally missed that! This makes things so much easier!
Yes, everything is much clearer now.
Thank you both for the time you took for explaining! I got all the answers I needed :slight_smile:

Hey ya’ll - love this thread. Used the documentation and this example to get some nice custom and automated firmware names. However, i have one glaring issue that i can not seem to solve.
If i use this format of build flags:

build_flags = 
    -DSOME_VARIABLE= 123

Everything works great.
However, if i have to define a preprocessor flag for a library - like this

build_flags = 
	-DASYNCWEBSERVER_REGEX

I get a ValueError: too many values to unpack (expected 2):

My customname.py file looks like this (very basic):

Import("env")

my_flags = env.ParseFlags(env['BUILD_FLAGS'])
defines = {k: v for (k, v) in my_flags.get("CPPDEFINES")}
# defines.get("PIO_SRC_TAG") - tag name
env.Replace(
    PROGNAME="%s-%s-%s-%s-%s" %
    (defines.get("PIO_SRC_NAM"), defines.get("VERSION"), str(env["BOARD"]),
     defines.get("PIO_SRC_REV"), defines.get("PIO_SRC_BRH")))

You cannot do that since an entry in CPPDEFINES might be a single value, not a tuple.

Why would you delete this?

yeah - that makes sense. How can i set it up to function with support for my current environment?

Just make your python code expect that the elements aren’t always a tuple:

Import("env")

my_flags = env.ParseFlags(env['BUILD_FLAGS'])
defines = dict()
for x in my_flags.get("CPPDEFINES"):
    if type(x) is tuple:
        (k,v) = x
        defines[k] = v
    else:
        defines[x] = "" # empty value
# defines.get("PIO_SRC_TAG") - tag name
env.Replace(
    PROGNAME="%s-%s-%s-%s-%s" %
    (defines.get("PIO_SRC_NAM"), defines.get("VERSION"), str(env["BOARD"]),
     defines.get("PIO_SRC_REV"), defines.get("PIO_SRC_BRH")))
1 Like

yeah - i haven’t done much python programming, so wasn’t too sure how to approach that. The logic makes sense, though - reading this. So the previous code was just a key-value for loop and the single-value preprocessor flags were causing issue since they were not tuples.

So, in the logic that you presented, first you check if the value is a tuple and parse it accordingly. Alight. Thank you very much.

EDIT: will take your suggestion and move forward, but your exact logic causes this error:
TypeError: unhashable type: 'list': thank you for clarifying the issue though :slight_smile:
I just cast the x (dictionary) to a tuple() and it seems to be working.

Nvm - all the errors are gone, but now i get None-None-esp32cam-None-None.elf as the name - where before (when i commented out of the single-value flags) i was getting the actual name.

These PIO_SRC_NAM, VERSION, PIO_SRC_REV and PIO_SRC_BRH defines don’t create themselves – in the examples above they were created by using the

code. You also have that in your project or no?

Absolutely - i do. It was working before i made the most recent changes.