Hallo,
I am looking for the source filename in the PlatformIO environment variables.
I tried: pio run —target envdump, but the name of the main source file is not included.
Is there a way to get it as a var and use it in a script?
What if there are multiple files in src/? What’s the end goal here?
No, I use only one file in the src dir, but within that filename is an version nummer embedded, which I use for version control of the code.
Within the compiled code I use the __name__ macro so I can see in the webserver of the ESP which version of the code is running.
When I am updating my code, I copy the current file to a new dir and update version number of the file, perform my modifications, update platform.ini and off I go with the new code.
Now I am testing a http update server for deep sleep unattended ESP’s and want to make a solution when I update the code, I publish this new version of the code to my internal webserver. When the ESP awakes a firmware check will be performed and when a newer firmware is detected the ESP will update. Because I use a common code base for several ESP’s with different hostnames and sensor configs, I want to make a different firmware file for each ESP. The filename should contain the hostname, codefilename and timestamp. I succeeded to get this mechanism going with hostname and timestamp, but I am missing the source code filename.
Looking at ŧhe docs and inspecting print(env.Dump()) one can see that the source directory is in PROJECT_SRC_DIR.
When I do
[env:esp32dev]
platform = espressif32
board = esp32dev
framework = arduino
extra_scripts = pre:modify_firmware_name.py
with modify_firmware_name.py as
from os import listdir
from os.path import isfile, join
Import("env")
# grab source directory from environment variable
src_dir = env["PROJECT_SRC_DIR"]
# grab list of all files in that folder
onlyfiles = [f for f in listdir(src_dir) if isfile(join(src_dir, f))]
# grab first file (ignore all others). assume len != 0
main_file = onlyfiles[0]
# remove cpp extension
main_file = main_file.replace(".cpp", "")
# change firmware name
env.Replace(PROGNAME="firmware_%s" % main_file)
and a src/my_main1.cpp file in the project, I get upon building…
Linking .pio/build/esp32dev/firmware_my_main1.elf
Retrieving maximum program size .pio/build/esp32dev/firmware_my_main1.elf
Checking size .pio/build/esp32dev/firmware_my_main1.elf
Advanced Memory Usage is available via "PlatformIO Home > Project Inspect"
RAM: [ ] 4.0% (used 13224 bytes from 327680 bytes)
Flash: [== ] 15.4% (used 201208 bytes from 1310720 bytes)
Building .pio/build/esp32dev/firmware_my_main1.bin
esptool.py v3.0
============== [SUCCESS] Took 9.35 seconds ==============
Tested your solution. Works as a charm. Just what I was looking for. Thnks.
I combined this script with the example script for combining the hostname and a timestamp to get a unique firmware file.
Other question: The ${UNIX_TIME} variable does produce a not user friendly timestamp.
Is there a way to produce a more user friendly time stamp in a ddmmyy_hh:mm:ss format in a way I can use it in the python script but also a a MACRO using in the source code?
In C/C++, you can use __DATE__ and __TIME__ macros. See docs. If you only want to use the unix timestamp, you will have to convert it using standard C library functions from time.h, specifically localtime() and strftime(), as seen e.g. here. In Python, just use the standard library function, as documtented.
>python
>>> import datetime
>>> datetime.datetime.now()
datetime.datetime(2021, 6, 8, 22, 17, 6, 594155)
>>> datetime.datetime.now().isoformat()
'2021-06-08T22:17:36.401890'
>>> datetime.datetime.now().ctime()
'Tue Jun 8 22:17:44 2021'
Ok. I have to convert the unix_time in the python script and in my sketch to have a user friendly time format. I will implement this. Thanks again.
I am not a python programmer, and I am stuck when I try to get the ${UNIX_TIME} through the buildflag FIRMWARE_TIME to use it as a variable in the python script.
Partial platform ini:
[ESP8266_Testmodule]
NAME = ESP---------------TEST
firmware_time = ${UNIX_TIME}
[env:ESP8266_Testmodule]
build_flags=
-D HOSTNAME=${ESP8266_Testmodule.NAME}
-D FIRMWARE_TIME=${ESP8266_Testmodule.firmware_time}
This works:
Partial modify_firmware_name.py
env.Replace(PROGNAME="firmware_%s_%s_%s" %(defines.get("HOSTNAME"), main_file, defines.get("FIRMWARE_TIME")))
This throws an error:
dt = datetime.datetime(defines.get("FIRMWARE_TIME"))
#dt = datetime.datetime.now()
test = defines.get("FIRMWARE_TIME")
print(test)
env.Replace(PROGNAME="firmware_%s_%s_%s" %(defines.get("HOSTNAME"), main_file, dt.strftime('%d%m%Y_%H%M%S')))
print(test) shows ${UNIX_TIME} and not the content of ${UNIX_TIME}
with dt = datetime.datetime.now() it works, but I don’t want take the risk that due compilation the sketch macro is a second different to the filename.
The error:
TypeError: an integer is required (got type str):
File "C:\Users\edaut\.platformio\penv\lib\site-packages\platformio\builder\main.py", line 175:
env.SConscript(item, exports="env")
File "C:\Users\edaut\.platformio\packages\tool-scons\scons-local-4.1.0\SCons\Script\SConscript.py", line 591:
return _SConscript(self.fs, *files, **subst_kw)
File "C:\Users\edaut\.platformio\packages\tool-scons\scons-local-4.1.0\SCons\Script\SConscript.py", line 280:
exec(compile(scriptdata, scriptname, 'exec'), call_stack[-1].globals)
File "C:\Users\edaut\OneDrive\Desktop\Raspberry\Arduino\PlatformIO\Domotica_devices_test_sw\modify_firmware_name.py", line 31:
dt = datetime.datetime(defines.get("FIRMWARE_TIME"))
I assume also that the error is referring to the use of ${UNIX_TIME} instead of the content of this variabele.
Can you help me out?
The core generates the UNIX_TIME environment variable here
so just use
# will be of type int
unix_time = env["UNIX_TIME"]
to retrieve the value in Python script. That will thus be the same as the same the value of the UNIX_TIME macro passed to the code.
You should be using the conversion function datetime.datetime.fromtimestamp as per python - Converting unix timestamp string to readable date - Stack Overflow convert from the unix timestamp to a datetime object.
So when I e.g. write
from os import listdir
from os.path import isfile, join
import datetime
Import("env")
# grab source directory from environment variable
src_dir = env["PROJECT_SRC_DIR"]
# grab list of all files in that folder
onlyfiles = [f for f in listdir(src_dir) if isfile(join(src_dir, f))]
# grab first file (ignore all others). assume len != 0
main_file = onlyfiles[0]
# remove cpp extension
main_file = main_file.replace(".cpp", "")
# get time and convert
unix_time = env["UNIX_TIME"]
dt = datetime.datetime.fromtimestamp(unix_time)
date_str = dt.strftime('%d%m%Y_%H%M%S')
# change firmware name
env.Replace(PROGNAME="firmware_%s_%s" % (main_file, date_str))
I get
Building .pio\build\esp32dev\firmware_main_09062021_235609.bin
Though note that there is some ambiguity in this time string if you want to be very precise: It does not specify a timezone, but uses the local timezone of the builder. So one cannot convert back to the unix timestamp value without knowing the timezone. You might want to change that to UTC+0 if that is a concern, see e.g. here and the related timezone info.
Very good. I am going to test this solution this weekend. Thank you for your fantastic and swift help.
Works as a charm. Thank you very much for your support.
Problem:
I tested the script modify_firmware_name.py with my test configuration.
This worked perfecty fine.
However in my production configuration with a lot of build flags, it throw an error:
ValueError: too many values to unpack (expected 2):
File “C:\Users\edaut.platformio\penv\lib\site-packages\platformio\builder\main.py”, line 175:
env.SConscript(item, exports=“env”)
File “C:\Users\edaut.platformio\packages\tool-scons\scons-local-4.1.0\SCons\Script\SConscript.py”, line 591:
return _SConscript(self.fs, *files, **subst_kw)
File “C:\Users\edaut.platformio\packages\tool-scons\scons-local-4.1.0\SCons\Script\SConscript.py”, line 280:
exec(compile(scriptdata, scriptname, ‘exec’), call_stack[-1].globals)
File “C:\Users\edaut\OneDrive\Desktop\Raspberry\Arduino\PlatformIO\Domotica_devices\modify_firmware_name.py”, line 26:
defines = {k: v for (k,v) in my_flags.get(“CPPDEFINES”)}
File “C:\Users\edaut\OneDrive\Desktop\Raspberry\Arduino\PlatformIO\Domotica_devices\modify_firmware_name.py”, line 26:
defines = {k: v for (k,v) in my_flags.get(“CPPDEFINES”)}
This error is thrown when using more then 25 build flags. Without this script the code compiles fine with more than 25 build fags.
The Python error says that one elementin my_flags.get("CPPDEFINES") was not unpackagable as a (k,v) pair but had more than 2 elements in it. See e.g. here. What does print(my_flags.get("CPPDEFINES")) show exactly?
Through your suggestion, I was able to solve this problem. I use some buildflags without a value, like
-D DEEP_SLEEP
-D SYSLOG
-D SERIAL_DEBUG
-D REMOTE_DEBUG
-D DHT22_SENSOR
The problem is solved when I give those flags a value
-D DEEP_SLEEP=1
-D SYSLOG =1
-D SERIAL_DEBUG=1
-D REMOTE_DEBUG = 1
-D DHT22_SENSOR=1
Is this a bug in PlatformIO? Or has the script to be altered to cope with this kind of buildfags?
No, the Python bug is genuine. When there are flags which have no value, but are just ‘defined’ with name, say -D DEEP_SLEEP, then those values are not decodable as key,value pair.
This is however poorly written in the documentation, in which the example
my_flags = env.ParseFlags(env['BUILD_FLAGS'])
defines = {k: v for (k, v) in my_flags.get("CPPDEFINES")}
is displayed. In this example it really works because all flags happen to be of key-value form.
I’ve also noticed that a year ago in Change name of firmware file - #10 by maxgerhardt where I wrote a better version which first checks whether the instance is a list (aka key,value) or just a singular value / name. An issue in GitHub - platformio/platformio-docs: PlatformIO Documentation should be filed to improve the documentation in these cases
Modified the script like your example. Works fine now with the orginal build flags. Thanxs.
Should I file an issue in Github, or do you file this?