Automating Compilation of Multiple Binaries from CSV Configurations: Seeking Advice and Tools

1

"Hello everyone,

I’m currently working on a project where I need to compile multiple binaries based on configurations listed in a CSV file. Each line in the CSV file represents a different configuration, and I need to compile a binary for each configuration. I tried using advanced scripting but i managed only to compile for one line.
Has anyone else encountered a similar situation? If so, how did you approach it? Are there any specific tools or scripts that could help streamline this task? Any advice or suggestions would be greatly appreciated!

Thank you in advance for your help!"

Python Script:

import csv


Import ("projenv")


csv_file = "settings.csv"

with open(csv_file, newline='') as csvfile:
    reader = csv.DictReader(csvfile)
    for row in reader:
        projenv.Append(CPPDEFINES=[
            ("AWS_IOT_PUBLISH_TOPIC", projenv.StringifyMacro(row["AWS_IOT_PUBLISH_TOPIC"])),
            ("AWS_IOT_SUBSCRIBE_TOPIC", projenv.StringifyMacro(row["AWS_IOT_SUBSCRIBE_TOPIC"])),
            ("SECRETS_FILE", projenv.StringifyMacro(row["SECRETS_FILE"]))])

platformio config file:

[env]
platform = espressif32
framework = arduino
lib_deps = 
    bblanchon/ArduinoJson @ ^7.0.4

    knolleary/pubsubclient @ ^2.8
monitor_speed = 115200
board = lolin32_lite





[env:production]
build_type = release
board = lolin32_lite
extra_scripts = scripts/buildAll.py
build_flags =
    '-D EAP_IDENTITY = "*********"'
    '-D EAP_PASSWORD = "*********"'
    '-D WIFI_SSID = "*********"'
    '-D EAP_ANONYMOUS_IDENTITY = "*********"'




[env:development]
build_type = release
board = lolin32_lite
extra_scripts = scripts/buildAll.py
build_flags =
    '-D DEVELOPMENT'
    '-D SerialDebugOn'
    '-D WIFI_SSID = "*********"'
    '-D WIFI_DEVPASS = "*********"'
2

I would have chosen a different approach:

Reference envrionment variables inside the macros like

build_flags =
 '-DAWS_IOT_PUBLISH_TOPIC=${sysenv.AWS_IOT_PUBLISH_TOPIC}'
 ...

(per docs), then write a regular python script that reads each row of the CSV and calls into pio run -e production with the environment variables set to the value from the read row (look into the subprocess python module).

I’ve not yet seen an extra_script being able to just start multiple firmware build operations with different settings. It can only affect the current one.

3

Thanks i have created a scrşot but i’m getting strange errors

python .\scripts\realbuildAll.py
Traceback (most recent call last):
  File "<frozen runpy>", line 198, in _run_module_as_main
  File "<frozen runpy>", line 88, in _run_code
  File "C:\Users\ruzgarerik\.platformio\penv\Scripts\platformio.exe\__main__.py", line 4, in <module>
  File "C:\Users\ruzgarerik\.platformio\penv\Lib\site-packages\platformio\__main__.py", line 21, in <module>   
    from platformio import __version__, exception, maintenance
  File "C:\Users\ruzgarerik\.platformio\penv\Lib\site-packages\platformio\maintenance.py", line 22, in <module>
    from platformio import __version__, app, exception, fs, telemetry
  File "C:\Users\ruzgarerik\.platformio\penv\Lib\site-packages\platformio\app.py", line 24, in <module>        
    from platformio import __version__, exception, fs, proc
  File "C:\Users\ruzgarerik\.platformio\penv\Lib\site-packages\platformio\fs.py", line 28, in <module>
    from platformio.compat import IS_WINDOWS
  File "C:\Users\ruzgarerik\.platformio\penv\Lib\site-packages\platformio\compat.py", line 27, in <module>
    from asyncio import create_task as aio_create_task
  File "C:\Users\ruzgarerik\.platformio\python3\Lib\asyncio\__init__.py", line 42, in <module>
    from .windows_events import *
  File "C:\Users\ruzgarerik\.platformio\python3\Lib\asyncio\windows_events.py", line 8, in <module>
    import _overlapped
OSError: [WinError 10106] The requested service provider could not be loaded or initialized
Traceback (most recent call last):
  File "<frozen runpy>", line 198, in _run_module_as_main
  File "<frozen runpy>", line 88, in _run_code
  File "C:\Users\ruzgarerik\.platformio\penv\Scripts\platformio.exe\__main__.py", line 4, in <module>
  File "C:\Users\ruzgarerik\.platformio\penv\Lib\site-packages\platformio\__main__.py", line 21, in <module>
    from platformio import __version__, exception, maintenance
  File "C:\Users\ruzgarerik\.platformio\penv\Lib\site-packages\platformio\maintenance.py", line 22, in <module>
    from platformio import __version__, app, exception, fs, telemetry
  File "C:\Users\ruzgarerik\.platformio\penv\Lib\site-packages\platformio\app.py", line 24, in <module>
    from platformio import __version__, exception, fs, proc
  File "C:\Users\ruzgarerik\.platformio\penv\Lib\site-packages\platformio\fs.py", line 28, in <module>
    from platformio.compat import IS_WINDOWS
  File "C:\Users\ruzgarerik\.platformio\penv\Lib\site-packages\platformio\compat.py", line 27, in <module>
    from asyncio import create_task as aio_create_task
  File "C:\Users\ruzgarerik\.platformio\python3\Lib\asyncio\__init__.py", line 42, in <module>
    from .windows_events import *
  File "C:\Users\ruzgarerik\.platformio\python3\Lib\asyncio\windows_events.py", line 8, in <module>
    import _overlapped
OSError: [WinError 10106] The requested service provider could not be loaded or initialized
Traceback (most recent call last):
  File "<frozen runpy>", line 198, in _run_module_as_main
  File "<frozen runpy>", line 88, in _run_code
  File "C:\Users\ruzgarerik\.platformio\penv\Scripts\platformio.exe\__main__.py", line 4, in <module>
  File "C:\Users\ruzgarerik\.platformio\penv\Lib\site-packages\platformio\__main__.py", line 21, in <module>
    from platformio import __version__, exception, maintenance
  File "C:\Users\ruzgarerik\.platformio\penv\Lib\site-packages\platformio\maintenance.py", line 22, in <module>
    from platformio import __version__, app, exception, fs, telemetry
  File "C:\Users\ruzgarerik\.platformio\penv\Lib\site-packages\platformio\app.py", line 24, in <module>
    from platformio import __version__, exception, fs, proc
  File "C:\Users\ruzgarerik\.platformio\penv\Lib\site-packages\platformio\fs.py", line 28, in <module>
    from platformio.compat import IS_WINDOWS
  File "C:\Users\ruzgarerik\.platformio\penv\Lib\site-packages\platformio\compat.py", line 27, in <module>
    from asyncio import create_task as aio_create_task
  File "C:\Users\ruzgarerik\.platformio\python3\Lib\asyncio\__init__.py", line 42, in <module>
    from .windows_events import *
  File "C:\Users\ruzgarerik\.platformio\python3\Lib\asyncio\windows_events.py", line 8, in <module>
    import _overlapped
OSError: [WinError 10106] The requested service provider could not be loaded or initialized
Traceback (most recent call last):
  File "<frozen runpy>", line 198, in _run_module_as_main
  File "<frozen runpy>", line 88, in _run_code
  File "C:\Users\ruzgarerik\.platformio\penv\Scripts\platformio.exe\__main__.py", line 4, in <module>
  File "C:\Users\ruzgarerik\.platformio\penv\Lib\site-packages\platformio\__main__.py", line 21, in <module>
    from platformio import __version__, exception, maintenance
  File "C:\Users\ruzgarerik\.platformio\penv\Lib\site-packages\platformio\maintenance.py", line 22, in <module>
    from platformio import __version__, app, exception, fs, telemetry
  File "C:\Users\ruzgarerik\.platformio\penv\Lib\site-packages\platformio\app.py", line 24, in <module>
    from platformio import __version__, exception, fs, proc
  File "C:\Users\ruzgarerik\.platformio\penv\Lib\site-packages\platformio\fs.py", line 28, in <module>
    from platformio.compat import IS_WINDOWS
PS D:\Dosyalar\Projeler\Aqualley\Embedded\aqualleyfw> python .\scripts\realbuildAll.py
Traceback (most recent call last):
  File "<frozen runpy>", line 198, in _run_module_as_main
  File "<frozen runpy>", line 88, in _run_code
  File "C:\Users\ruzgarerik\.platformio\penv\Scripts\platformio.exe\__main__.py", line 4, in <module>
  File "C:\Users\ruzgarerik\.platformio\penv\Lib\site-packages\platformio\__main__.py", line 21, in <module>   
    from platformio import __version__, exception, maintenance
  File "C:\Users\ruzgarerik\.platformio\penv\Lib\site-packages\platformio\maintenance.py", line 22, in <module>
    from platformio import __version__, app, exception, fs, telemetry
  File "C:\Users\ruzgarerik\.platformio\penv\Lib\site-packages\platformio\app.py", line 24, in <module>
    from platformio import __version__, exception, fs, proc
  File "C:\Users\ruzgarerik\.platformio\penv\Lib\site-packages\platformio\fs.py", line 28, in <module>
    from platformio.compat import IS_WINDOWS
  File "C:\Users\ruzgarerik\.platformio\penv\Lib\site-packages\platformio\compat.py", line 27, in <module>
    from asyncio import create_task as aio_create_task
  File "C:\Users\ruzgarerik\.platformio\python3\Lib\asyncio\__init__.py", line 42, in <module>
    from .windows_events import *
  File "C:\Users\ruzgarerik\.platformio\python3\Lib\asyncio\windows_events.py", line 8, in <module>
    import _overlapped
OSError: [WinError 10106] The requested service provider could not be loaded or initialized
Traceback (most recent call last):
  File "<frozen runpy>", line 198, in _run_module_as_main
  File "<frozen runpy>", line 88, in _run_code
  File "C:\Users\ruzgarerik\.platformio\penv\Scripts\platformio.exe\__main__.py", line 4, in <module>
  File "C:\Users\ruzgarerik\.platformio\penv\Lib\site-packages\platformio\__main__.py", line 21, in <module>
    from platformio import __version__, exception, maintenance
  File "C:\Users\ruzgarerik\.platformio\penv\Lib\site-packages\platformio\maintenance.py", line 22, in <module>
    from platformio import __version__, app, exception, fs, telemetry
  File "C:\Users\ruzgarerik\.platformio\penv\Lib\site-packages\platformio\app.py", line 24, in <module>
    from platformio import __version__, exception, fs, proc
  File "C:\Users\ruzgarerik\.platformio\penv\Lib\site-packages\platformio\fs.py", line 28, in <module>
    from platformio.compat import IS_WINDOWS
  File "C:\Users\ruzgarerik\.platformio\penv\Lib\site-packages\platformio\compat.py", line 27, in <module>
    from asyncio import create_task as aio_create_task
  File "C:\Users\ruzgarerik\.platformio\python3\Lib\asyncio\__init__.py", line 42, in <module>
    from .windows_events import *
  File "C:\Users\ruzgarerik\.platformio\python3\Lib\asyncio\windows_events.py", line 8, in <module>
    import _overlapped
OSError: [WinError 10106] The requested service provider could not be loaded or initialized
Traceback (most recent call last):
  File "<frozen runpy>", line 198, in _run_module_as_main
  File "<frozen runpy>", line 88, in _run_code
  File "C:\Users\ruzgarerik\.platformio\penv\Scripts\platformio.exe\__main__.py", line 4, in <module>
  File "C:\Users\ruzgarerik\.platformio\penv\Lib\site-packages\platformio\__main__.py", line 21, in <module>
    from platformio import __version__, exception, maintenance
  File "C:\Users\ruzgarerik\.platformio\penv\Lib\site-packages\platformio\maintenance.py", line 22, in <module>
    from platformio import __version__, app, exception, fs, telemetry
  File "C:\Users\ruzgarerik\.platformio\penv\Lib\site-packages\platformio\app.py", line 24, in <module>
    from platformio import __version__, exception, fs, proc
  File "C:\Users\ruzgarerik\.platformio\penv\Lib\site-packages\platformio\fs.py", line 28, in <module>
    from platformio.compat import IS_WINDOWS
  File "C:\Users\ruzgarerik\.platformio\penv\Lib\site-packages\platformio\compat.py", line 27, in <module>
    from asyncio import create_task as aio_create_task
  File "C:\Users\ruzgarerik\.platformio\python3\Lib\asyncio\__init__.py", line 42, in <module>
    from .windows_events import *
  File "C:\Users\ruzgarerik\.platformio\python3\Lib\asyncio\windows_events.py", line 8, in <module>
    import _overlapped
OSError: [WinError 10106] The requested service provider could not be loaded or initialized
Traceback (most recent call last):
  File "<frozen runpy>", line 198, in _run_module_as_main
  File "<frozen runpy>", line 88, in _run_code
  File "C:\Users\ruzgarerik\.platformio\penv\Scripts\platformio.exe\__main__.py", line 4, in <module>
  File "C:\Users\ruzgarerik\.platformio\penv\Lib\site-packages\platformio\__main__.py", line 21, in <module>
    from platformio import __version__, exception, maintenance
  File "C:\Users\ruzgarerik\.platformio\penv\Lib\site-packages\platformio\maintenance.py", line 22, in <module>
    from platformio import __version__, app, exception, fs, telemetry
  File "C:\Users\ruzgarerik\.platformio\penv\Lib\site-packages\platformio\app.py", line 24, in <module>
    from platformio import __version__, exception, fs, proc
  File "C:\Users\ruzgarerik\.platformio\penv\Lib\site-packages\platformio\fs.py", line 28, in <module>
    from platformio.compat import IS_WINDOWS
  File "C:\Users\ruzgarerik\.platformio\penv\Lib\site-packages\platformio\compat.py", line 27, in <module>
    from asyncio import create_task as aio_create_task
  File "C:\Users\ruzgarerik\.platformio\python3\Lib\asyncio\__init__.py", line 42, in <module>
    from .windows_events import *
  File "C:\Users\ruzgarerik\.platformio\python3\Lib\asyncio\windows_events.py", line 8, in <module>
    import _overlapped
OSError: [WinError 10106] The requested service provider could not be loaded or initialized
PS D:\Dosyalar\Projeler\Aqualley\Embedded\aqualleyfw>

I installed platformio completely from vscode