Calling pio programatically or setting extra_scripts with command line flags

Hello!
I’m working on a script to call PIO to build and flash firmware for a bunch of devices.
Right now, we’re writing device specific config to a file in theproject, and calling pio with the --upload-port option to point to each different device.

I now want to upload multiple ways(flash over serial or OTA), and that involves changing the extra_scripts config directive.
The only option I see right now is to run sed on platformio.ini, which doesn’t seem great.
Is there any way I missed to set it via command line flag?
Is there any way to programmatically start a PIO build?

Cheers

Not quite sure what you’re trying to do here… can this not be done with the different device environments? i.e. a device environment for each device for say serial uploading, and then a duplicate set for OTA uploading? Then you can just run whichever combination of builds/uploads you need?

Right now, my source of truth for device configs is outside of platformio, I’m using pio to just build and flash stuff.

Right now I have a script, autoprogram.py, and a config file, devices.json. The autoprogram takes the list of devices to build/flash from the device file, writes a config file in the directory with the pio will build so the device has a config, and then the script runs pio with the --upload-port flag.

Someone said that they think every config option is also a flag that can be passed to pio, so I will try passing --extra-scripts late tonight.

Ideally, I’m hoping for a link to documentation I can’t find on how to call pio from within python without running a subshell.

PlatformIO does have some pretty advanced scripting that I’m only just starting to scratch the surface on… maybe the custom targets stuff is what you need? Either way, there are some other examples on that page that might help you or give some ideas. including how to import platformio in your python script :wink:

So that page appears to be about extra scripts pio can run, and not about calling pio.

It even says:

You can not run/debug these scripts directly with Python interpreter. They will be loaded automatically when you processing project environment using platformio run command.

All the imports I saw were by scripts called from extra_scripts, am I missing something else on that page?

Looking at the source for pio run, there is no way to pass --extra-scripts
https://github.com/platformio/platformio-core/blob/develop/platformio/commands/run.py#L33
so that won’t work either.

I guess I’ll just use sed on platformio.ini before I call pio run.
It’s hacky but it will work.

Yeah, thinking about it further it seems a bit chicken and the egg for your usage… the scripts are all about how to add stuff to the different environments/boards you might have set up in platformio.ini, whereas it seems that how you are using it is in the form of a standalone json file that would create the environments itself…

Why that way? Why not have all the environments configured directly in the platformio.ini file, and then just building/uploading the specific devices you want to build/upload to? i.e. pio run -t target-board --upload-port Would save the messing around with the platformio.ini for each board your program…

To be honest, I don’t understand your problem. You can declare multiple build environments in platformio.ini and later switch between them via pio run -e myenv.

Also, maybe this will help Environment variables — PlatformIO latest documentation

PlatformIO Core is fully flexible. You should not have any problems to use it with outside scripts. Please provide more detailed use case and we will help.

Yeah, I think PLATFORMIO_EXTRA_SCRIPTS will work!

Could you put a short description of ways you can set extra scripts on the page documenting extra_scripts?
http://docs.platformio.org/en/latest/projectconf/section_env_advanced.html#projectconf-extra-scripts
I had looked for ways to set it, but completely missed the environment variables page.

Use case explanation:
TL;DR: We need a script that can deal with multiple PIO projects and 100+ devices in multiple locations that need slightly different configs, we need to store device configs in the easiest to read/modify way possible.

More detailed: We have 100+ tasmota [ GitHub - arendst/Tasmota: Alternative firmware for ESP8266 with easy configuration using webUI, OTA updates, automation using timers or rules, expandability and entirely local control over MQTT, HTTP, Serial or KNX. Full documentation at ] devices, as well as a bunch of other devices that are not yet compiled with pio(That’s what my other thread here is about, and I hope to have time to confirm the fix works on my end later this week).

We have written several scripts to help flash devices, the current version you can see here: https://github.com/i3detroit/Automatic-IoT-module-programming/tree/serial
Apologies for the current state of everything there, I just started cleaning up the python version someone else put together else wrote a few days ago.

tasmota.yaml is the current set of device configs, but I plan to simplify that and make it not yaml so it’s readable, and if possibly simplify it even more soon.

Our script writes a config file, runs sed on platformio.ini to set flashing mode, and then calls pio for every device.
Using the environment variable will make that much cleaner.

Before now, we had no way to set flashing mode, so there was no reason to try to change extra_scripts, it was just set it once, and then it would be fine.

There are also non-tasmota devices we have to deal with, so right now that’s just a completely separate script, but I’m hoping to port those other codebases to platformio(That’s the other thread I have here, I’m hoping to confirm the fix worked later this week, sorry about the delay) and then use our new script to flash those as well.

Edit: Completely different issue: the forum told me I can’t link to the host docs.platformio.org in my reply, I had to edit the above link in. No idea why.

How about with the next platformio.ini?

[env:tasmota_group_1]
custom options for group 1

[env:tasmota_group_n]
custom options for group n

So, you will have different configurations for your devices. Later, you can flash them with specified -e ....

1 Like