PlatformIO Community

Platform configuration in custom tasks?

I’ve been trying to capture some of my one-off external tools as part of my platformio configuration via tasks. For example, I have a script to launch GDB against my ESP8266 module that I’d like to create as a new task. Is it possible to determine platformio configuration variables as part of task definitions? For example, in this command:

~/.platformio/packages/toolchain-xtensa/bin/xtensa-lx106-elf-gdb

Is there a way to determine the path to the GDB executable? Similarly, is there a way to determine the ELF file:

~/source/my/project/.pio/build/nodemcuv2_serial_gdb/firmware.elf

I’ve read various parts of the documentation, but I’m clearly missing something. Any input appreciated.
Thanks!

I’m not entirely sure what you’re trying to do, but there are ways to get the gdb executable path for the currently used toolchain. You can do that as an adaption of that code, by changing the "g++" to "gdb".

The path to the current general build folder can be obtained by env.subst("$BUILD_DIR") as seen in e.g. docs. Given then the name of the environment, you can automatically decude the the <environment name>\firmware.elf final part of the path. I’m sure if you do print(env.Dump()) there will be some attribute in env that will reveal the name of the environment for you.

Thanks. Very helpful. It looks like I could probably use this information along with https://docs.platformio.org/en/latest/projectconf/advanced_scripting.html#custom-targets to get what I need if I’m reading this correct?

Yes with that you can create a new task. Just note that actions can also be a (list) of Python function names that you want to call – such an example is also a bit further down in the docs.

I also have an example of a custom target implemention here.

Thanks!

Funny enough, I was looking for something like your tool recently. I didn’t manage to get the right combination of search terms for this to show up, but it will come in very handy.

Thanks to all of your help, I was able to get most of the custom script put together for launching GDB. I have all of the parameters I need to launch the command. However, I’m really naive in terms of Python (long-time Java programmer). What I can’t figure out is the right set of parameters I want to use to launch GDB as a subprocess with my parameters and have proper shell interaction. Can you point me in the direction of any examples that might help me out?

Hmm maybe try the code in the question and the first answer at https://stackoverflow.com/questions/325463/launch-a-shell-command-with-in-a-python-script-wait-for-the-termination-and-ret, but with stdin=subpress.PIPE in the argument list of subprocess.Popen so that that is forwarded? (Or maybe without both stdin and stdout set specifically so that it goes to the sys.stdout and sys.stdin directly).

The first solution with os.execlp is rather destructive – replaces the currently running process with the program invocation you choose. Once you exit GDB, execution will not be returned to the Python script, since it doesn’t exist anymore. So no processing can be done after the GDB invocation in Python if that’s needed.

The other solution would be to use PIPE for both stdin and stdout (and stderr I guess) and periodically poll user input and feed it into the process, see https://stackoverflow.com/questions/50434204/running-interactive-program-from-within-python

Maybe also a really primitive os.system(cmd) works?

Thanks again for all of your help. I’m so close!

I’ve been able to get this basically working using Popen(…).wait():

subprocess.Popen([gdb, "-x", gdb_commands, elf_file], shell=True, start_new_session=True).wait()

While this works, I don’t see the (gdb) prompt until after I enter the command and hit return. I thought os.execl might work better:

os.execl(gdb, "-x", gdb_commands, elf_file)

However, that seems to ignore the -x parameter for some reason:

For help, type "help".
Type "apropos word" to search for commands related to "word"...
"/tmp/commands7kleuwet.gdb": not in executable format: file format not recognized
"/home/xxxx/source/personal/yyyy/.pio/build/nodemcuv2_serial_gdb/firmware.elf" is not a core dump: file format not recognized

Do you have any suggestions?
Thanks again!

Probably because the standard-output is buffered and it only gets flushed with a newline to the user, but GDB just writes (gdb) without flushing / newline. See https://stackoverflow.com/a/52851238/5296568 for workarounds. An easy one might be adding bufsize=0, universal_newlines=True to the arguments of the .Popen() call.

Thanks. Buffering makes sense, although I have not yet found the magic incantation :slight_smile: Any idea why execl doesn’t work at all? I’ve poked around, but haven’t found a good example to see what I might be doing wrong when trying that invocation.

Okay I think this might be a very special case with the SCons script invocation.

The main python script / SCons probably internally creates a new process for each invocation of a user’s extra_script. And there’s some mechanism to read from that process’s output and display it back in the main processes’s output. But that thing is also buffered or reads line-wise. A simple test like

import sys
Import("env")
print("Normal output")
print("No newline at end but flushed", end="")
sys.stdout.flush()
env.Exit(-1)

as an extra_scripts script reveals output

Dependency Graph
|-- <ESP8266WiFi> 1.0 (C:\Users\Max\.platformio\packages\framework-arduinoespressif8266\libraries\ESP8266WiFi)
Building in release mode
Normal output
======================== [FAILED] Took 0.99 seconds ========================

so the second output is not displayed although we flush the output of our process. If one would execute that as a normal python script (with env stuff removed)

>python test.py
Normal output
No newline at end but flushed

it does output.

@ivankravets do you know about these special cases when an extra_script does print something but without a newline (but still flushes) why it wouldn’t output something in the terminal?

The special usecase here is that e.g. pio run -t specialgdb would launch the user, in the terminal, in a GDB session, but when using code like

import os 
import re
import sys 
import time
import subprocess
Import("env")
#print(env.Dump())

gdb_path = ""

# Find the current platform compiler by searching the $PATH
# which will be in a platformio toolchain bin folder
path_regex = re.escape(env['PROJECT_PACKAGES_DIR'])
gcc = "gdb"
if env['PLATFORM'] == 'win32':
    path_separator = ';'
    path_regex += r'.*\\bin'
    gcc += ".exe"
else:
    path_separator = ':'
    path_regex += r'/.+/bin'

# Search for the compiler
print("PATH: " + str(env['ENV']['PATH'].split(path_separator)))
for pathdir in env['ENV']['PATH'].split(path_separator):
    if not re.search(path_regex, pathdir, re.IGNORECASE):
        continue
    for filepath in os.listdir(pathdir):
        if not filepath.endswith(gcc):
            continue
        # Use entire path to not rely on env PATH
        filepath = os.path.sep.join([pathdir, filepath])
        print("Compiler found: " + str(filepath))
        gdb_path = str(filepath)

def command_gdb(*args, **kwargs):
    print("Entrypoint")
    # attempt to set output to unbuffered - unsuccessfull though
    os.environ["PYTHONUNBUFFERED"] = "1"
    print("GDB path = " + gdb_path)
    # does not work: input is not echoed, "(gdb)" prompt does not appear
    os.system(gdb_path)
    #os.execl(gdb_path, "--help")
    # workaround.
    # for windows: "start" <command> opens .exe directly wihch opens a cmd "GUI" window.
    # for linux we probably want to open a new GUI shell or something?
    # process output is not displayed inline..
    #subprocess.Popen(["start", gdb_path], bufsize=0, shell=True, start_new_session=True, stdout=sys.stdout, stdin=sys.stdin).wait()

env.AddCustomTarget(
    name="specialgdb",
    dependencies=None,
    actions=[
        command_gdb
    ],
    title="Launch special GDB",
    description="Launch special GDB"
)

the output is

command_gdb(["specialgdb"], [])
Entrypoint
GDB path = C:\Users\Max\.platformio\packages\toolchain-xtensa\bin\xtensa-lx106-elf-gdb.exe
GNU gdb (GDB) 8.2.50.20180723-git
Copyright (C) 2018 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
Type "show copying" and "show warranty" for details.
This GDB was configured as "--host=x86_64-w64-mingw32 --target=xtensa-lx106-elf".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
    <http://www.gnu.org/software/gdb/documentation/>.

For help, type "help".
Type "apropos word" to search for commands related to "word".

when the final lines should have been, when just invoking the program

grafik

Also output is not echoed which is a mystery to me.

@csetera there’s also a workaround at the bottom of that code that you might want to look at.

One can also do other fancy workarounds. Like e.g., let the Python script write a .bat file that has the correct invocation line and then wrap it like pio run -t specialgdb && run_gdb.bat in one command

Thanks for continuing to dig into this. I’m glad to see that it isn’t my lack of Python experience that is causing this to happen. When you mention the workaround, you are talking about using “start” or similar? Since I don’t actually care about the results and just want to launch things, I could potentially try something like that.

This is largely why I was hoping I could get execl working since I was hoping it would replace all of this additional processing with the new application. Am I understanding the behavior of execl correctly? If so, I’d love to figure out why it seemed to be “eating” my -x parameter and confusing GDB.

Yes, see $GDB https://github.com/platformio/platform-espressif8266/blob/develop/builder/main.py#L158

Yes, see

elf_path = join("$BUILD_DIR", "${PROGNAME}.elf")