Eager to implement handling of mbedignore within pio

Hello!

I would like to contribute to Plarformio Core by submitting support for mbedignore file within the project.

I have already written those pre-extra scripts. One of them is mbedignore.py which may be used as a pre: script to ignore specific folders within the mbed framework.

The script works in such a way that for each entry in the .mbedignore within the PIO project, it creates a .mbedignore file within the mbed-os framework, which is a package downloaded and maintained by pio (e.g. on Linux it is ~/.platformio/packages/framework-mbed). Each entry in the main .mbedignore is a subdirectory within the mbed-os framework. In turn, the script puts in each such subdirectory a .mbedignore file that has content “*” to match everything within that directory. In case the .mbedignore already exists in that subdirectory, it appends a line with “*” to that .mbedignore file. The script is able to undo the operations, because it puts the previous, global .mbedignore (from the root PIO project) to the mbed framework root directory. On each invocation (pio run runs the pre: script) this script compares the .mbedignore from the root directory of the PIO project against the one inside mbed framework. The positive diff are the subdirectories which shall be ignored, and the negative diff are the subdirectories which shall be unignored. This means that switching between projects that use different .mbedignore files is supported.

As you can see, this is quite a struggle and complex logic. Mbed already supports having a single .mbedignore within the root repository of the main project. Generally, mbed-only projects use the mbed-cli to build, run, test, … the project. For such a project, under the mbed-os subdirectory, the sources of mbed OS are downloaded. The directory tree for such a project looks like that:

|
| ---- .mbedignore
| ---- mbed-os/
| ... others

.mbedignore contains such entries:

mbed-os/connectivity/cellular/*
mbed-os/connectivity/drivers/cellular/* 
mbed-os/connectivity/netsocket/source/Cellular*.*

So as you can see, it takes into account a single .mbedignore without a need to put .mbedignore within each subdirectory which is an entry in the file.

Since single .mbedignore files, within the mbed framework package downloaded by pio, are used normally as mbed-cli would do it, and I couldn’t find any code within pio-core repository which reimplements that, my conclusion is that pio-core utilizes mbed-cli to build the project. I tried to figure out how does it do it, but with no luck. My main question is then:

How is mbed-cli used by pio-core? Could you reference the code which executes it?

Ideally, I would like to reuse the normal mbed-cli behavior that handles the .mbedignore within the root directory of the project. Then the complex logic related to ignoring/unignoring based on positive/negative diff would not be needed.

If it is not possible, then I would like to incorporate the script to pio-core, but some improvements would have to be made, since the mbedignore.py is not ideal:

  • Support “official” file format (thanks @copperbot-d): at least the use of mbed-os prefix and * “match all” suffix. That would allow easier project porting from “mbed-only” projects to “pio-with-mbed” projects. I don’t plan to support the format entirely, especially the ?, [seq], [!seq] patterns. They are used rarely.
  • Advanced handling of switching between various mbed-based projects. If the user has multiple “pio-with-mbed” projects, and switches between them. It means that some projects may use .mbedignore and some not at all. The implementation would have to take that case into account to unignore the subdirectories that has been ignored by the project run previously.

Once again, this would be supported out-of-the-box if we could supply a single .mbedignore to the mbed-cli.

1 Like

Not directly. To build the firmware for any framework, PlatformIO pushes the work into a dedicated builder script for the framework. Look at e.g. platform-ststm32/builder at develop · platformio/platform-ststm32 · GitHub, you will see a main.py script (this is mainly responsble for upload logic and high-level firmware generation and conversion logic) and then the sub-builder scripts like framework/mbed.py. These are however are only forwarder-scripts which forward the work to framework-mbed/platformio/platformio-build.py, which located in the package of the mbed-os framework. This is done so that the package can arbitrary implement the builder script as it needs for a particular mbed-os version, otherwise we’d have to have one static script in the platform supporting every version.

The script that is found there is also centralized in GitHub - platformio/builder-framework-mbed: ARM mbed build script for PlatformIO Build System. The real work happens here, by using the Python APIs provided by the mbed-os framework. You can see e.g. the self.ignore_dirs variable being used in this function. This script would need changing to support that, no need to modify the core (platformio/platformio-core), that’s the wrong place. The script should be locally in C:\Users\<user>\.platformio\packages\framework-mbed\platformio\platformio-build.py (and friends).

Mbed-CLI is just a command-line tool frontend to the Python functionality of the mbed-os APIs.

I can also suggest reading through this to understand the inner workings.

How did you see that? It must 100% be called when building. Even when it’s not actually rebuilding files I think. You can try a pio run -t clean.

Yes, you are right.

Sorry, I have edited the wrong part of the code. When I realized that, I have removed the post.

I have created a PR: Load .mbedignore from project root directory by KKoovalsky · Pull Request #26 · platformio/builder-framework-mbed · GitHub.

tldr: @maxgerhardt I couldn’t use self.ignore_dirs, because of a bug in MBED OS’s build API. See the PR for more explanation.

2 Likes

Looking great :slight_smile:

I couldn’t use self.ignore_dirs, because of a bug in MBED OS’s build API. See the PR for more explanation.

My hunch on using self.ignore_dirs was not founded at all, just came from a 3-second look at the source code looking for something that says “ignore” and has to do with the build process. But interesting that it incovered an mbed-os API bug. Looking forward to the merge, this is a good improvement.