PlatformIO integration into Embeetle IDE

[kristof.mulier]

We’re building a new, free IDE for embedded software:

https://embeetle.com

Embeetle IDE is not based on any existing IDE, such as Eclipse or VSCode. We built it from scratch, focusing solely on embedded software. That gives us a great flexibility in shaping its design - an advantage we wouldn’t have if we had based it on an existing, general-purpose IDE.

We would like to integrate PlatformIO into into Embeetle. Unfortunately, there is a technical barrier here before we can even get started. I’ll explain.

Embeetle: self-contained projects

In Embeetle, we stick to the “self-contained project” principle. Each Embeetle project should have all its source files within the project folder (and its sub-folders). Even libraries should be inside the project. There are pro and contra arguments for this. I’ll just mention them for completeness, although they’re not the topic of this post:

• PRO:

  • It’s easy to navigate through the code-base. There’s no need to figure out a “library system” first, to know what files actually belong to the project and where to find them. A pure, self-contained project is as simple as it can get: only the files inside the project folder belong to the project. No confusion possible - not for the user, nor for the source code analyzer/parser.

  • You send a project to a colleague, and it will just work on his PC. There are no dependencies on files outside the project folder.

• CONTRA:

  • All libraries are duplicated each time you create a new project. However, this shouldn’t be a big deal if you use a version control system like Git.

Now we get to the actual problem at hand: PlatformIO is following a different strategy here. Unlike Embeetle, PlatformIO projects rely on external libraries (without a local copy in the project itself).

PlatformIO: 'external' libraries

My experience with PlatformIO is still limited, but I understand that the typical project looks like this:

📂 my_project
├ 📁 include         project header files and header
│                    files from external libraries
├ 📁 lib             project specific / private libraries
├ 📁 src             project source files
└ 📄 platformio.ini  project config

Only the so-called “private / specific” libraries are fully present within the project folder. All other libraries are somewhere in an external folder, such as ~/.platformio/. Take for example the Arduino Wire library, which can be found at:

~/.platformio/packages/framework-arduino-avr/libraries/Wire/src/Wire.cpp

Is there a way to instruct PlatformIO to make all libraries local? In other words, can one “ask” PlatformIO to keep a local copy in the my_project/lib folder for each library?

If yes, that would make it so much easier for us to integrate PlatformIO into Embeetle.

[maxgerhardt]

Redirect the needed folders to the current project directory per documentation.

Example

[platformio]
; redirect *all* packages, so e.g. framework-arduino-avr. 
; this would however also redirect compiler packages, like toolchain-atmelavr, which 
; you might want to .gitignore.. (they're OS  & arch specific anyways)
packages_dir = $PROJECT_DIR/packages
; redirect the standard `.pio\lideps` directory if needed for better visibility
libdeps_dir = $PROJECT_DIR/libdeps
; redirect global libraries which normall go into <home dir>/.platformio/libs
globallib_dir = $PROJECT_DIR/global_libs

[env:uno]
platform = atmelavr
board = uno
framework = arduino
lib_deps =
   bblanchon/ArduinoJson @ ^6.18.0

Such a project would have a folder structure like…

You can also see that that is used by other projects like GitHub - Zetaohm/DaisyPod_Platformio_Example.

Tagging @ivankravets because of new IDE integration.

[kristof.mulier]

Hi @maxgerhardt ,
Thank you very much for the suggestion. I just tried it, and it works great

Of course, it’s a bit unfortunate that the whole toolchain ends up inside the project. Perhaps this behavior can be modified with some fine-tuning?

PS: Maybe not very relevant at the moment, but I was just wondering what this line in your platformio.ini script means:
    lib_deps = bblanchon/ArduinoJson @ ^6.18.0.

Kind regards,
Kristof

[maxgerhardt]

@kristof.mulier

Of course, it’s a bit unfortunate that the whole toolchain ends up inside the project. Perhaps this behavior can be modified with some fine-tuning?

It would be good if this feature was inside the core maybe, Issues · platformio/platformio-core · GitHub is open for feature requests. Probably needs a bit of extra-code to handle two simulatenous “package” directories then, one for framework code and one for toolchains / other stuff. Pretty sure there’s no fine-tuning options of thaht as of now.

@kristof.mulier

PS: Maybe not very relevant at the moment, but I was just wondering what this line in your platformio.ini script means:
lib_deps = bblanchon/ArduinoJson @ ^6.18.0.

lib_deps declares a library dependency (ArduinoJson) to PlatformIO’s library dependency finder (LDF) system. It’s the canonical way of adding libraries from the PlatformIO library registry to a PlatformIO project. These libraries are stored in the usually hidden folder .pio\libdeps\<environment name>\library name are not pushed to the project repo – PlatformIO can just download them from the registry of the required version anytime, so no point in storing them per-project. The lib/ folder is usually where user-created libraries are stored. It was just there to show that usually hidden library storage folder can be made more visible in the project.

[maxgerhardt]

Ah, on a different sidenode: Usually integration of an IDE with PlatformIO is done by teaching PlatformIO how to generate a project file, specific for that IDE. Take a look in the templates (tpls) folder in the core. So e.g. for VSCode, PlatformIO generates a .vscode/c_cpp_properties.json in which it puts all the build information (e.g. activated global macros, C/C++ versions, …), path information for the framework files, project files and concrete toolchain executable, wherever they might be – since PlatformIO is the build system, it knows where everything is, and the IDE shouldn’t try and figure this out in its own. Then the IDE also doesn’t need to have restrictions on where files are stored (e.g. only in the project folder) if it reads this path information.

For building then, the IDE always just calls into PlatformIO (pio run ..) and doesn’t attempt to build the files on its own – otherwise it would have to figure out / emulate the entire PlatformIO build system to get to the same build commands.

So if that’s the way you want to go, have a look at the contributing (and developing) page for PlatformIO, then you guys can implement whatever template logic is necessary for your IDE, akin to the 11 other IDE templates already present. Once that logic is added, you can tell PlatformIO to generate the project files for you ide with the pio init --ide=name command.

Or contact Ivan Kravets, linked above, for integration help.

[kristof.mulier]

POST 1/3:
I’m spreading this post over three posts, because I’m limited in the amount of links and figures I can add per post as a new user.

1. About the PIO library dependency finder

@maxgerhardt

lib_deps declares a library dependency (ArduinoJson) to PlatformIO’s library dependency finder (LDF) system. It’s the canonical way of adding libraries from the PlatformIO library registry to a PlatformIO project. These libraries are stored in the usually hidden folder .pio\libdeps\<environment name>\library name

Thank you for the explanation. Please excuse me for my lack of knowledge about PlatformIO. I started just recently diving into PIO, and I’m still exploring.

The libraries downloaded by the “PIO library dependency finder”, are they all stored into the $PROJECT_DIR/.pio/libdeps/<environment>/ folder? It looks like some libraries are stored in ~/.platformio/packages/ folder (such as ~/.platformio/packages/framework-arduino-avr/libraries/). As of now, it’s unclear to me where the “PIO library dependency finder” stores the required libraries.

[kristof.mulier]

POST 2/3:
I’m spreading this post over three posts, because I’m limited in the amount of links and figures I can add per post as a new user.

2. About PIO integration

@maxgerhardt

Usually integration of an IDE with PlatformIO is done by teaching PlatformIO how to generate a project file, specific for that IDE. Take a look in […]

Thank you so much for this information. I will certainly dive into that material

3. Automatic source file selection

@maxgerhardt

For building then, the IDE always just calls into PlatformIO (pio run ..) and doesn’t attempt to build the files on its own – otherwise it would have to figure out / emulate the entire PlatformIO build system to get to the same build commands.

Right, that looks the best way to approach this. In fact, there is another technical barrier here for full integration into Embeetle IDE. I’ll explain.

Very early on in the process of making Embeetle, we figured that most people get lost in embedded software projects because of the huge quantity of files (eg. driver files) - even though 90% of them are not used in the project at hand. Therefore, our code parser separates the “used” source files from the “unused”. It basically starts from main.c and recursively travels through the codebase to figure it all out. Next, the code parser visualizes this information in the filetree:

Each file gets a green (used) or red (unused) checkbox. Folders get a checkbox too, based on the status of their codefiles (can be mixed red/green if some of their files are used and others unused).

What a delight if you open the Drivers folder, and you can immediately distinguish the “used” from the “unused” ones! Even better, you write #include my_driver.h at the top of your main.c file, and - magically - the my_driver.h and my_driver.c pop their checkbox green in the filetree. Brilliant, right? (Sorry if this sounds like an ad - I’m just being carried away here )

We named it the automatic source file selection mechanism. It’s all explained in this comic:
https://embeetle.com/#comics/src_files_selection

[kristof.mulier]

POST 3/3:
I’m spreading this post over three posts, because I’m limited in the amount of links and figures I can add per post as a new user.

Now, all of this requires the IDE to have some level of control over the build system. In Embeetle, we auto-generate the file filetree.mk and put it right next to the user’s makefile:

All our sample projects have a default makefile that already includes this filetree.mk. The user is free to change whatever he wants in this makefile. As long as it includes filetree.mk and properly uses it to decide what source files to build - the system works.

What about PIO?

So how can we make this work with PIO? I foresee some troubles here. On the one hand, our source code parser will be (trying) to figure out all the files that are being used. On the other hand, PIO is doing something similar.

This is quite a difficult challenge. To solve this gracefully, we might need to get in contact with the PIO developers?

What do you think?

Many thanks for your help. Please feel free to add me on LinkedIn, I’d like to stay in touch. You can find me there by my full name: Kristof Mulier.

Kind regards,
Kristof

[maxgerhardt]

@kristof.mulier

It looks like some libraries are stored in ~/.platformio/packages/ folder (such as ~/.platformio/packages/framework-arduino-avr/libraries/). As of now, it’s unclear to me where the “PIO library dependency finder” stores the required libraries.

The framework, in this case the Arduino AVR framework, does have built-in libraries, and so does virtually every other Arduino core, and lots of other frameworks. PlatformIO doesn’t put them there, the Arduino core is laid out in such a way that these libraries are there. See e.g. GitHub - arduino/ArduinoCore-avr: The Official Arduino AVR core and GitHub - adafruit/ArduinoCore-samd. PlatformIO internally adds these to the “search path” of libraries (done e.g. here), so when the firmware code does e.g. #include <Wire.h>, PlatformIO will also be looking in the libraries/ folder of the Arduino core and then find e.g. Wire. PlatformIO will never modify this Arduino core source data. It won’t put user libraries or libraries the user wants to pull from the registry in ~/.platformio/packages/<arduino core>/libraries. Only libraries declared via lib_deps will go into the folder designated by the libdips_dir setting (which is usually .pio\libdeps\<env>\<lib>), nothing else.

[maxgerhardt]

@kristof.mulier

Now, all of this requires the IDE to have some level of control over the build system

Hm yes this is where the problem comes in, since PlatformIO is technically the build system, and it just informs the IDE about what files it would build, with which settings, etc., as mentioned above … I don’t know the specifics of your IDE, but you might be able to have PlatformIO generate this filetree.mk (which is the “project configuration file” the IDE needs to operate and visualize all the code?)

@kristof.mulier

To solve this gracefully, we might need to get in contact with the PIO developers?

Can’t hurt to sync with them. The “Contact Us” page at About Us | PlatformIO Labs has an E-Mail address which should be used for these inquires I think. I’ve also already tagged the PlatformIO / PIO Labs CEO Ivan above, maybe he has additional thoughts to share when he reads the post :).

[kristof.mulier]

Thank you so much @maxgerhardt for all the efforts you did today, to help us out

For full integration with the filetree in Embeetle IDE, there should be a two-way communication between the IDE and the build system. Ideally that looks like this:

                    Generate a list of used
                    (relevant) source files
┌────────┐         <========================  ┌────────────┐
│EMBEETLE│                                    │BUILD SYSTEM│
│IDE     │          Provide a list of         │(PIO)       │
└────────┘          user-overrides            └────────────┘
                   ========================>
Display a red                                 Build only the
or green checkbox                             relevant source
for each source                               files, respect the
file, indicating                              manual overrides
its "used"/"unused"                           from the user (force
status                                        include/exclude)

In other words:

• The build system (eg. PIO) chooses the relevant source files and communicates that to Embeetle IDE. Embeetle then displays that in the filetree: every source file gets a red or green checkbox indicating its “used” vs “unused” status. The role of Embeetle is hereby only to give feedback.

• The user can override the used/unused status of a source file in the filetree. He clicks on the checkbox and “forces” the status of the given file. The checkbox then turns red/green (whatever the user chose) and displays a lock on top of it, to make clear that this was a user override. The IDE then communicates this to the build system, which must then take the necessary steps to ensure that the user’s wishes are respected.

The “necessary steps” in our standard Embeetle projects simple: the filetree.mk file (which basically lists all the files that must be compiled) gets regenerated, taking the user overrides into account. I bet that this won’t be so simple when it comes to PIO.

A potential route we can take here, is to work in two steps. First, we integrate PIO in such a way that the Filetree merely gives feedback about files being “used”/“unused”, without allowing the user to override that. In a second step, we take care of the user overrides.

[ivankravets]

Have you seen src_filter option? You generate src_filter on-the-fly.

[kristof.mulier]

Thank you for the tip @ivankravets . The src_filter only works for the src folder, right? Is there something similar for other files, eg. library files, framework files, … ?

[ivankravets]

Yes, users have controls on the source code:

• src_filter - filter project source files
• srcFilter - filter library source files
• Build Middlewares - filter different source files (project, libraries, frameworks, etc.).