Hello everyone!
For a personal project I would like to create a custom Platform that would allow me to automatically download, build and upload a custom WiFi firmware (nexmon) to a Raspberry Pi 3 Model B.
I am pretty new to platformio but I’ve read myself through the docs and looked at the examples and tried to reproduce some things but I just don’t really get the grasp of the general procedure and am kinda stuck at the moment, so I would really appreciate some help/pointing in the right direction.
What I did so far is:
Creating my nexmon-platform folder with the following structure
/nexmon-platform/
/boards/
/raspberrypi_3b.json
/builder/
/frameworks/
/nexmon.py
/main.py
/platform.json
/platform.py
Now to my problem:
I don’t really know whether my current approach is even correct. Since nexmon is supposed to be a firmware patch for the target board I figured it would fall under the category of framework so that I could specify in the nexmon.py how the firmware is build etc but I don’t really know how to go on from here or whether I am even going into the right direction so some advise in regard to downloading/building/uploading custom firmware with platformio would be much appreciated!.
Technically we call the platform after the type of chips it compiles for – Nexmon only handles Broadcom chips (or Cypress as they’re bought up now?) so a name like platform-broadcom or platform-cypress would be more standard.
I like the choice that you have a Raspberry Pi 3B as board, but still note that it’s the Broadcom WiFi SoC that you’re compiling a firmware for, not the Pi’s main SoC. But since indeed many BCM chips are associated (in different firmware versions) to different devices, like some smartphone or some Raspberry, it makes sense to create a raspberrypi_3b board definition rather than a BCMXYZ definition, you can also directly associate information like target firmware version needed for the build process in there.
But according to my info the Pi 3 Model B has a “BCM43143”, which isn’t supported by nexmon. The bcm43455c0 for the Pi 3B+ is supported though, as well as the bcm43430a1 for a Pi 3 (that’s what I read when I google it).
The data in nexmon/firmwares at master · seemoo-lab/nexmon · GitHub seems pretty constant for the target chips, so that data can be in a common framework-nexmon package. The builder script builder/frameworks/nexmon.py can then include the right files in the include path / build process.
In the nexmon.py builder script you then have to replicate the build process as the Makefiles in the original project does it. You can ofc. also call into the same shell scripts or whatever that the original build process does so that you don’t have to re-implement everything in python.
Creating a platform is not the easiest thing but doable, but if you’re trying to copy what nexmon does it might get a little bit difficult. You have to fully understand their build process and everything that happens in between – like microcode extraction, extracting flash patches, calling custom GCC toolchains, rebuilding the firmware image, calling a custom assembler on some files for the D11 core etc.
Okay, wow so first of all thank you very much for your detailed reply, this is incredibly helpful for someone like me, who has never done this before!
Now when it comes to the target board I wasn’t sure whether I should target the actual board, the chip is installed on, or the chip directly. But since different boards come with different environments (as my goal in some distant future is to support all the nexmon compatible devices) I thought it would be better to target the board and your explanation backs my approach so I will continue to go for that.
About the Pi 3 Model B support part: I have this board lying around here and I already went through the process of manually flashing Nexmon onto it and except for some compatibility problems I had with the latest Kernel, after downgrading I was able to flash Nexmon onto it without a problem and make use of e.g. the monitoring mode. The original spec of the board actually specifies that it uses the bcm43438, which still isn’t listed on nexmon BUT the explanation is in the readme: “bcm43430a1 was wrongly labeled bcm43438 in the past.”
So the hardware actually matches but thank you for your concern!
Now when it comes to building nexmon for my target, as far as I understand and as you said my nexmon.py would have to include the path to e.g. bcm43430a1 and then go through the build process of the specific firmware in that directory. Now the quick and “hacky” approach that came into my mind, just to get things running, was to just get the path to the specific firmware and execute all the make files. Is that what you meant with calling into the same shell scripts, or is this just an unrealistic “dream” of mine?
In regards of the toolchain: As you said I would have to create a custom toolchain package. I would then have to add this custom toolchain to my platform.json like this:
Now when it comes to understanding the commands/scripts that come from platformio/SCons I find in other implementations, are there any good resources/docs that you could recommand that explain/summarize which command does what? I wasn’t really able to find that many insightful sites about that.
Anyways, thank you again for giving me this incredibly helpful overview! It explained/summarized really well what I wasn’t able to grasp after hours of googling and looking through docs and examples.
I know that this probably isn’t the easiest of projects but I also didn’t expect this to be done in a weekend for someone who has never done something like that before.
With some very crude hacking like that can be done, but it isn’t supposed to work that way. You’re right about how it’s supposed to work – SCons needs all the compile options and to-be-compiled sources and include files and linker flags and linker search paths and hook functions for pre/post actions on some specific files etc. and it will then go off and build it with these settings.
It is possible for the builder script to execute an arbitrary command. The ESP-IDF builder script for ESP32 does that e.g. for generating a specific file. Just calls cmake on a data_file_embed_asm.cmake file and gets the output, using env.Execute(). See here.
But if you do that for the whole build process while not adding any files to the build process (env["CPPPATH"] is empty), the SCons build system would complain with “No files to build – error”. So there at least has to be some dummy file. The other ‘gotcha’ is that before calling make nexmon expects you to execute a shell script that fills some environment variables. When make is called through env.Execute(), it probably gets some empty / clean environment that doesn’t have all these needed variables, so care must be taken to generate those and inject them into the environment as well. See SCons doc on how to control the environment variables. (hint: search env = Environment(ENV = os.environ) in the given link, modify env["ENV"]).
The above link is also a pretty good documentation on what variables / keys are within that env object that is getting used for building. Also, I’ve written the custom platform GitHub - maxgerhardt/platform-w60x: PlatformIO platform for WinnerMicro W60X chips. for some WinnerMicro W600 chips and it is a self-contained platform with builder scripts for the Arduino core and the vendor’s SDK, along with the needed packages / declaration for which compiler to use etc. So some things can also be looked up in there.
Also, the general Advanced Scripting documentation is good there, since that’s a page on how to modify the build process, thus also giving insights on how the build process actually works.
Yes partly, but you’re missing an owner and a version, though. See e.g.
Well actually there are two ways for this – in the platform.json you reference package names that are contained in the PlatformIO trusted package registry (see Service End for Bintray, JCenter, GoCenter, and ChartCenter | JFrog with e.g. toolchain-gccarmnoneeabi). So you can’t reference a path on disk there where your toolchain is, you have to pack and publish the package, which invokes the review process by the PlatformIO team. Also, every uploaded package needs to have a package.json metainformation file in it, but you can grab an existing one from e.g. the toolchain-gccarmnoneeabi package as linked in the bintray or in your local installation (cat ~/.platformio/packages/toolchain-gccarmnoneeabi/package.json).
The way that lets you circumvent the PlatformIO registry and work with local files is that you let your platform.json point to some valid,existing package but then in the platformio.ini of the project immediately overwrite it with e.g .platform_packages = toolchain-gccarmnoneeabi@file://<full path here>, see docs and example.
Yeah that’s kind of what I expected. Would have been to easy to be true if it was possible like imagined in the “hacky” way but now I am excited to get into it!
Overall I can’t thank you enough for the provided docs/examples and explanations. I think I understand what I have to do now. Guess I will just start to dig myself through the provided links while I build the platform up from the ground.
[Meta-Post:] Another side effect of adding nothing to the SCons build system (in terms of include paths, c files to build etc) would be that you couldn’t do any sensible code editing in any editor that PlatformIO supports – PlatformIO wouldn’t export any include search paths since SCons wasn’t given anything in the first place. Running pio run / “Build” would just be a very-very fancy way of calling make a few times in a few directories, as if it were an overglorified shell script on top of nexmon. If you only need to quickly build the code of the target project that’s also what I would recommend, actually. Using the PlatformIO build system and taking the time and work to integrate / replicate the build system of the target project in PlatformIO should have the goal of actually adding value to that by making it nicely editable in any code editor that PlatformIO supports, compatible with the basic build/upload procedure and more easily configurable through the platformio.ini; not just for the sake of being usable in PlatformIO by itself. So I’d say that if something is to be integrated in PlatformIO, then all the way, not a quick-hack that calls into the original build system .
.