Arduino Due (or other SAM3/4) upload fails with ATMEL ICE JTAG on VSCode with PlatformIO

Development Platforms
,
8

I can explain some things high-level and by example, and then link to some further documentation. PlatformIO is an extensive piece of software, so please read this first as a general terminology introduction.

And actually I’ve always been bugged by how much is not documented well, so I’m explaining it more extensively for others as future reference, too, as a “How does PlatformIO work internally”.

Also one may note that in general, a PlatformIO doesn’t have to understand all of this to develop a firmware, the the general docs regarding e.g. platformio.ini, libraries etc. totally suffice. For more advanced stuff, Advanced Scripting is also well-explained with templates that can be copied and adapted without having to understand all underlying things in too much detail. It is however important for people looking to create custom PlatformIO extensions, such as, own platforms, own board definitions, support for other frameworks, etc.

The inner workings of PlatformIO

Involved entities

For reference, these will be explained later on. All these ‘things’ have something to do with the build process.

Packages

So for PlatformIO, the compiler, the framework, uploader tools etc. is a ‘package’. At the start of compilation, PlatformIO prints the used pacakges, e.g.

HARDWARE: AT91SAM3X8E 84MHz, 96KB RAM, 512KB Flash
DEBUG: Current (atmel-ice) External (atmel-ice, blackmagic, jlink, stlink)
PACKAGES:
 - framework-arduino-sam 1.6.12
 - framework-cmsis 1.40500.0 (4.5.0)
 - framework-cmsis-atmel 1.2.2
 - toolchain-gccarmnoneeabi 1.70201.0 (7.2.1)
LDF: Library Dependency Finder -> http://bit.ly/configure-pio-ldf

The packages are stored in your home directory (C:\Users\<user>\ or /home/<user>) and then .platformio/packages/<package name>. Note that there might be multiple packages with the same name but different versions, this will be indicated by a @<version> at the end of the folder name.

All packages must have a package.json file which declares needed meta-information. For example, with which architecutre the package is compatible with (for packages containing binaries compiled for a specific arch & OS, e.g. Windows x64), the version (more below), the name, where it was sourced from, etc… For decoding the version numbers (e.g., 1.70201.0) see Find the mbed version which was used before a backup? - #4 by maxgerhardt and the notes on Semantic Versioning.

So the local compiler might e.g. be in C:\Users\<user>\.platformio\packages\toolchain-gccarmnoneeabi\bin\arm-none-eabi-gcc (and related). The same for the other packages like the Arduino core or accomanying CMSIS files.

PlatformIO sources the content of these packages from its internal registiry (PlatformIO trusted registry). The PlatformIO staff, and also users, can upload packages (aka .tar.gz or .zip files) to this registry. The PlatformIO staff in turn gets the content for the package from the vendor’s official websites (like, ARM GCC, the framework code, etc…), as indicated in the url info of the package.json . See here for how to search the registry. An example where you can see what versions of toolchain-gccarmnoneeabi has built-in is e.g. here. Examples of a package.json should be sourced from the official PlatformIO pacakges (see e.g. locally in C:\Users\<user>\.platformio\packages\toolchain-gccarmnoneeabi\package.json). Here is also an example of mine.

Using the platformio.ini directive platform_packages can be used to easily manipulate the source or version of a pacakge.

Let’s e.g. say that I’m not happy that the standard Arduino Due project is using a ARM GCC compiler of version 7.2.1. I have multiple options. First, I can take a look at what other versions the PlatformIO staff has for toolchain-gccarmnoneeabi, as linked above. This will lead me to e.g. discover that there is a package version 1.90301.200702, which per above link encodes GCC 9.3.1, datecode 2020, June 2nd. So, I can add

platform_packages = 
   toolchain-gccarmnoneeabi@1.90301.200702

to my platformio.ini to tell PlatformIO to use that package. Recompilation of the project will now download the compiler (if not already present) and show

PACKAGES:
 - framework-arduino-sam 1.6.12
 - framework-cmsis 1.40500.0 (4.5.0)
 - framework-cmsis-atmel 1.2.2
 - toolchain-gccarmnoneeabi 1.90301.200702 (9.3.1)

the appropriate new compiler being used. 2 simple lines in conjuction with info from the registry.

Let’s say the PlatformIO registry does not have the compiler version I want. I want to e.g. use the latest-greatest GCC 10. So, I go ahead and download gcc-arm-none-eabi-10-2020-q4-major-win32.zip for my OS from the official website and unpack it somewhere on my computer. To make it usable with PlatformIO, the unpacked folder needs a package.json. I source that from the previos package.json located on my computer per above (or download an example package from the registry) and just modify the version field, encoding the inner version (10.2.1) and date. That gives me e.g.

{
  "name": "toolchain-gccarmnoneeabi",
  "version": "1.100201.201103",
  "description": "GNU toolchain for Arm Cortex-M and Cortex-R processors",
  "keywords": [
    "toolchain",
    "build tools",
    "compiler",
    "assembler",
    "linker",
    "preprocessor",
    "arm"
  ],
  "homepage": "https://developer.arm.com/tools-and-software/open-source-software/developer-tools/gnu-toolchain/gnu-rm",
  "license": "GPL-2.0-or-later",
  "system": [
    "windows_amd64"
  ],
  "repository": {
    "type": "git",
    "url": "https://gcc.gnu.org/git/gcc.git"
  }
}

This file is put in the extract compiler package. Now I can go ahead and again use platform_packages to make use of it. Note that the source accepts all different kinds of URLs too, just like with library installs. So one possibility is using the file:// pseudo-protocol to refer to my downloaded package.

So for me, I’m using

platform_packages = 
   toolchain-gccarmnoneeabi@file://C:\Users\Max\Downloads\gcc-arm-none-eabi-10-2020-q4-major

in my platformio.ini to refer to that compiler. On the next recompile of the project, that package will be copied into the internal package folder (<home>/.platformio/packages/), verified and used.

>pio run -e due
Processing due (platform: atmelsam; board: due; framework: arduino)
-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Tool Manager: Installing file://C:\Users\Max\Downloads\gcc-arm-none-eabi-10-2020-q4-major
Tool Manager: toolchain-gccarmnoneeabi @ 1.100201.201103 has been installed!
Verbose mode can be enabled via `-v, --verbose` option
CONFIGURATION: https://docs.platformio.org/page/boards/atmelsam/due.html
PLATFORM: Atmel SAM (6.2.0) > Arduino Due (Programming Port)
HARDWARE: AT91SAM3X8E 84MHz, 96KB RAM, 512KB Flash
DEBUG: Current (atmel-ice) External (atmel-ice, blackmagic, jlink, stlink)
PACKAGES:
 - framework-arduino-sam 1.6.12
 - framework-cmsis 1.40500.0 (4.5.0)
 - framework-cmsis-atmel 1.2.2
 - toolchain-gccarmnoneeabi 1.100201.201103 (10.2.1)
LDF: Library Dependency Finder -> http://bit.ly/configure-pio-ldf
..
RAM:   [          ]   2.6% (used 2544 bytes from 98304 bytes)
Flash: [          ]   2.0% (used 10508 bytes from 524288 bytes)
Building .pio\build\due\firmware.bin
============== [SUCCESS] Took 50.56 seconds ==============

Of course, to make it better repeatable, one could use the command line tools associated with packages, specifically pio package pack to create a compressed toolchain-gccarmnoneeabi-<version>.tar.gz file, which can then be uploaded wherever (e.g., on a webserver or git) and then referred to via that URL.

Many Arduino-specific frameworks e.g. already have a package.json file in them, e.g. Arduino-ESP32. So to get the laster master branch version of that, it’s as simple as saying

platform_packages = 
   framework-arduinoespressif32@https://github.com/espressif/arduino-esp32.git

in the platformio.ini, additionally with #branch at the end of the URL. PlatformIO recognizes this as a git url and will invoke git to clone the repo and start the installation procedure.

Some others don’t have a package.json. For these, an easy way to use them would be to fork them and add the package.json, then refer to the forked repo.

Note however that changing the compiler should be done with care, most frameworks or Arduino cores are written with only the compiler in mind that they’re using in the Arduino IDE, and other compilers might break things.

Notes on SCons

Scons is the build system, implemented in Python, on which PlatformIO is based. One could say that PIO is very advanced extension on top of SCons. In order to understand the PlatformIO build process, we need to understand a little bot of SCons. Specifically, PlatformIO uses SCons 4.1 per currently per this.

Scons can also be used a standalone tool. For a project, a small Python script can be written which conveys to the SCons build system how the source code for the project shall be built. I recommend reading through this mini introduction. The full documentation is at https://scons.org/doc/4.1.0/HTML/scons-design/ and https://scons.org/doc/4.1.0/HTML/scons-user.html.

As a short info to kinda be on the same level, some take away infos are:

  • SCons always thinks in the context of a construction environment, (Python class Environment), see docs. To construct on, in pure SCons, one would do env = Environment().
  • SCons has classes for Program, Library, SharedLibrary to control what the build output is
  • compiler options are modified through special variables in the environment object, the so called Construction Variables (Appendix A of above scons-user doc). These are accessed via e.g. env["CC"], as a Python dictionary, or more conveniently with env.Append() and env.Replace()
  • Important options to know would e.g. be
    • env["CC"] is the C compiler executable
    • env["CXX"] is the C++ compiler executable
    • same or AS (assembler), AR (archiver), etc.
    • env["CXXFLAGS"] for options affecting only the C++ compiler
    • env["CFLAGS"] for options affecting only the C compiler
    • env["CCFLAGS"] for options affecting both the C and C++ compiler
    • env["CPPPATH"] for specifying include directories
    • env["CPPDEFINES"] for storing a key-value (or just value) list for -D defines, applying to eveyrthing using the C preprocessor (thus also C and C++ compiler invocations)
    • etc., etc…

So a simple SCons script to build a program hello.c with some options would e.g. look like

# construct a environment with underlying compiler MinGW GCC
# (would otherwise use microsoft visual c compiler on my Windows system)
env = Environment(tools = ['mingw'])
# add GCC-specific compiler info and a macro value
env.Append(CCFLAGS= ["-march=native"])
env.Append(CPPDEFINES= [("MY_VALUE", 1)])
env.Program('hello.c')

save in a new directory as build.py and then with the hello.c code

#include <stdio.h>
int main() {
  printf("MY_VALUE is %d\n", MY_VALUE);
  return 0;
}

I e.g. get

>C:\Users\Max\.platformio\packages\tool-scons\scons.bat --sconstruct=build.py -Q
gcc -o hello.o -c -march=native -DMY_VALUE=1 hello.c
gcc -o hello.exe hello.o
>hello.exe
MY_VALUE is 1

So, SCons works pretty nicely and has used the compiler options given to it to produce the hello.exe program.

Platform

Used packages, platform.json

Which exact packages are used and of which version they are used depend on platform code and manifest. A platform is e.g. platform-atmelsam, for all Atmel SAM type microcontrollers. It contains a platform.json which declares the packages to be pulled from the PlatformIO trusted registry (or some other source), e.g. the compiler.

Note however that with scripting, the to-be-used package can also be exchanged dynamically. See e.g. this code in which the compiler package version is adapted for a specific framework.

Platforms are stored locally in <user directory>/.platformio/platforms.

The platform.json also stores a mapping of the possible framework values (e.g. arduino, mbed, …) to the SCons builder script needed to build them, see e.g. here.

Board definitions, boards/

A board definition (official doc) is a JSON file from which the PlatformIO core and the builder scripts will pull information in the build process. They are stored in the boards/ folder of the platform (example for atmelsam). The file names in there (e.g. adafruit_feather_m0.json) are example the possible usable board = xyz value that can be used in the platformio.ini, without the .json extension. Note that per above documentation, a project can also have boards/ directory which will be additionally searched for by PlatformIO, so projects can have needed custom board definitions directly in the project folder.

The board definition file has a few sections, most notablly build for build information, debug for information needed for debug probes, frameworks as the list of supported framework = .. values, name and upload for configuration of upload tools.

How these values are used depends on the code in the PlatformIO core and the platform code. For example, using the Python code board = env.BoardConfig() to get the board config and then board.get(), these values can be extracted and used. An example for that is here, where the code confingures the C compiler flag -mcpu= in accordance to what is stored in the "cpu" attribute of the "build" section.

platform.py

This script is optional (default behavior is applied then), but many platforms make use of it.

This Python script is loaded by the core when it encounters that a project is using a certain platform = xyz value. So e.g. for a project using platform = atmelsam, PlatformIO will know to download the atmelsam platform from the registry if no present, and then run the <home>/.platformio/platforms/atmelsam/platform.py script.

The task of the platform.py script in general is to expose a class (in this case AtmelsamPlatform) that derives from the PlatformIO core’s PlatformBase class while implementing certain functions. These functions get called from the core, whcich are

The platform.py script for AtmelSAM implements these functions, along with some helper functions. Basically:

  • configure_default_packages() checks the board information for which Arduino core implememtation is used (switchable by build.core) and activates that specific package (in the form of framework-arduino-<core name>), it sets the correct compiler versions that each individual Arduino core needs, activates the packages for specific uploader programs needed for the boards and disables the unused packages. This is necessary because the Atmel SAM supports a large set of frameworks and boards, and for each them the right package must be used.
  • get_boards() just calls into PlatformBase.get_boards() and then calls the _add_default_debug_tools() helper function which adds board-specific debug server information to each board object. The function has the task of returning an array of all boards supported by the platform, or a specific board if the id parameter is set.
  • _add_default_debug_tools() is a function setting up the debug["tools"] array of the board information object with information on how to start or use a certain debug tool with that board. By default, for every Atmel SAM board, the ability to debug (and upload) via the methods "blackmagic", "jlink", "atmel-ice", "cmsis-dap", "stlink" as added. You can see e.g. in the JLink exaple that a dictionary debug["tools"]["jlink"] is set up, with information on what the package is in which the debug tool is located (tool-jlink), what the main executable is (executable, JLinkGDBServerCL.exe), and with what arguments to call this executable (arguments), made possible with information pulled from the board manifest. See e.g. how "-device", debug.get("jlink_device")) is added as one argument, with the jlink_device information coming from the board manifest
  • configure_debug_options() is used to allow a slight reconfiguration of the debug options with info pulled from dynamic changes to the board manifest, as one can do via the platformio.ini, or info from the platformio.ini configuration in general. Here e.g. for adding the -speed <speed> option for JLink, pulled from here.

builder/main.py

This script must be there. It is invoked by the core as the first main SCons build script.

The builder/main.py for Atmel SAM is quite typical of all these scripts. A rough overview of the important steps: It

  1. Obtains the variables for the SCons enviroment (env), the PlatformIO platform object (platform) and the board here
  2. Sets up the to-be-used compiler executable here
  3. Sets up a construct of “builders”, that is converter commands that e.g. convert from the final elf file to a .bin or a .hex file, using objcopy here.
  4. If no framework = .. line is specified, invoke the _bare.py builder program here. More about framework builder scripts below.
  5. Declares the main firmware build target (${PROGNAME}.elf) here
  6. Sets up uploaders. This is distinct from debuggers. Uploaders only have the ability to upload a program (think e.g., upload via bossac or avrdude) but cannot always debug a firmware. On the reverse, a debugger tool can always be used to upload a program (think e.g., openocd can be used to establish a debug connection to the board and push a firmware on it). Specifically, it sets up the env variables UPLOADER, UPLOADERFLAGS and UPLOADCMD which will later be used by PlatformIO to invoke the uploader program. Done here.

Framework builder script (e.g. frameworks/arduino.py )

After the builder/main.py, if a framework = .. has been declared in the project’s platformio.ini., the builder file corresponding to it (set via the platform.json) is invoked.

So for Arduino projects, this will e.g. be builder/frameworks/arduino.py and related.

The main task of that file is to:

  • figure out which actual builder script to use (remember:different boards may use different Arduino core implementations, and each might need a different builder script) and invoke it in-line, done here
  • set up the SCons build environment variables regarding compiler flags, linker flags, include directories, etc, e.g. here and here
  • setup linker script
  • tell SCons which source files to build, e.g. the board-specific variant code and the actual Arduino core code

The actual implementation of that can e.g. be very different from framework to framework. In some cases, this is even chained with other build system, e.g., the builder script for the ESP-IDF framework calls into ESP-IDF’s native CMake build system to extract out the build commands. For mbed-os builds, Python functionality from the mbed-cli build system and mbed-os is used. For all Arduino cores though, a builder script is written that mimicks what the Arduino IDE would do based on the compiler options found in the platform.txt file of the Arduino core (example). That’s of course the only way to stay compatible with th Arduino IDE, by having the same compiler settings and used compiler, for a given core version.

Switching platform versions and their effect

Above we could see all the things contained in the platform – board definitions, platform script and builder scripts to build stuff, and the platform.json for used packages declaration. So, using one particular version of the platform will freeze all of that. At e.g. https://github.com/platformio/platform-atmelsam/releases we can see the released versions and what changed in them. We can use these version numbers in a platform = xyz@<version> version expression in the platformio.ini to make PlatformIO use that specific version. This is very useful for freezing a project’s used versions and thus ensuring a developed firmware works, even if underlying, platform and package updates are released.

Let’s e.g. say that my project needs the Adafruit Arduino core version 1.7.2, introduced per above in platform version 6.1.0, and the project must not use the newer 1.8.3 core version, introduced with platform version 6.2.0. Then I can write

platform = atmelsam@6.1.0

to freeze the used platform. This includes all of the above, also the package.json which declares the used package version. So with this, the project will use the platform of exactly that version, all the packages it declared at that version, board definition files, builder scripts etc.

There may be special cases in which wants platform updates (for e.g. build script fixes or features) but a lower / different package version for something (e.g. the Arduino core). Then a platform_packages declaration should be used per above.

The PlatformIO core

The core lives at https://github.com/platformio/platformio-core and implements the pio commandline tool as well as all the base classes for everything else (platforms, packages, etc…). If one is interested one can read a bit through its code, I’ve already linked to important places in the core where it interacts with a platform above. Other notable parts are e.g. this where the project is basically built, in accordance with platformio.ini options like build_flags (feed into SCons as env.ProcessFlags(env.get("BUILD_FLAGS"))), build_unflags etc.

2 Likes