Adding a new framework to Platformio

FAQ
1

I am working on adding the SmingRTOS Framework to platformio.

SmingRTOS is a framework for Espressif ESP8266, ESP32 is planned.
Based on Espressif RTOS SDK, currently version 1.4.0

If possible I like to do the integration simultaneous for platformio 2.9 & 3.0

Until now looking at several other framework integration…
Planning to work from bottom up :

  • define frameworks and dependent package
  • create build script when the above is defined correct.
  • working with platformio cli first
    Current issues/questions.

Tox :
To be able to run the tox - develop I had to “hack” the tox.ini because of Interpreter not found python2.7 message
Updated the
[testenv:develop]
basepython = python2.7

to

[testenv:develop]
basepython = python

And this command now worked → was able to create an environment which uses my local platformio repo.

Question : What is the right way to solve ?

Started to add framework definitions in : espressif.py & base.py
Running “platformio platforms install espressif --with-package sming-framework”
gives : “Error: Detected unknown package ‘sming-framework’”

Question : where and how do I define sming-framework & contents of the package.

Regards,
Herman

2 
platformio platform install espressif

Development platform will be installed to ~/.platformio/platforms/espressif

  "frameworks": {
    "arduino": {
      "package": "framework-arduinoespressif",
      "script": "builder/frameworks/arduino.py"
    },
    "simba": {
      "package": "framework-simba",
      "script": "builder/frameworks/simba.py"
    },
    "sming": {
      "package": "framework-sming",
      "script": "builder/frameworks/sming.py"
    }
  },
  "packages": {
    "framework-sming": {
      "type": "framework",
      "optional": true,
      "version": "~1.1.0"
    }
}
  • Create folder ~/.platformio/packages/framework-sming and palce here source code of the framework.
  • Add package.json to ~/.platformio/packages/framework-sming folder with the next contents:
{
    "description": "Sming Framework for Espressif",
    "name": "framework-sming",
    "system": "*",
    "url": "...",
    "version": "1.1.0"
}

  "frameworks": [
    "arduino", 
    "simba",
    "sming"
  ],
  • Build project via platformio run 
[env:smingtest]
platform = espressif
framework = sming
board = esp01
3

Hi,
Some probably basic questions, learning on the job.

  • SDK / packages
    There is the package sdk-esp8266 but I see also sdk files within the framework arduino.
    As SmingRTOS is based on the ESP RTOS SDK I created a package sdk-esp8266-RTOS.
    But how would you handle version updates when one framework uses a newer version of the SDK ?

  • Compiler options
    I want/need to use other CFLAGS & CCFLAGS than defined within platforms\espressif\builder\main.py
    Trying to overrule those in sming.py doesn’t seem to work.
    Can you give a hint on how to proceed ?

  • Project init
    I don’t see the option to define the framework when doing a platformio init.
    Can change the plarformio.ini file afterwards which works for cli projects.
    But when having an eclipse project, all “include setting” within eclipse stay on the “initial framework”

  • Linux/windows/mac in packages
    Probably need to add a tools package with programs which are platform dependent.
    How would I add these to the package.

  • platform/framework definitions.
    Not sure yet, so need further thoughts from me but :
    Looks like there are definitions within the espressif platform which are more or less framework dependent.
    What is the basic idea between framework and platform separation ?

More to come…

4

Arduino Framework for ESP8266 uses own SDK and keeps it within own source files.

You can place own version of SDK to framework-sming package.

You need


env.Replace(
    CFLAGS=[..],
    CCFLAGS=[...]
)

See SCons docs: 7.2. Construction Environments
Also, take a look at 7.2.8-7.2.13 sections SCons 4.6.0

Sorry, we don’t have this option yet :pensive:
Nevertheless, you can edit platformio.ini with required framework and repeat pio init --ide eclipse -b ***. In this case, PlatformIO will regenerate project with the new framework.

  • You can create own Package Repository Manifest file and define own packages. See current package repositories:
  • You can place own files to existing package repositories (ask us here on the forum).

About which definitions do you mean? The framework is a part of a development platform. However, we have a case where the one framework is used in the many development platforms. Take a look on these development platforms:

and common frameworks:

5

Thanks, helping a lot and getting forward.
Learning three items simultaneous (python, cons, platformio) sometimes gives trouble.

On platform/framework definitions
I see in espressif platform main.py among others :

FRAMEWORK_ARDUINOESP8266_DIR=platform.get_package_dir(
“framework-arduinoespressif”),
SDK_ESP8266_DIR=platform.get_package_dir(“sdk-esp8266”),

#
# Upload
#

UPLOADER="esptool",
UPLOADEROTA=join("$FRAMEWORK_ARDUINOESP8266_DIR", "tools", "espota.py"),

and

‘“%s”’ % join(“$FRAMEWORK_ARDUINOESP8266_DIR”,
“bootloaders”, “eboot”, “eboot.elf”),

Which are specific for a framework.

And related :

Configure native SDK

Where I would expect native SDK programming also be a framework.

Question on this :
I see in the compile ouput :

xtensa-lx106-elf-ranlib .pioenvs\nodemcuv2\libFrameworkArduinoVariant.a
xtensa-lx106-elf-ranlib .pioenvs\nodemcuv2\libFrameworkArduino.a

Can you point me to the location where this is included and/or defined ?

Another question I have which probably related to scons but maybe you can give me the starting point/example to use in sming.py

After compiling and before uploading I need to run another program, call it for now “SmingProg inputfile”
I understand that I have to create a builder and prepend/append it to the env.
But I cannot get that to work, any hints for an example or direction to look ?

6

These environment variables will be used only for Arduino Framework. SCons allows defining multiple variables for build environment and allows to use on demand. In any case, please ignore these Arduino’s things in a main script. We will cleanup it later.

These magics lines talk PlatformIO Build System to build all source files from core directory and make static library. See

The PlatformIO System Build API of env.BuildLibrary is the next

env.BuildLibrary(out_dir, src_dir)

That means, that PlatformIO Build System will find all C/C++/ASM compatible files in src_dir, compile them to out_dir, archive to a static library and link to the final firmware.

See source code:
https://github.com/platformio/platformio/blob/feature/platformio-30/platformio/builder/tools/platformio.py#L236

@valeros will prepare something for you.

7

Thanks, getting a better understanding.

For @valeros :
No need for example now, also learning scons better.

8

Hi,
When updating/testing the sming.py I regular like to understand scons results.

Question : How can I add command line options to the scons process.
As an example “scons --debug=prepare” or “scons --tree=all”

9

You need

export SCONSFLAGS=" --debug=prepare"
pio run ...
10

Thanks, works for getting scons options.

FYI :
When using SCONSFLAGS= --tree=all → works OK
When using SCONSFLAGS= --tree=status I get the following error :
…
…
scons: `.pioenvs\nodemcuv2\firmware.bin’ is up to date.
E = exists
R = exists in repository only
b = implicit builder
B = explicit builder
S = side effect
P = precious
A = always build
C = current
N = no clean
H = no cache

[E B C ]±.pioenvs\nodemcuv2\firmware.bin
[E B C ] ±.pioenvs\nodemcuv2\firmware.elf
[E B C ] | ±.pioenvs\nodemcuv2\src\application.o
AttributeError: ‘NoneType’ object has no attribute ‘get_contents’:
File “c:\userdata\piodata\packages\tool-scons\script..\engine\SCons\Script\Main.py”, line 1346:
_exec_main(parser, values)
File “c:\userdata\piodata\packages\tool-scons\script..\engine\SCons\Script\Main.py”, line 1309:
_main(parser)
File “c:\userdata\piodata\packages\tool-scons\script..\engine\SCons\Script\Main.py”, line 1091:
nodes = build_targets(fs, options, targets, target_top)
File “c:\userdata\piodata\packages\tool-scons\script..\engine\SCons\Script\Main.py”, line 1283:
jobs.run(postfunc = jobs_postfunc)
File “c:\userdata\piodata\packages\tool-scons\script..\engine\SCons\Job.py”, line 111:
self.job.start()
File “c:\userdata\piodata\packages\tool-scons\script..\engine\SCons\Job.py”, line 399:
task.postprocess()
File “c:\userdata\piodata\packages\tool-scons\script..\engine\SCons\Script\Main.py”, line 288:
tp.display(t)
File “c:\userdata\piodata\packages\tool-scons\script..\engine\SCons\Script\Main.py”, line 419:
SCons.Util.print_tree(t, func, prune=self.prune, showtags=s)
File “c:\userdata\piodata\packages\tool-scons\script..\engine\SCons\Util.py”, line 286:
print_tree(C, child_func, prune, idx, margin, visited)
File “c:\userdata\piodata\packages\tool-scons\script..\engine\SCons\Util.py”, line 286:
print_tree(C, child_func, prune, idx, margin, visited)
File “c:\userdata\piodata\packages\tool-scons\script..\engine\SCons\Util.py”, line 286:
print_tree(C, child_func, prune, idx, margin, visited)
File “c:\userdata\piodata\packages\tool-scons\script..\engine\SCons\Util.py”, line 260:
tags.append(’ C’[IDX(root.is_up_to_date())])
File “c:\userdata\piodata\packages\tool-scons\script..\engine\SCons\Node\FS.py”, line 3286:
if not self.changed(r):
File “c:\userdata\piodata\packages\tool-scons\script..\engine\SCons\Node\FS.py”, line 3241:
has_changed = SCons.Node.Node.changed(self, node)
File "c:\userdata\piodata\packages\tool-scons\script..\engine\SCons\Node_init.py", line 1462:
contents = self.get_executor().get_contents()
[ ERROR ] Took 0.79 seconds

11

Please wait for a while… @valeros prepared a “beta” version of Sming integration. He is going to create a separate branch in the repository and will explain here how to use. You can discuss here further steps about integration.

12

Ok, great.

I will continue learning platformio & scons with my current trial of sming.py.

@valeros : Don’t go to far in the beta version of sming. I take the opportunity to (re)structure the sming build process while integrating into platformio.

PS : Will be out coming week for a short holiday, don’t expect answers then :slight_smile:

13

Hi @hreintke, looks like you are using Windows machine, so here are a few steps to try beta version:

  • If you still don’t have PlatformIO 3.0 install it.
  • Install beta version of espressif development platform from Github:
pio platform install git+https://github.com/platformio/platform-espressif.git#feature/framework-sming

pio run -d C:\Users\YOUR_USER_NAME\.platformio\platforms\espressif\examples\sming-basic-wifi\

If you want to try another board with more than 512K memory, please edit main.py:161 line according to your memory size.

PS:
Can we use esptool-ck instead of esptool2 and esptool.py for splitting and uploading firmware?
Is there a dependency on the toolchain version?
Which version of the SDK is preferable?

14

Hi Vallerii,

Thanks for time and effort to start & help on the sming framework.

First remarks on the above.

That’s it for now, more to come.

15

No, see command above

pio platform install git+https://github.com/platformio/platform-espressif.git#feature/framework-sming
16

But then I get :

C:\temp>pio platform install git+https://github.com/platformio/platform-espressif.git#feature/framework-sming

If you like PlatformIO, please:

Installing platform git+https://github.com/platformio/platform-espressif.git#feature/framework-sming @ latest:
‘git’ is not recognized as an internal or external command,
operable program or batch file.

Or when trying without the “git+”

C:\temp>pio platform install GitHub - platformio/platform-espressif8266: Espressif 8266: development platform for PlatformIO
Installing platform GitHub - platformio/platform-espressif8266: Espressif 8266: development platform for PlatformIO @ latest:
Downloading…
Error: Can not unpack file ‘c:\userdata\piodata\platforms\installing-i10nbv-package\framework-sming’

17

Please install Git Git - Downloads

P.S: Users will not need to install Git, this step is required for Windows developers.

18

Ah, sorry.
Was to much focused on platformio that I missed it as a windows error message.

  • Installation
    Hope not to make another mistake like that but now I have :
    Installing platform git+https://github.com/platformio/platform-espressif.git#feature/framework-sming @ latest:
    git version 1.8.3.msysgit.0
    Cloning into ‘c:\userdata\piodata\platforms\installing-1za5ys-package’…
    remote: Counting objects: 87, done.
    remote: Compressing objects: 100% (57/57), done.
    emote: Total 87 (delta 44), reused 59 (delta 22), pack-reused 0
    Unpacking objects: 100% (87/87), done.
    Installing package tool-scons @ >=2.3.0,<2.6.0:
    Downloading [####################################] 100%
    Unpacking [####################################] 100%
    Installing package toolchain-xtensa @ ~1.40802.0:
    Downloading [####################################] 100%
    Unpacking [####################################] 100%
    Installing package tool-esptool @ ~1.408.0:
    Downloading [####################################] 100%
    Unpacking [####################################] 100%
    Installing package tool-esptoolpy @ ~1.10100.0:
    Error: Detected unknown package ‘tool-esptoolpy’

  • Tools
    Yes, as I see it now, we can change the tools Sming is using.
    There are two build processes within Sming, one “normal” and one including “rboot”
    Rboot being the OTA functionality in Sming.
    I will combine the two when integrating in platformio

  • Toolchain version.
    Most Sming users are using toolchain version 5.1.0, platformio is 4.8.2.
    Remember from a time ago that there were issues with a toolchain version but need to check what issues and which version
    What is the upgrade policy of platformio for packages/toolchains/SDK’s ?
    Especially when multiple framework are using the same package ?
    Or can there be multiple versions of the same package within platformio.

  • Integration code
    Looked globally within the beta implementation made by @valeros.
    Was surprised that the Sming specific items were split between main.py and sming.py.
    Isn’t the main.py maintained by platformio team and sming.py by sming team.
    One of the options I was thinking of is taking the approach from simba framework and just put a stub/include in plaformio/sming.py and keep the actual integration code in the sming framwork itself.
    What is your opinion on that ?

19

Installation issue solved.
Starting further testing.

Have the first small Sming application running using platformio.
Thanks for the help on that.

Will continue after return from my holiday.

20

PlatformIO 3.0 allows having different toolchains, SDK, etc per different framework, boards… I’ll explain you later how to make it.

That was the main goal of PlatformIO 3.0 - split PlatformIO 2.0 to

  • PlatformIO 3.0 Core
  • PlatformIO 3.0 compatible development platforms

You even can have own espressif-sming development platform with own packages and build scripts. However, I don’t recommend to do it in this situation. The idea to keep Sming build script in the Sming’s repository is very good! We don’t have any objections.

We are so glad to answer your questions and help you! Have a nice holiday! :sunny: