Custom framework + custom board? I get a python assertion error

PlatformIO Core
, ,
1

Update

I found what’s causing this (sorta)

In [platform]/espressif32/builder/frameworks/espidf.py function get_idf_venv_dir, the call to get_original_version fails because the version is '3.50201.0+sha.8ef99ab. It has the +sha... tacked onto the end, and get_original_version won’t parse it because it has 3 dots (.) in it:

def get_original_version(version):
    if version.count(".") != 2:
        return None
    _, raw = version.split(".")[:2]
    if int(raw) <= 99:
        return None
    if int(raw) <= 9999:
        return "%s.%s" % (raw[:-2], int(raw[-2:]))
    return "%s.%s.%s" % (raw[:-4], int(raw[-4:-2]), int(raw[-2:]))

Update 2

The version data is ultimately coming from .../packages/framework-espidf/.git/.piopm But I cannot figure out what writes this file. I did a search for the string .piopm in the ~/.platformio directory – got a few hits, but nothing related to what creates this file.

Original post:

I have a custom framework (based on ESP32 IDF) being pulled from a repository, named esp-idf-ssr. It’s defined in my platformio.ini file (see below). I also have a custom board located in boards/tc.json in the top level of my project.

The tc board does show up for the espressif32 platform when you issue pio boards

The tc.json file is identical as the esp32dev board, except I added the custom framework.

The Goal

Fix this bug: I2C driver bug (IDFGH-2966) · Issue #4999 · espressif/esp-idf · GitHub

The potential fix

Use a customized esp-idf that has this fix

The problems:

Custom framework with stock board:

When using espidf-ssr framework with esp32dev board:

Error: This board doesn't support espidf-ssr framework!

Custom framework with custom board:

When using espidf-ssr framework with tc board:

Processing TC (platform: espressif32 @ 6.5.0; framework: espidf-ssr; board: tc)
-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Verbose mode can be enabled via `-v, --verbose` option
CONFIGURATION: https://docs.platformio.org/page/boards/espressif32/tc.html
PLATFORM: Espressif 32 (6.5.0) > Espressif ESP32 Dev Module
HARDWARE: ESP32 240MHz, 320KB RAM, 4MB Flash
DEBUG: Current (cmsis-dap) External (cmsis-dap, esp-bridge, esp-prog, iot-bus-jtag, jlink, minimodule, olimex-arm-usb-ocd, olimex-arm-usb-ocd-h, olimex-arm-usb-tiny-h, olimex-jtag-tiny, tumpa)
PACKAGES: 
 - framework-espidf-ssr @ 3.50201.0+sha.70c9b04 
 - tool-esptoolpy @ 1.40501.0 (4.5.1) 
 - toolchain-xtensa-esp32 @ 8.4.0+2021r2-patch5
AssertionError: :
  File "/home/epoulsen/.platformio/penv/lib/python3.10/site-packages/platformio/builder/main.py", line 173:
    env.SConscript("$BUILD_SCRIPT")
  File "/home/epoulsen/.platformio/packages/tool-scons/scons-local-4.7.0/SCons/Script/SConscript.py", line 612:
    return _SConscript(self.fs, *files, **subst_kw)
  File "/home/epoulsen/.platformio/packages/tool-scons/scons-local-4.7.0/SCons/Script/SConscript.py", line 279:
    exec(compile(scriptdata, scriptname, 'exec'), call_stack[-1].globals)
  File "/home/epoulsen/.platformio/platforms/espressif32/builder/main.py", line 346:
    target_elf = env.BuildProgram()
  File "/home/epoulsen/.platformio/packages/tool-scons/scons-local-4.7.0/SCons/Util/envs.py", line 251:
    return self.method(*nargs, **kwargs)
  File "/home/epoulsen/.platformio/penv/lib/python3.10/site-packages/platformio/builder/tools/piobuild.py", line 61:
    env.ProcessProgramDeps()
  File "/home/epoulsen/.platformio/packages/tool-scons/scons-local-4.7.0/SCons/Util/envs.py", line 251:
    return self.method(*nargs, **kwargs)
  File "/home/epoulsen/.platformio/penv/lib/python3.10/site-packages/platformio/builder/tools/piobuild.py", line 121:
    env.BuildFrameworks(env.get("PIOFRAMEWORK"))
  File "/home/epoulsen/.platformio/packages/tool-scons/scons-local-4.7.0/SCons/Util/envs.py", line 251:
    return self.method(*nargs, **kwargs)
  File "/home/epoulsen/.platformio/penv/lib/python3.10/site-packages/platformio/builder/tools/piobuild.py", line 347:
    SConscript(env.GetFrameworkScript(name), exports="env")
  File "/home/epoulsen/.platformio/packages/tool-scons/scons-local-4.7.0/SCons/Util/envs.py", line 251:
    return self.method(*nargs, **kwargs)
  File "/home/epoulsen/.platformio/penv/lib/python3.10/site-packages/platformio/builder/tools/pioplatform.py", line 58:
    assert p.frameworks and framework in p.frameworks
============================================================================== [FAILED] Took 0.15 seconds ==============================================================================

Environment    Status    Duration
-------------  --------  ------------
TC             FAILED    00:00:00.149
========================================================================= 1 failed, 0 succeeded in 00:00:00.149 =========================================================================

Stock framework, but override framework-espidf with custom URL:

platformio.ini snippets:

platform = espressif32 @ 6.5.0
platform_packages = framework-espidf@ <redacted>/esp-idf-ssr.git#v5.2.1-ssr
framework = espidf
board = esp32dev

Here, I’m simply trying to override the existing framework-espidf for use with the esp32dev board. I get:

TypeError: can only concatenate str (not "NoneType") to str:
  File "/home/epoulsen/.platformio/penv/lib/python3.10/site-packages/platformio/builder/main.py", line 173:
    env.SConscript("$BUILD_SCRIPT")
  File "/home/epoulsen/.platformio/packages/tool-scons/scons-local-4.7.0/SCons/Script/SConscript.py", line 612:
    return _SConscript(self.fs, *files, **subst_kw)
  File "/home/epoulsen/.platformio/packages/tool-scons/scons-local-4.7.0/SCons/Script/SConscript.py", line 279:
    exec(compile(scriptdata, scriptname, 'exec'), call_stack[-1].globals)
  File "/home/epoulsen/.platformio/platforms/espressif32/builder/main.py", line 346:
    target_elf = env.BuildProgram()
  File "/home/epoulsen/.platformio/packages/tool-scons/scons-local-4.7.0/SCons/Util/envs.py", line 251:
    return self.method(*nargs, **kwargs)
  File "/home/epoulsen/.platformio/penv/lib/python3.10/site-packages/platformio/builder/tools/piobuild.py", line 61:
    env.ProcessProgramDeps()
  File "/home/epoulsen/.platformio/packages/tool-scons/scons-local-4.7.0/SCons/Util/envs.py", line 251:
    return self.method(*nargs, **kwargs)
  File "/home/epoulsen/.platformio/penv/lib/python3.10/site-packages/platformio/builder/tools/piobuild.py", line 121:
    env.BuildFrameworks(env.get("PIOFRAMEWORK"))
  File "/home/epoulsen/.platformio/packages/tool-scons/scons-local-4.7.0/SCons/Util/envs.py", line 251:
    return self.method(*nargs, **kwargs)
  File "/home/epoulsen/.platformio/penv/lib/python3.10/site-packages/platformio/builder/tools/piobuild.py", line 347:
    SConscript(env.GetFrameworkScript(name), exports="env")
  File "/home/epoulsen/.platformio/packages/tool-scons/scons-local-4.7.0/SCons/Script/SConscript.py", line 676:
    return method(*args, **kw)
  File "/home/epoulsen/.platformio/packages/tool-scons/scons-local-4.7.0/SCons/Script/SConscript.py", line 612:
    return _SConscript(self.fs, *files, **subst_kw)
  File "/home/epoulsen/.platformio/packages/tool-scons/scons-local-4.7.0/SCons/Script/SConscript.py", line 279:
    exec(compile(scriptdata, scriptname, 'exec'), call_stack[-1].globals)
  File "/home/epoulsen/.platformio/platforms/espressif32/builder/frameworks/espidf.py", line 1251:
    ensure_python_venv_available()
  File "/home/epoulsen/.platformio/platforms/espressif32/builder/frameworks/espidf.py", line 1224:
    venv_dir = get_idf_venv_dir()
  File "/home/epoulsen/.platformio/platforms/espressif32/builder/frameworks/espidf.py", line 1178:
    env.subst("$PROJECT_CORE_DIR"), "penv", ".espidf-" + idf_version

The question:

How can I use my custom framework? I only added the custom board thinking I needed to do that to avoid the This board doesn't support espidf-ssr framework! error

Contents of my config files

platformio.ini

[env] ## Defaults
monitor_speed = 115200
platform = espressif32 @ 6.5.0
platform_packages = 
    framework-espidf-ssr @ git@<redacted>/esp-idf-ssr.git#v5.2.1-ssr
framework = espidf-ssr

board = tc

boards/tc.json

{
    "build": {
      "arduino":{
        "ldscript": "esp32_out.ld"
      },
      "core": "esp32",
      "extra_flags": "-DARDUINO_ESP32_DEV",
      "f_cpu": "240000000L",
      "f_flash": "40000000L",
      "flash_mode": "dio",
      "mcu": "esp32",
      "variant": "esp32"
    },
    "connectivity": [
      "wifi",
      "bluetooth",
      "ethernet",
      "can"
    ],
    "debug": {
      "openocd_board": "esp-wroom-32.cfg"
    },
    "frameworks": [
      "arduino",
      "espidf",
      "espidf-ssr"
    ],
    "name": "Espressif ESP32 Dev Module",
    "upload": {
      "flash_size": "4MB",
      "maximum_ram_size": 327680,
      "maximum_size": 4194304,
      "require_upload_port": true,
      "speed": 460800
    },
    "url": "https://en.wikipedia.org/wiki/ESP32",
    "vendor": "Espressif"
  }
2

@maxgerhardt Can you comment on what creates the .piopm file?

3

@ivankravets Any ideas?

4

ESP-IDF is a really, really complicated framework. The builder script for it is no joke and taps into many internals of CMake and SCons and PlatformIO. So if people attempt to “add supported for a framework based on ESP-IDF” by themselves and they run into errors and it’s not trivial, it’s in the “good luck” terroritory.

I don’t understand at all. If all you want is a ESP-IDF version that PlatformIO already supports that has one teeny tiny patch in it for I2C, you go the way of just creating a new version of the framework-espidf package and then reference it with platform_packages = <link to the ZIP of your forked package>. The only caveats are

  1. It must have a package.json in that package as seen in C:\users\<user>\.platformio\packages\framework-espidf
  2. You should not pull it via the git protocol because then you’ll end up with this +shaXXXXX in the version field which the builder script fails to decode

You do not need (and want) to invent a new framework type for that.

Another possibility is to also add a per project script that dynamically patches the buggy code or redirects the compilation into a user provided, fixed file. There are official examples for this.

https://docs.platformio.org/en/latest/scripting/examples/override_package_files.html

5

Max,

Thank you for the detailed response, I’ll try the zip file.

But I have to ask about the “do not use git” because it was your post that indicated that that was the way to go about it.

Additionally, the .git/.piopm file is clearly created by PIO itself, which then fails to parse it correctly. What’s troubling is that I can’t even figure out what mechanism writes this file in the first place.

6

Git is usable to replace almost all platform packages, except framework-espidf because the builder script tries to decode the version in a kind of broken way. You’re hitting the one edge case.

7

Great, I’m an edge case. LOL. Thanks for the info.

8

This didn’t work.

I have a zipfile that contains all of the ESP-IDF (and submodules that were downloaded), with this package.json (which is identical to the one installed in the ~/.platformio dir:

{
  "name": "framework-espidf",
  "version": "3.50102.240122",
  "title": "Espressif IoT Development Framework",
  "description": "Espressif IoT Development Framework. Official development framework for ESP32 chip",
  "keywords": [
    "framework",
    "esp32",
    "esp32s2",
    "espressif"
  ],
  "homepage": "https://docs.espressif.com/projects/esp-idf/en/latest/esp32/",
  "license": "Apache-2.0",
  "repository": {
    "type": "git",
    "url": "https://github.com/espressif/esp-idf"
  }
}

The result is:

--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Tool Manager: Installing esp-idf-ssr.zip @ *
UnknownPackageError: Could not find the package with 'esp-idf-ssr.zip @ *' requirements for your system 'linux_x86_64'

Relevant platformio.ini

board = esp32dev
framework = espidf
platform = espressif32
platform_packages = esp-idf-ssr.zip

Directory structure inside zipfile:

unzip -t esp-idf-ssr.zip| head
Archive:  esp-idf-ssr.zip
    testing: add_path.sh              OK
    testing: CMakeLists.txt           OK
    testing: COMPATIBILITY_CN.md      OK
    testing: COMPATIBILITY.md         OK
    testing: components/              OK
    testing: components/esp-tls/      OK
    testing: components/esp-tls/esp_tls.c   OK
    testing: components/esp-tls/esp_tls_errors.h   OK
    testing: components/esp-tls/CMakeLists.txt   OK
    [etc]

package.json is found therein.

I also tried a tar file, and I tried using an absolute path to the zipfile – no difference.

platform_packages = /home/epoulsen/workspaces/TC/esp-idf-ssr.zip@3.50201.240515

resulted in

Processing TC (board: esp32dev; framework: espidf; platform: espressif32)
----------------------------------------------------------------------------------------------------
Tool Manager: Installing esp-idf-ssr @ 3.50201.240515
UnknownPackageError: Could not find the package with 'esp-idf-ssr @ 3.50201.240515' requirements for your system 'linux_x86_64'

Nor does it work if I point it to a directory with the zipfile contents.

9

No, you have to tell what framework that replaces.

platform_packages = 
   framework-espidf@file://esp-idf-ssr.zip

https://docs.platformio.org/en/latest/projectconf/sections/env/options/platform/platform_packages.html

1 Like
10

@maxgerhardt

Where can I find the repository for framework-espidf?

My version, which is ESP-IDF 5.2.1 crashes with Cache disabled but cached memory region accessed for reasons I cannot fathom, but works with the official framework-espidf

I’d like to see if there are other changes?

Diff shows that version.txt in the PIO dir is 5.2.1 and for mine it is 5.1.2 – this file does not exist in my zipfile.

11

So sounds like https://github.com/platformio/platform-espressif32/issues/1360 which was solved by a toolchain upgrade? Did you try using the injected package with platform = espressif32@6.7.0?

12

6.7.0 does the same thing, but I tried 6.6.0, and it seems to be working now.

Once again, thank you!