Another newbie question : how do I get the esp8266 SDK sources to compile?

Hi,
The esp8266 SDK build doesn’t work for me any more, so I try to support it with platformio. It requires building all or a subset of the components that make up the SDK.

How do I enumerate these, and the include directories in there ?

Including 100 lines like

    join(FRAMEWORK_DIR, "include"),
    join(FRAMEWORK_DIR, “components”, “app_update/include”),
    join(FRAMEWORK_DIR, “components”, “bootloader_support/include”),

doesn’t look right. I looked at the platformio-espressif32 but it appears to rely on cmake/the esp-idf to do that. I don’t find much platformio documentation at all (for a platformio user there’s stuff but I don’t find anything related to what I’m asking), is the standard approach “look for existing examples” ?

So thanks to earlier help I was able to get things going a bit but the above is meant to get me further than :

victus: {333} pio run 
Processing esp8266-d1mini (platform: esp8266rtos; framework: esp8266rtos; board: d1_mini)
------------------------------------------------------------------------------------------------------------------------
Verbose mode can be enabled via `-v, --verbose` option
CONFIGURATION: https://docs.platformio.org/page/boards/esp8266rtos/d1_mini.html
PLATFORM: Espressif 8266 - RTOS v3.4 (4.2.1) > WeMos D1 R2 and mini
HARDWARE: ESP8266 80MHz, 80KB RAM, 4MB Flash
PACKAGES: 
 - esp8266rtos @ 0.1.0 
 - tool-esptool @ 1.413.0 (4.13) 
 - tool-esptoolpy @ 1.30000.201119 (3.0.0) 
 - toolchain-xtensa @ 1.40802.0 (4.8.2)
FRAMEWORK_DIR  /home/danny/.platformio/packages/esp8266rtos
env.GetLibBuilders()  []
LDF: Library Dependency Finder -> https://bit.ly/configure-pio-ldf
LDF Modes: Finder ~ chain, Compatibility ~ soft
Found 0 compatible libraries
Scanning dependencies...
No dependencies
Building in release mode
Compiling .pio/build/esp8266-d1mini/src/main.o
Compiling .pio/build/esp8266-d1mini/src/startup.o
src/main.c:2:31: fatal error: freertos/FreeRTOS.h: No such file or directory
 #include "freertos/FreeRTOS.h"
                               ^
compilation terminated.
*** [.pio/build/esp8266-d1mini/src/main.o] Error 1
src/startup.c:22:23: fatal error: nvs_flash.h: No such file or directory

*******************************************************************
* Looking for nvs_flash.h dependency? Check our library registry!
*
* CLI  > platformio lib search "header:nvs_flash.h"
* Web  > https://registry.platformio.org/search?q=header:nvs_flash.h
*
*******************************************************************

 #include "nvs_flash.h"
                       ^
compilation terminated.
*** [.pio/build/esp8266-d1mini/src/startup.o] Error 1
============================================== [FAILED] Took 0.42 seconds ==============================================

so building the list of includes from a script, and then obviously generating the libraries to avoid the subsequent linker errors.

Thanks !

Going on a bit - I added some code I just wrote but I’m new to python.So the include string I build gets inserted between quotes, meaning it won’t work.

    CPPDEFINES=[
        ("F_CPU", "$BOARD_F_CPU"),
        "__ets__",
        "ICACHE_FLASH"
    ],

    CPPPATH= esp8266builder.build_includes(FRAMEWORK_DIR),

    LIBPATH=[
#        join(FRAMEWORK_DIR, "lib"),
#        join(FRAMEWORK_DIR, "ld")
        join(FRAMEWORK_DIR, "components/esp8266/lib"),
        join(FRAMEWORK_DIR, "components/esp-wolfssl/wolfssl/lib"),
    ],
def component_includes(c):
    r = ''
    if c.name == "freertos":
        r = r + " -I" + str(c) + "/port/esp8266/include"
    inc = c / "include"
    if inc.exists():
        r = r + " -I" + str(inc)
    privinc = c / "priv_include"
    if privinc.exists():
        r = r + " -I" + str(privinc)
    return r

def list_includes(p):
    path = Path(p)
    r = ''
    for item in path.iterdir():
        s = str(item.name)
        if item.is_dir():
            i = component_includes(item)
            r = r + i
    return r

def build_includes(p):
    # esp8266rtos/include
    r = " -I" + p + "/include"
    # Get list of component includes
    components = str(p) + "/components"
    r = r + list_includes(components)
    return r

gives :

xtensa-lx106-elf-gcc -o .pio/build/esp8266-d1mini/src/main.o -c -std=gnu99 -Wpointer-arith -Wno-implicit-function-declaration -Wl,-EL -fno-inline-functions -nostdlib -Os -mlongcalls -mtext-section-literals -falign-functions=4 -U__STRICT_ANSI__ -ffunction-sections -fdata-sections -DPLATFORMIO=60118 -DESP8266 -DARDUINO_ARCH_ESP8266 -DARDUINO_ESP8266_WEMOS_D1MINI -DF_CPU=80000000L -D__ets__ -DICACHE_FLASH -Isrc "-I/home/danny/.platformio/platforms/esp8266rtos/builder/ -I/home/danny/.platformio/packages/esp8266rtos/include -I/home/danny/.platformio/packages/esp8266rtos/components/esp_https_ota/include -I/home/danny/.platformio/packages/esp8266rtos/components/tcpip_adapter/include -I/home/danny/.platformio/packages/esp8266rtos/components/spiffs/include -I/home/danny/.platformio/packages/esp8266rtos/components/esp_common/include -I/home/danny/.platformio/packages/esp8266rtos/components/wpa_supplicant/include -I/home/danny/.platformio/packages/esp8266rtos/components/nvs_flash/include -I/home/danny/.platformio/packages/esp8266rtos/components/app_update/include -I/home/danny/.platformio/packages/esp8266rtos/components/bootloader_support/include -I/home/danny/.platformio/packages/esp8266rtos/components/http_parser/include -I/home/danny/.platformio/packages/esp8266rtos/components/spi_flash/include -I/home/danny/.platformio/packages/esp8266rtos/components/log/include -I/home/danny/.platformio/packages/esp8266rtos/components/freertos/port/esp8266/include -I/home/danny/.platformio/packages/esp8266rtos/components/freertos/include -I/home/danny/.platformio/packages/esp8266rtos/components/tcp_transport/include -I/home/danny/.platformio/packages/esp8266rtos/components/esp_http_client/include -I/home/danny/.platformio/packages/esp8266rtos/components/esp_http_server/include -I/home/danny/.platformio/packages/esp8266rtos/components/esp_gdbstub/include -I/home/danny/.platformio/packages/esp8266rtos/components/protocomm/include -I/home/danny/.platformio/packages/esp8266rtos/components/wear_levelling/include -I/home/danny/.platformio/packages/esp8266rtos/components/pthread/include -I/home/danny/.platformio/packages/esp8266rtos/components/vfs/include -I/home/danny/.platformio/packages/esp8266rtos/components/openssl/include -I/home/danny/.platformio/packages/esp8266rtos/components/wifi_provisioning/include -I/home/danny/.platformio/packages/esp8266rtos/components/heap/include -I/home/danny/.platformio/packages/esp8266rtos/components/spi_ram/include -I/home/danny/.platformio/packages/esp8266rtos/components/mdns/include -I/home/danny/.platformio/packages/esp8266rtos/components/esp_event/include -I/home/danny/.platformio/packages/esp8266rtos/components/lwip/include -I/home/danny/.platformio/packages/esp8266rtos/components/jsmn/include -I/home/danny/.platformio/packages/esp8266rtos/components/esp8266/include -I/home/danny/.platformio/packages/esp8266rtos/components/esp_ringbuf/include" src/main.c

What am I doing wrong ?

Is something looking wrong? The include paths are in the compile command so that worked.

Note that usually would write

        join(FRAMEWORK_DIR, "components", "esp-wolfssl", "wolfssl, "lib"),

so each component of the path individually.

See last piece I included (the gcc command), there are a couple of quotes that prevent the list from working. They’re at the second -I option and almost of the end of the (lengthy) line.

I can hardcode all that but that’s a list of 50+ directories that need to be included.

And after that, I have to build code to compile some 800 source files that are in the SDK, they’re needed for building applications.

CPPPATH is supposed to be an array of paths, not -I flags, make sure of that.

That changes the output a bit but the problem remains. Pasting only a bit now :

-I/home/danny/.platformio/packages/esp8266rtos/components/esp8266/include -I/home/danny/.platformio/packages/esp8266rtos/components/esp8266 "-I/home/danny/.platformio/platforms/esp8266rtos/builder/ /home/danny/.platformio/packages/esp8266rtos/include

Are you sure you’re using the proper join(path1, path2, path3, ..) syntax to concatena paths instead of a pure += "string constant"? If in doubt, print the whole list that you’re setting CPPPATH equal to and inspect it that each entry is a proper path.

Am I allowed to use a construction like

    CPPPATH=[ 
        esp8266builder.build_includes(FRAMEWORK_DIR)
    ],

if that function outputs a long sequence of join(path1, path2), sequences ?

I’ve partially changed my script and the gcc command still looks suspicious :

-I/home/danny/.platformio/packages/esp8266rtos/components/esp8266/include -I/home/danny/.platformio/packages/esp8266rtos/components/esp8266 "-I/home/danny/.platformio/platforms/esp8266rtos/builder/ join(/home/danny/.platformio/packages/esp8266rtos, include), join(/home/danny/.platformio/packages/esp8266rtos/components/esp_https_ota, include),

Looks like the library finder and my stuff don’t get along.

That looks like it literally included the string join(...) instead of calling it as a function – huh?

No if build.includes() already returns a list[str] then you need to just CPPPATH=esp8266builder.build_includes(FRAMEWORK_DIR), that was already correct.

Ok so my function should produce a comma-separated list of names ? (The names being directories containing include files ?)

No, a literal list of regular str objects, not joined by a comma or something.

include_paths = [] #empty list
for path in all_package_include_paths:
   include_paths.append(path)
return include_paths

It didn’t work with the commas so I removed those.
Added a print statement at the end.

My code :

 import os
 from pathlib import Path
 
victus: {689} svn commit 
X11 forwarding request failed on channel 0
Adding         README.md
Sending        builder/frameworks/esp8266builder.py
Transmitting file data ..done
Committing transaction...
    if c.name == 'esp8266':
        r = r + ' ' + str(c) + '/include/driver'
    if c.name == 'lwip':
        r = r + ' ' + str(c) + '/lwip/src/include'
        r = r + ' ' + str(c) + '/port/esp8266/include'
        r = r + ' ' + str(c) + '/include/apps'
        r = r + ' ' + str(c) + '/include/apps/dhcpserver'
        r = r + ' ' + str(c) + '/include/apps/sntp'
        r = r + ' ' + str(c) + '/include/apps/ping'
    if c.name == 'freertos':
        r = r + ' ' + str(c) + '/port/esp8266/include'
        r = r + ' ' + str(c) + '/port/esp8266/include/freertos'
        r = r + ' ' + str(c) + '/include/freertos'
        r = r + ' ' + str(c) + '/include/freertos/private'
    if c.name == 'heap':
        r = r + ' ' + str(c) + '/port/esp8266/include'
    # only for newlib
    platfinc = c / 'platform_include'
    if platfinc.exists():
        r = r + ' ' + str(c) + '/platform_include'
    return r

def list_includes(p):
    path = Path(p)
    r = ''
    for item in path.iterdir():
        s = str(item.name)
        if item.is_dir():
            i = component_includes(item)
            r = r + i
    return r

def build_includes(p):
    r = ''
    # Get list of component includes
    components = str(p) + "/components"
    r = r + list_includes(components)
    # esp8266rtos/include, put at the end to avoid trailing comma
    r = r + ' ' + p + '/include'
    print("build_includes -> " + r)
    return r

Its output :

Processing esp8266-d1mini (platform: esp8266rtos; framework: esp8266rtos; board: d1_mini)
--------------------------------------------------------------------------------
CONFIGURATION: https://docs.platformio.org/page/boards/esp8266rtos/d1_mini.html
PLATFORM: Espressif 8266 - RTOS v3.4 (4.2.1) > WeMos D1 R2 and mini
HARDWARE: ESP8266 80MHz, 80KB RAM, 4MB Flash
PACKAGES:
 - esp8266rtos @ 0.1.0
 - tool-esptool @ 1.413.0 (4.13)
 - tool-esptoolpy @ 1.30000.201119 (3.0.0)
 - toolchain-xtensa @ 1.40802.0 (4.8.2)
build_includes ->  /home/danny/.platformio/packages/esp8266rtos/components/esp_https_ota/include /home/danny/.platformio/packages/esp8266rtos/components/tcpip_adapter/include /home/danny/.platformio/packages/esp8266rtos/components/spiffs/include /home/danny/.platformio/packages/esp8266rtos/components/esp_common/include /home/danny/.platformio/packages/esp8266rtos/components/wpa_supplicant/include /home/danny/.platformio/packages/esp8266rtos/components/nvs_flash/include /home/danny/.platformio/packages/esp8266rtos/components/app_update/include /home/danny/.platformio/packages/esp8266rtos/components/bootloader_support/include /home/danny/.platformio/packages/esp8266rtos/components/http_parser/include /home/danny/.platformio/packages/esp8266rtos/components/spi_flash/include /home/danny/.platformio/packages/esp8266rtos/components/log/include /home/danny/.platformio/packages/esp8266rtos/components/freertos/include /home/danny/.platformio/packages/esp8266rtos/components/freertos/port/esp8266/include /home/danny/.platformio/packages/esp8266rtos/components/freertos/port/esp8266/include/freertos /home/danny/.platformio/packages/esp8266rtos/components/freertos/include/freertos /home/danny/.platformio/packages/esp8266rtos/components/freertos/include/freertos/private /home/danny/.platformio/packages/esp8266rtos/components/tcp_transport/include /home/danny/.platformio/packages/esp8266rtos/components/esp_http_client/include /home/danny/.platformio/packages/esp8266rtos/components/esp_http_server/include /home/danny/.platformio/packages/esp8266rtos/components/newlib/platform_include /home/danny/.platformio/packages/esp8266rtos/components/esp_gdbstub/include /home/danny/.platformio/packages/esp8266rtos/components/protocomm/include /home/danny/.platformio/packages/esp8266rtos/components/wear_levelling/include /home/danny/.platformio/packages/esp8266rtos/components/pthread/include /home/danny/.platformio/packages/esp8266rtos/components/vfs/include /home/danny/.platformio/packages/esp8266rtos/components/openssl/include /home/danny/.platformio/packages/esp8266rtos/components/wifi_provisioning/include /home/danny/.platformio/packages/esp8266rtos/components/heap/include /home/danny/.platformio/packages/esp8266rtos/components/heap/port/esp8266/include /home/danny/.platformio/packages/esp8266rtos/components/spi_ram/include /home/danny/.platformio/packages/esp8266rtos/components/mdns/include /home/danny/.platformio/packages/esp8266rtos/components/esp_event/include /home/danny/.platformio/packages/esp8266rtos/components/lwip/include /home/danny/.platformio/packages/esp8266rtos/components/lwip/lwip/src/include /home/danny/.platformio/packages/esp8266rtos/components/lwip/port/esp8266/include /home/danny/.platformio/packages/esp8266rtos/components/lwip/include/apps /home/danny/.platformio/packages/esp8266rtos/components/lwip/include/apps/dhcpserver /home/danny/.platformio/packages/esp8266rtos/components/lwip/include/apps/sntp /home/danny/.platformio/packages/esp8266rtos/components/lwip/include/apps/ping /home/danny/.platformio/packages/esp8266rtos/components/jsmn/include /home/danny/.platformio/packages/esp8266rtos/components/esp8266/include /home/danny/.platformio/packages/esp8266rtos/components/esp8266/include/driver /home/danny/.platformio/packages/esp8266rtos/components/esp_ringbuf/include /home/danny/.platformio/packages/esp8266rtos/include
LDF: Library Dependency Finder -> https://bit.ly/configure-pio-ldf
LDF Modes: Finder ~ chain, Compatibility ~ soft
Found 45 compatible libraries

which appears correct.

And a bit lower :

xtensa-lx106-elf-gcc -o .pio/build/esp8266-d1mini/src/main.o -c -std=gnu99 -Wpointer-arith -Wno-implicit-function-declaration -Wl,-EL -fno-inline-functions -D__ESP_FILE__=__FILE__ -D__GNU_SOURCE -Os -mlongcalls -mtext-section-literals -falign-functions=4 -U__STRICT_ANSI__ -ffunction-sections -fdata-sections -DPLATFORMIO=60118 -DESP8266 -DARDUINO_ARCH_ESP8266 -DARDUINO_ESP8266_WEMOS_D1MINI -DF_CPU=80000000L -D__ets__ -DICACHE_FLASH -Isrc -I/home/danny/.platformio/packages/esp8266rtos/components/esp_ringbuf/include -I/home/danny/.platformio/packages/esp8266rtos/components/esp_ringbuf -I/home/danny/.platformio/packages/esp8266rtos/components/heap/include -I/home/danny/.platformio/packages/esp8266rtos/components/heap/src -I/home/danny/.platformio/packages/esp8266rtos/components/spi_flash/include -I/home/danny/.platformio/packages/esp8266rtos/components/spi_flash/src -I/home/danny/.platformio/packages/esp8266rtos/components/nvs_flash/include -I/home/danny/.platformio/packages/esp8266rtos/components/nvs_flash/src -I/home/danny/.platformio/packages/esp8266rtos/components/wpa_supplicant/include -I/home/danny/.platformio/packages/esp8266rtos/components/wpa_supplicant/src -I/home/danny/.platformio/packages/esp8266rtos/components/app_update/include -I/home/danny/.platformio/packages/esp8266rtos/components/app_update -I/home/danny/.platformio/packages/esp8266rtos/components/bootloader_support/include -I/home/danny/.platformio/packages/esp8266rtos/components/bootloader_support/src -I/home/danny/.platformio/packages/esp8266rtos/components/tcpip_adapter/include -I/home/danny/.platformio/packages/esp8266rtos/components/tcpip_adapter -I/home/danny/.platformio/packages/esp8266rtos/components/log/include -I/home/danny/.platformio/packages/esp8266rtos/components/log -I/home/danny/.platformio/packages/esp8266rtos/components/esp_event/include -I/home/danny/.platformio/packages/esp8266rtos/components/esp_event -I/home/danny/.platformio/packages/esp8266rtos/components/esp_http_server/include -I/home/danny/.platformio/packages/esp8266rtos/components/esp_http_server/src -I/home/danny/.platformio/packages/esp8266rtos/components/esp_http_client/include -I/home/danny/.platformio/packages/esp8266rtos/components/esp_http_client -I/home/danny/.platformio/packages/esp8266rtos/components/http_parser/include -I/home/danny/.platformio/packages/esp8266rtos/components/http_parser/src -I/home/danny/.platformio/packages/esp8266rtos/components/freertos/include -I/home/danny/.platformio/packages/esp8266rtos/components/freertos -I/home/danny/.platformio/packages/esp8266rtos/components/esp_common/include -I/home/danny/.platformio/packages/esp8266rtos/components/esp_common/src -I/home/danny/.platformio/packages/esp8266rtos/components/esp8266/include -I/home/danny/.platformio/packages/esp8266rtos/components/esp8266 "-I/home/danny/.platformio/platforms/esp8266rtos/builder/ /home/danny/.platformio/packages/esp8266rtos/components/esp_https_ota/include /home/danny/.platformio/packages/esp8266rtos/components/tcpip_adapter/include /home/danny/.platformio/packages/esp8266rtos/components/spiffs/include /home/danny/.platformio/packages/esp8266rtos/components/esp_common/include /home/danny/.platformio/packages/esp8266rtos/components/wpa_supplicant/include /home/danny/.platformio/packages/esp8266rtos/components/nvs_flash/include /home/danny/.platformio/packages/esp8266rtos/components/app_update/include /home/danny/.platformio/packages/esp8266rtos/components/bootloader_support/include /home/danny/.platformio/packages/esp8266rtos/components/http_parser/include /home/danny/.platformio/packages/esp8266rtos/components/spi_flash/include /home/danny/.platformio/packages/esp8266rtos/components/log/include /home/danny/.platformio/packages/esp8266rtos/components/freertos/include /home/danny/.platformio/packages/esp8266rtos/components/freertos/port/esp8266/include /home/danny/.platformio/packages/esp8266rtos/components/freertos/port/esp8266/include/freertos /home/danny/.platformio/packages/esp8266rtos/components/freertos/include/freertos /home/danny/.platformio/packages/esp8266rtos/components/freertos/include/freertos/private /home/danny/.platformio/packages/esp8266rtos/components/tcp_transport/include /home/danny/.platformio/packages/esp8266rtos/components/esp_http_client/include /home/danny/.platformio/packages/esp8266rtos/components/esp_http_server/include /home/danny/.platformio/packages/esp8266rtos/components/newlib/platform_include /home/danny/.platformio/packages/esp8266rtos/components/esp_gdbstub/include /home/danny/.platformio/packages/esp8266rtos/components/protocomm/include /home/danny/.platformio/packages/esp8266rtos/components/wear_levelling/include /home/danny/.platformio/packages/esp8266rtos/components/pthread/include /home/danny/.platformio/packages/esp8266rtos/components/vfs/include /home/danny/.platformio/packages/esp8266rtos/components/openssl/include /home/danny/.platformio/packages/esp8266rtos/components/wifi_provisioning/include /home/danny/.platformio/packages/esp8266rtos/components/heap/include /home/danny/.platformio/packages/esp8266rtos/components/heap/port/esp8266/include /home/danny/.platformio/packages/esp8266rtos/components/spi_ram/include /home/danny/.platformio/packages/esp8266rtos/components/mdns/include /home/danny/.platformio/packages/esp8266rtos/components/esp_event/include /home/danny/.platformio/packages/esp8266rtos/components/lwip/include /home/danny/.platformio/packages/esp8266rtos/components/lwip/lwip/src/include /home/danny/.platformio/packages/esp8266rtos/components/lwip/port/esp8266/include /home/danny/.platformio/packages/esp8266rtos/components/lwip/include/apps /home/danny/.platformio/packages/esp8266rtos/components/lwip/include/apps/dhcpserver /home/danny/.platformio/packages/esp8266rtos/components/lwip/include/apps/sntp /home/danny/.platformio/packages/esp8266rtos/components/lwip/include/apps/ping /home/danny/.platformio/packages/esp8266rtos/components/jsmn/include /home/danny/.platformio/packages/esp8266rtos/components/esp8266/include /home/danny/.platformio/packages/esp8266rtos/components/esp8266/include/driver /home/danny/.platformio/packages/esp8266rtos/components/esp_ringbuf/include /home/danny/.platformio/packages/esp8266rtos/include" src/main.c

which still contains superfluous (double) quotes, and directories without -I before them (because I didn’t put it there).

So I think the whole output of my script gets prefixed with a single -I and a double-quote, and suffixed with a double quote.

Which is wrong.

Nowhere does this make use of a list (https://www.geeksforgeeks.org/python/python-lists/)

Our latest messages were more or less simulaneous, I missed that. Python and platformio newbie I’m afraid.
Thanks, I’ll address that.

Thanks, this got me a lot further.