OpenBSD: esp8266 partially working, help with atmelavr

Development Platforms
1

Hi,

i partially got platformio running on OpenBSD for my esp8266 chips. I hade to make two fake packages for toolchain-xtensa und tool-esptool (see GitHub - lotherk/platformio-openbsd-toolchain-xtensa: Fake toolchain-xtensa package for platformio to run on OpenBSD) to build the firmware.

The trick is basically just symlinking things from the system to ~/.platformio/packages/xyz. See https://termbin.com/qlwy as an example for the toolchain-xtensa. You get the idea.

However, I couldn’t upload my project with pio as I couldn’t make the same trick work for tool-mkspiffs. Uploading directly with esptool.py worked. So currently I’m using pio run to compile the project for my esp8266 and esptool.py to upload it. It’s not nice but works.

I tried this trick also to make atmelavr work on OpenBSD but couldn’t convince pio to accept my toolchain-atmelavr fake package. I tried multiple things, comparing folder structure from a linux installation etc. but for the sake of me can’t get it to work.

All the binaries needed for compiling esp, atmelavr and all the tools needed for uploading like mkspiffs, esptool, avrdude etc. are available on my system and are working.

My actual question is what do I have to do to make toolchain-atmelavr and tool-mkspiffs work with my symlink magic. The fake package.json seem correct (just like the ones I generate for toolchain-xtensa and tool-esptool).

Also, why are all the binaries needed in pio packages and why isn’t it possible for pio to just use what is inside PATH?

As a last resort I would also be thankful for a tip on how to make pio print the actual command it is using for compiling, uploading etc. so I would just run it once on Linux and copy the commands into a Makefile… :wink:

I sure can build all the firmware using a linux vm but I want it on my OpenBSD directly and am willing to help upstream.
I can compile and provide binaries if needed. I am running OpenBSD-current but could set up a VM with the current stable RELEASE if that would be needed.

I seem not to be the only one trying to get pio working on *BSD as can be seen on some posts here on the forum and on github for example from user trombik, which were the inspiration for the fake packages thing. Sorry, I only can post 2 links as a new user here in this forum so you have to search the links yourself.

Any help or tipps are greatly appreciated. Also happy new year to everyone of you.

2

@ivankravets OpenBSD toolchains still seem incomplete, in regards to GitHub - trombik/platformio-freebsd-toolchain-xtensa: Trick platformio so that it thinks toolchain-xtensa, a platformio package, is installed. With recent ESP8266/Arduino (3.0.0), This workaround does not work anymore. Use https://github.com/trombik/freebsd-ports-esp-quick-toolchain/ instead.

@lotherk I think it’s best if the toolchains just get uploaded – although one could use the same tricks and those should work too. Given a locally compiled version of GitHub - igrr/mkspiffs: Tool to build and unpack SPIFFS images so that it gives you the needed executables

grafik
grafik914×306 17.4 KB

(especially mkspiffs_espressif8266_arduino in your case) plus an adapted package.json of like

{
  "name": "tool-mkspiffs",
  "version": "2.230.0",
  "description": "Tool to build and unpack SPIFFS images",
  "keywords": [
    "tools",
    "build tools",
    "filesystem"
  ],
  "license": "MIT",
  "system": [
    "freebsd_amd64"
  ],
  "repository": {
    "type": "git",
    "url": "https://github.com/igrr/mkspiffs"
  }
}
1 Like
3

thank you for your answer.

the same tricks don’t work for atmelavr toolchain, at least I can not figure out what is the right directory structure.

regarding mkspiffs:

I have a /usr/local/bin/mkspiffs on my system and now created a fake package with a symlink to mkspiffs_espressif8266_arduino: https://termbin.com/p8xs and package.json https://termbin.com/mwj2. It does not work, though. pio still wants to install tool-mkspiffs:

Tool Manager: Installing tool-mkspiffs @ ~1.200.0
Error: Could not find the package with ‘tool-mkspiffs @ ~1.200.0’ requirements for your system ‘openbsd_amd64’

what is the magic in pio to check if a package is valid? knowing that would also help with toolchain-atmelavr.

4

Could you provide us fixed packages for OpenBSD?

5

What do you mean with “fixed packages”?

6

Sorry, I meant “working packages”.

7

I don’t have any working packages as per my post above. I only have the symlink magic, which does not work for tool-mkspiffs and toolchain-atmelavr (which I am trying to solve here).

I would be glad to help but I don’t understand pio’s packaging and couldn’t find any useful information.

8

Where were the atmel toolchains sourced from for OpenBSD? Standard packet manager?

9

Yes, everything comes from standard OpenBSD packages.

I have installed:
arduino-1.8.10v0 open-source electronics prototyping platform
arduino-esp8266-2.7.4p0 esp8266 arduino core toolset
esptool-ck-0.4.13 firmware creation helper for esp8266 chips
py3-esptool-3.0 esp8266 and esp32 rom bootloader communication tool
mkspiffs-0.2.3 tool to build and unpack SPIFFS images
avr-binutils-2.30 Atmel AVR 8-bit RISC microcontrollers’ GNU binutils
avr-gcc-5.4.0p3 Atmel AVR 8-bit RISC microcontrollers’ XGCC
avr-libc-2.0.0 libc for Atmel AVR 8-bit RISC microcontrollers

All I did to install them was pkg_add arduino-esp8266 which pulled all the other packages as dependencies, IIRC.

10

Before this dies again, please give me some informations regarding the packaging so I can get this working. Or at least some info how pio decides if a package is valid or not.

11

Well I was setting up a virutal machine with OpenBSD but I’m stuck in a corner after somehow incorrectly installing an XFCE desktop environment (according to a tutorial) so that I’m not stuck in the late 1980’s with the default desktop. Might retry again with a different guide.

How to create packages: First of all, you should get familiar with what packages are needed when working in what environment. These are listed in the platform.json of each paltform (e.g. platform-atmelavr/platform.json at develop · platformio/platform-atmelavr · GitHub for Atmel-AVR and platform-espressif8266/platform.json at develop · platformio/platform-espressif8266 · GitHub for ESP8266).

Only some of these packages contain compiled binaries that need to be adapted for every operating system and architecture, others are just generic source code packages, like e.g. the Arduino core.

For Atmel-AVR, the needed packages are e.g. toolchain-atmelavr (version ~1.50400.0, maps to GCC 5.4.0) and tool-avrdude (version ~1.60300.0, maping to 6.3). Analoguous for ESP8266 (toolchain-xtensa, tool-mkspiffs, tool-mklittlefs).

First of all, one should take a look at the package for other OSes so that one knows what files and folders should be in there as a reference. Those can all be searched on Service End for Bintray, JCenter, GoCenter, and ChartCenter | JFrog.

Example: https://bintray.com/platformio/tool-packages/tool-avrdude-freebsd_amd64#files is the FreeBSD package version for tool-avrdude. The archive can be downloaded, extracted and inspected.

grafik
grafik803×446 24.8 KB

The most important thing here is what PlatformIO calls into, the avrdude executatble, and the package.json package description. The rest are self-created helper files or the files of the original program.

So to create the OpenBSD package version of that, we would need to:

  • download and compile avrdude 6.3 from source for OpenBSD (assuming x64) natively, to obtain all the needed program files
  • create an adapted package.json that has the correct system information
    • the package.json of the package for other OSes is helpful here – in this case, we would just need to replace freebsd_amd64 with openbsd_amd64
  • create a folder with the program files and the package.json
    • plus possibly a special case here: the avrdude.conf file may have been expanded by the PlatformIO maintainers, so it’s best to copy that from the original package;
  • files and folders should now be pretty much the same than in the package version for either FreeBSD if it exists or Linux
  • call the command pio package pack on that directory path
    • → this will create the needed .tar.gz file, which is the package file
    • those can be uploaded and the link posted here so that maintainers may test and integrate them
    • this file should also be usable with a PlatformIO project in the platform_packages command and the file:// protocol for reference, so they can be locally tested

If toolchains e.g. were downloaded using a package manager, the procedure for “downloading and compiling the source code” would just be replaced by “locating the files and copying them in a folder”.

12

Thank you very much! Could one please test the packages under https://konradlother.com/pub/platformio/

They are bulit on a current snapshot of OpenBSD and work fine on my machine with file:// in platform_packages:

platform_packages =
tool-mkspiffs @ file:///home/lotherk/code/platformio/tool-mkspiffs-openbsd_amd64-1.102.0.tar.gz
tool-esptool @ file:///home/lotherk/code/platformio/tool-esptool-openbsd_amd64-1.409.0.tar.gz
tool-mklittlefs @ file:///home/lotherk/code/platformio/tool-mklittlefs-openbsd_amd64-1.203.200522.tar.gz
toolchain-xtensa @ file:///home/lotherk/code/platformio/toolchain-xtensa-openbsd_amd64-1.50200.0.tar.gz

I can build and upload with them.

I will work on the atmelavr things as soon as possible, too.

Again, thank you very much!

Edit:
I will also make this for OpenBSD/arm64.

Edit2:
If you want to test these packages you only have to install a bare OpenBSD in a VM, saying “no” to running X and just ssh into it and run:

pkg_add python3 py3-pip arduino-esp8266
pip3 install platformio

1 Like
13

toolchain-atmelavr and tool-avrdude are ready for testing, too.

14

Thank you, I’ll test both toolchains in a fresh VM and give some feedback :slight_smile:

If all works on a fresh install I’d say they’re good enough to be reuploaded as official packages by @ivankravets so that OpenBSD support becomes more complete.

15

I just don’t know how the binaries will behave longterm as they’re build on a current snapshot and things might change and may need to be recompiled and re-uploaded.

Also not sure what happens if someone with OpenBSD 6.7 pulls the packages.

I’m currently writing some scripts that help packaging and might be useful.

16

Well surely OpenBSD won’t just break binary compatibility for old compiled binaries out of nothing with the kernel or used libraries?

17

I can not give you a qualified answer to that.

18

I see, OpenBSD or the BSDs seem to be a different kind of beast here with the package management system… They take “ports” as recipicies to compile things from source? (what I read)

I’ll probably be able to test the packages tomorrow after… reinstalling openbsd on a bigger harddrive and better partition table to get more than -36.7MB free space after installation (yes, negative)…

openbsd# pio run
Processing uno (platform: atmelavr; board: uno; framework: arduino)
------------------------------------------------------------------------------------------------------------------------
Tool Manager: Installing file:///root/pkgs/toolchain-atmelavr-openbsd_amd64-1.50400.0.tar.gz
Unpacking  [###################################-]   99%  00:00:00
/: write failed, file system is full

Unpacking...
^C
openbsd# df -h
Filesystem     Size    Used   Avail Capacity  Mounted on
/dev/wd0a      986M    973M  -36.7M   104%    /

I also noticed that the toolchain for atmel is quite big. I do remember that too when compiling GCC that all executables would have huge amount of debug information in it. That can be run through strip to fix that without breaking the binary though. That would decrease distribution size a lot.

19

Well I tried this on a fresh install and it failed :confused:

openbsd$ pio run
Processing uno (platform: atmelavr; board: uno; framework: arduino)
------------------------------------------------------------------------------------------------------------------------
Tool Manager: Installing file:///home/max/pkgs/toolchain-atmelavr-openbsd_amd64-1.50400.0.tar.gz
Unpacking  [####################################]  100%
Tool Manager: toolchain-atmelavr @ 1.50400.0 has been installed!
Tool Manager: Installing platformio/framework-arduino-avr @ ~5.1.0
Downloading  [####################################]  100%
Unpacking  [####################################]  100%
Tool Manager: framework-arduino-avr @ 5.1.0 has been installed!
Tool Manager: Installing file:///home/max/pkgs/tool-avrdude-openbsd_amd64-1.60300.200527.tar.gz
Tool Manager: tool-avrdude @ 1.60300.200527 has been installed!
Tool Manager: Installing platformio/tool-scons @ ~4.40001.0
Downloading  [####################################]  100%
Unpacking  [####################################]  100%
Tool Manager: tool-scons @ 4.40001.0 has been installed!
Verbose mode can be enabled via `-v, --verbose` option
CONFIGURATION: https://docs.platformio.org/page/boards/atmelavr/uno.html
PLATFORM: Atmel AVR (3.1.0) > Arduino Uno
HARDWARE: ATMEGA328P 16MHz, 2KB RAM, 31.50KB Flash
DEBUG: Current (avr-stub) On-board (avr-stub, simavr)
PACKAGES:
 - framework-arduino-avr 5.1.0
 - tool-avrdude 1.60300.200527 (6.3.0)
 - toolchain-atmelavr 1.50400.0 (5.4.0)
LDF: Library Dependency Finder -> http://bit.ly/configure-pio-ldf
LDF Modes: Finder ~ chain, Compatibility ~ soft
Found 5 compatible libraries
Scanning dependencies...
No dependencies
Building in release mode
Compiling .pio/build/uno/src/main.cpp.o
ld.so: avr-g++: can't load library 'libc++.so.6.0'
Killed
Archiving .pio/build/uno/libFrameworkArduinoVariant.a
ld.so: avr-gcc-ar: can't load library 'libc++.so.6.0'
Killed
Compiling .pio/build/uno/FrameworkArduino/CDC.cpp.o
Compiling .pio/build/uno/FrameworkArduino/HardwareSerial.cpp.o
ld.so: avr-g++: can't load library 'libc++abi.so.4.0'
Killed
ld.so: avr-g++: can't load library 'libc++.so.6.0'
Killed
Compiling .pio/build/uno/FrameworkArduino/HardwareSerial0.cpp.o
ld.so: avr-g++: can't load library 'libc++abi.so.4.0'
Killed
Compiling .pio/build/uno/FrameworkArduino/HardwareSerial1.cpp.o
ld.so: avr-g++: can't load library 'libc++abi.so.4.0'
Killed
Compiling .pio/build/uno/FrameworkArduino/HardwareSerial2.cpp.o
ld.so: avr-g++: can't load library 'libc++.so.6.0'
*** [.pio/build/uno/src/main.cpp.o] Error 137
Killed
*** [.pio/build/uno/libFrameworkArduinoVariant.a] Error 137
*** [.pio/build/uno/FrameworkArduino/HardwareSerial.cpp.o] Error 137
*** [.pio/build/uno/FrameworkArduino/CDC.cpp.o] Error 137
*** [.pio/build/uno/FrameworkArduino/HardwareSerial0.cpp.o] Error 137
*** [.pio/build/uno/FrameworkArduino/HardwareSerial1.cpp.o] Error 137
*** [.pio/build/uno/FrameworkArduino/HardwareSerial2.cpp.o] Error 137
Compiling .pio/build/uno/FrameworkArduino/HardwareSerial3.cpp.o
ld.so: avr-g++: can't load library 'libc++.so.6.0'
Killed
*** [.pio/build/uno/FrameworkArduino/HardwareSerial3.cpp.o] Error 137
============================================= [FAILED] Took 481.76 seconds =============================================

My system has a /usr/lib/libc++.so.5.0.

openbsd# uname -a
OpenBSD openbsd.lan 6.8 GENERIC.MP#98 amd64
20

I have both libc++.so.5.0 and libc++.so.6.0. This must have changed in one of the last snapshots:

OpenBSD t450s 6.8 GENERIC.MP#268 amd64

A really bad Idea I had is to package just symlinks to the system directories. That would require the user to install the system packages, tho. I guess this is what platformio wants to prevent with its packaging for it to “just work”. An advantage is that the package would be architecture independent. Maybe there is a way that pio could inform the user to install the required packages on their system? Still sounds bad.

I don’t have a good Idea for packaging binaries for OpenBSD, then again I am no expert in that field.

For now I reference the tools I created in my platformio to https://myserver.

Edit:
I have uploaded the scripts to build the packages: GitHub - lotherk/platformio-openbsd-packages

1 Like