PioRemote - build remote not working

sticilface · November 5, 2016, 12:20pm

Have to hand it to you @ivankravets this is looking amazing!

as you know i was interested in this remote building. I can’t get pioremote to work on my freebsd, posted a seperate issue for that as there is no package for it. However, i can install it on my virtual machine. Everything seems to work, except for this error when i build remote…

➜  MySONOFF git:(master) ✗ platformio remote run -v -r -e default
PlatformIO Plus (https://pioplus.com) v0.4.1
[Sat Nov  5 12:14:29 2016] Processing default (build_flags: -Wl,-Tesp8266.flash.4m.ld, lib_deps: ArduinoJson, ESPAsyncTCP, ESPAsyncWebServer, AsyncMqttClient, Bounce2, board_f_cpu: 160000000L, platform: espressif8266, board: esp12e, framework: arduino, board_flash_mode: qio, upload_port: 192.168.1.196, board_f_flash: 80000000L)
--------------------------------------------------------------------------------------------------------------------------------------------------------
Library Storage: /tmp/tmpNahHHK/.piolibdeps
Looking for ArduinoJson library in registry
Found: http://platformio.org/lib/show/64/ArduinoJson
LibraryManager: Installing id=64
Downloading  [####################################]  100%
Unpacking  [####################################]  100%
ArduinoJson @ 5.6.7 has been successfully installed!
Looking for ESPAsyncTCP library in registry
Found: http://platformio.org/lib/show/305/ESPAsyncTCP
LibraryManager: Installing id=305
Downloading  [####################################]  100%
Unpacking  [####################################]  100%
ESPAsyncTCP @ 1.0.0 has been successfully installed!
Looking for ESPAsyncWebServer library in registry
Found: http://platformio.org/lib/show/306/ESPAsyncWebServer
LibraryManager: Installing id=306
Downloading  [####################################]  100%
Unpacking  [####################################]  100%
ESPAsyncWebServer @ 49d5a24815 has been successfully installed!
Installing dependencies
Looking for ESPAsyncTCP library in registry
Found: http://platformio.org/lib/show/305/ESPAsyncTCP
LibraryManager: Installing id=305
ESPAsyncTCP @ 1.0.0 is already installed
Looking for AsyncMqttClient library in registry
Found: http://platformio.org/lib/show/346/AsyncMqttClient
LibraryManager: Installing id=346
Downloading  [####################################]  100%
Unpacking  [####################################]  100%
AsyncMqttClient @ 0.5.0 has been successfully installed!
Installing dependencies
Looking for ESPAsyncTCP library in registry
Found: http://platformio.org/lib/show/305/ESPAsyncTCP
LibraryManager: Installing id=305
ESPAsyncTCP @ 1.0.0 is already installed
Looking for Bounce2 library in registry
Found: http://platformio.org/lib/show/1106/Bounce2
LibraryManager: Installing id=1106
Downloading  [####################################]  100%
Unpacking  [####################################]  100%
Bounce2 @ 2.1 has been successfully installed!
Collected 28 compatible libraries
Looking for dependencies...
Skip platform incompatible dependency {u'platforms': [u'espressif'], u'frameworks': [u'arduino'], u'name': u'ESPAsyncTCP', u'authors': u'Hristo Gochkov'}
Library Dependency Graph
|-- <ArduinoJson> v5.6.7 (/tmp/tmpNahHHK/.piolibdeps/ArduinoJson_ID64)
|-- <ESPAsyncTCP> v1.0.0 (/tmp/tmpNahHHK/.piolibdeps/ESPAsyncTCP_ID305)
|-- <ESPAsyncWebServer> (/tmp/tmpNahHHK/.piolibdeps/ESPAsyncWebServer_ID306)
|   |-- <ESPAsyncTCP> v1.0.0 (/tmp/tmpNahHHK/.piolibdeps/ESPAsyncTCP_ID305)
|   |-- <ArduinoJson> v5.6.7 (/tmp/tmpNahHHK/.piolibdeps/ArduinoJson_ID64)
|-- <AsyncMqttClient> v0.5.0 (/tmp/tmpNahHHK/.piolibdeps/AsyncMqttClient_ID346)
|   |-- <ESPAsyncTCP> v1.0.0 (/tmp/tmpNahHHK/.piolibdeps/ESPAsyncTCP_ID305)
|-- <Bounce2> v2.1 (/tmp/tmpNahHHK/.piolibdeps/Bounce2_ID1106)
Error: Nothing to build. Please put your source code files to '/tmp/tmpNahHHK/src' folder
============================================================== [ERROR] Took 1.02 seconds ==============================================================
Environment default	[ERROR]
============================================================== [ERROR] Took 1.02 seconds ==============================================================

====================================================================== [SUMMARY] ======================================================================
Environment beta   	[SKIP]

sticilface · November 5, 2016, 12:27pm

here is the platformio.ini if that helps

[platformio]
src_dir = ./MySONOFF ; keeps arduino happy :)
data_dir = ./MySONOFF/data

[env:default]

platform = espressif8266
framework = arduino
board = esp12e
build_flags = -Wl,-Tesp8266.flash.4m.ld
board_flash_mode = qio
board_f_cpu = 160000000L
board_f_flash = 80000000L
upload_port = 192.168.1.196 ; square
lib_deps = ArduinoJson, ESPAsyncTCP, ESPAsyncWebServer, AsyncMqttClient, Bounce2


[env:beta]

platform = espressif8266
framework = arduino
board = esp12e
build_flags = -Wl,-Tesp8266.flash.4m.ld -DDEBUG_CODE=Serial
board_flash_mode = qio
board_f_cpu = 160000000L
board_f_flash = 80000000L
upload_port = 192.168.1.196 ; tv
;upload_port = /dev/cu.usbserial
;upload_speed = 460800
lib_deps = ArduinoJson, ESPAsyncTCP, ESPAsyncWebServer, AsyncMqttClient, Bounce2
; -DDebug_ESPManager=Serial

ivankravets · November 5, 2016, 12:35pm

Thanks a lot for trying!

I’ll preapre binary for FreeBSD. Which version of FreeBSD do you use?
What is your scheme? What is a type of host machine where PIO Remote Agent is started?
What is a type of host machine where you have run this platformio remote run -v -r -e default.
Why do you need this -r? Do you know what does it mean? Are you sure to want build whole project on remote side?

Please sorry for a lot of questions. I’ll improve current docs.

ivankravets · November 5, 2016, 12:43pm

This part of config caused this error. Currently, PIO Remote doesn’t handle custom directories. That is is very complicated question from a security point. Of course, that is our problem and we will resovle that. What is more, all configuration options related to PATH/DIR will be removed from config. That is a current implementation. Need to start from something.

Using 0.4.1 version of PIO Remote, you have 2 options

Please don’t use -r option in combination with custom src/lib dirs. In this case al lthings will be built on the current client host machine and only files required for uploading/SPIFFS will be transferred to a target machine
Put source code of project to src directory and remove this part

[platformio]
src_dir = ./MySONOFF ; keeps arduino happy :)
data_dir = ./MySONOFF/data

Would be thankful for any feedback.

sticilface · November 5, 2016, 1:02pm

I suspected that this was the issue. that is why i posted it.

The reason for this is that i alternate quite frequently between arduinoIDE and platformio, especially when debugging I use the stack trace tool from me-no-dev. the IDE is also useful for one off upload to this port, oh and try this port type things. To keep it arduino compatible the .ino has to be in a folder with the same name, and putting it in src kind borks that. I guess I can link them so that it works! I will try.

to answer your questions
I’ll preapre binary for FreeBSD. Which version of FreeBSD do you use? 10.3-STABLE
What is your scheme? What is a type of host machine where PIO Remote Agent is started? Ubuntu 16
What is a type of host machine where you have run this platformio remote run -v -r -e default. Mac OSsierra
Why do you need this -r? Do you know what does it mean? Are you sure to want build whole project on remote side? yes totally aware. this was the whole point. the reason... my mac is a 5 year old air, my freebsd is an 8core xeon..

The docs are fine, i managed very easily to follow everything and get it working.

ivankravets · November 5, 2016, 1:06pm

I got you and will fix all these issues. I’ll post updates here. Thanks!

sticilface · November 5, 2016, 1:12pm

thats great.

so i removed the src= from platformio.ini, symlinked the .ino into the src folder.
The project builds locally but again I get the nothing to build msg on the remote.

here is the directory structure

├── www
│   ├── untitled
│   ├── preload.js
│   ├── jquery.mobile-1.4.5.min.js
│   ├── jquery.mobile-1.4.5.min.css
│   ├── jquery-1.11.1.min.js
│   ├── index.js
│   ├── index.htm
│   └── events.js
├── tmp.package
│   ├── manifest.json
│   └── data
├── src -> ./MySONOFF
│   ├── version.h
│   ├── data
│   └── MySONOFF.ino
├── scripts
│   ├── tmp.package
│   ├── package.json
│   ├── node_modules
│   ├── gulpfile.js
│   ├── deploy.sh
│   ├── data
│   └── buildmanifest.py
├── platformio.ini
├── lib
│   ├── readme.txt
│   ├── async-mqtt-client -> /Users/amelvin/Documents/Arduino/libraries/async-mqtt-client
│   ├── WemosArduino -> /Users/amelvin/Documents/Arduino/libraries/WemosArduino
│   ├── OneButton -> /Users/amelvin/Documents/Arduino/libraries/OneButton
│   ├── ESPmanager -> /Users/amelvin/Documents/Arduino/libraries/ESPmanager
│   ├── ESPdeviceFinder -> /Users/amelvin/Documents/Arduino/libraries/ESPdeviceFinder
│   ├── ESPAsyncWebServer -> /Users/amelvin/Documents/Arduino/libraries/ESPAsyncWebServer
│   ├── ESPAsyncTCP -> /Users/amelvin/Documents/Arduino/libraries/ESPAsyncTCP
│   └── ArduinoJson -> /Users/amelvin/Documents/Arduino/libraries/ArduinoJson
├── MySonnOff.sublime-workspace
├── MySonnOff.sublime-project
└── MySONOFF
    ├── version.h
    ├── data
    └── MySONOFF.ino

ivankravets · November 5, 2016, 1:38pm

Just for expreiment, try to create generic PlatformIO project wihout any symlinks with default src, lib and data folder. It should work.

sticilface · November 5, 2016, 1:54pm

correct it does. although it then fails as there is a dependent lib, that is not part of the src, or an available lib! getting there though!

ivankravets · November 5, 2016, 4:28pm

Please run pio update on the both side: agent & clients". They should update PIO Plus to 0.4.2 where I fixed all issues that were related to remote builder. Also, don’t forget to restart agent.

P.S: To be honest, I didn’t expect that someone will use --build-remotely for embedded development platforms. I made that for desktop development platforms. For example, you can code on Mac/Win and build native binary for RPi that is used as agent’s machine. In any case, that is a good that this option has other use case.
P.S.S: FreeBSD binary in progress. I’m new to this OS and need to configure build environment.

sticilface · November 5, 2016, 6:15pm

ok…

so slightly different error now.
➜ MySONOFF git:(master) ✗ platformio remote run -v -r -e default PlatformIO Plus (https://pioplus.com) v0.4.2 The maximum allowed size for archived project is 1048576 bytes

You are correct that this would be a novel use, but i remember asking you about it a long time ago initially with the freebsd stuff, as my server is so much faster at compiling than my laptop.

ivankravets · November 5, 2016, 6:38pm

That is good error What are you going to transfer to remote machine? 1Gb of new film? Yep, that is a joke

I see on the last 0.4.2 PIO Plus code and it archives data_dir even when buildfs or buildfsota targets are not used. This is my fault. I’ll fix it.
Need to think on the better implementation of current PIO Remote Protocol v1.1. Your use case shows me that developers can have data directory with a big size and “network producing” in this case should be chunked.

Temporary, try to define data_dir in the project to non existing path. I want to be sure that generic workflow with firmware building and uploadings works well.

Thanks!

sticilface · November 5, 2016, 7:31pm

mmm… so i did not actually have a data_dir defined in the platformio.ini file. I do however, have a www directory in the project root that contains all my html stuff for the project…

so im not sure what it is uploading that is so large!

the whole folder / project is 150mb, but this is because i use gulp to automate production of the html page. so i wonder if it copying all the subfolders as well… and the ./scripts + ./scripts/node_modules

ivankravets · November 5, 2016, 7:34pm

It copies data direcotry. What is the size of it?

sticilface · November 5, 2016, 7:38pm

this is the project root. there is no 1st level data directory…

➜  MySONOFF git:(master) ✗ tree -L 2
.
├── MySONOFF
│   ├── MySONOFF.ino
│   ├── data
│   └── version.h
├── MySonnOff.sublime-project
├── MySonnOff.sublime-workspace
├── lib
│   ├── ArduinoJson -> /Users/amelvin/Documents/Arduino/libraries/ArduinoJson
│   ├── ESPAsyncTCP -> /Users/amelvin/Documents/Arduino/libraries/ESPAsyncTCP
│   ├── ESPAsyncWebServer -> /Users/amelvin/Documents/Arduino/libraries/ESPAsyncWebServer
│   ├── ESPdeviceFinder -> /Users/amelvin/Documents/Arduino/libraries/ESPdeviceFinder
│   ├── ESPmanager -> /Users/amelvin/Documents/Arduino/libraries/ESPmanager
│   ├── OneButton -> /Users/amelvin/Documents/Arduino/libraries/OneButton
│   ├── WemosArduino -> /Users/amelvin/Documents/Arduino/libraries/WemosArduino
│   ├── async-mqtt-client -> /Users/amelvin/Documents/Arduino/libraries/async-mqtt-client
│   └── readme.txt
├── platformio.ini
├── scripts
│   ├── buildmanifest.py
│   ├── data
│   ├── deploy.sh
│   ├── gulpfile.js
│   ├── node_modules
│   ├── package.json
│   └── tmp.package
├── src
│   ├── MySONOFF.ino
│   └── version.h
├── tmp.package
│   ├── data
│   └── manifest.json
└── www
    ├── events.js
    ├── index.htm
    ├── index.js
    ├── jquery-1.11.1.min.js
    ├── jquery.mobile-1.4.5.min.css
    ├── jquery.mobile-1.4.5.min.js
    ├── preload.js
    └── untitled

19 directories, 21 files

I’m using src, now. not the original folder as i had originally mentioned mySONOFF

ivankravets · November 5, 2016, 7:48pm

Please decide how do you want to use PIO. You have 2 source locations: ./MySONOFF an src. As I’ve said above, I already fixed issue with custom src_dir in conf. You can keep project compatible between Arduino/PlatformIO.
PIO Remote doesn’t use FOLDERS. It gets data from general settings configured in platformio.ini.
When you start remote building on remote machine, it copies *_dir (see Redirecting...): src_dir, lib_dir, data_dir.

Now, try to COPY somewhere these contents and calculate a size.

sticilface · November 5, 2016, 7:59pm

ah ok. got thank, sorry i didn’t quite understand how it was working. The issue then is that i’ve symlinked in several lib folders (which also has the same sort of thing… www folder, scripts with node_modules…) which will total 100mg as well…) I suppose the way round it is to only symlink in the src folder within each lib and not the whole thing. i will give it a go!

sticilface · November 5, 2016, 8:04pm

and that worked a treat!

it was copying the whole *_lib as you said, which in my case included a lot of files not required!

ivankravets · November 5, 2016, 8:04pm

Maybe, that is our issue? Currently, we transfer ALL data from specified dirs above. Maybe, need to pack only C/C++ related files?

sticilface · November 5, 2016, 8:11pm

very good stuff though. fantastic work. if i -clean locally i go from 47seconds to 21 seconds. now if it implemented a similar cache / diff it would be lightening fast, but that would be pretty hard to implement!