Me again… I am trying to integrate ArduinoOTA into my project to be able to update the code even if the device is on-site. My problem is, the device is in another network, behind a firewall. The firewall is configured to pass port 3232 to the device. But OTA updates are failing constantly, the invitation is never answered:
esptool.py v2.6
Advanced Memory Usage is available via "PlatformIO Home > Project Inspect"
RAM: [= ] 5.3% (used 28372 bytes from 532480 bytes)
Flash: [=== ] 30.6% (used 400518 bytes from 1310720 bytes)
Configuring upload protocol...
AVAILABLE: esp-prog, espota, esptool, iot-bus-jtag, jlink, minimodule, olimex-arm-usb-ocd, olimex-arm-usb-ocd-h, olimex-arm-usb-tiny-h, olimex-jtag-tiny, tumpa
CURRENT: upload_protocol = espota
Uploading .pio\build\az-delivery-devkit-v4\firmware.bin
18:11:20 [DEBUG]: Options: {'timeout': 10, 'esp_ip': '192.168.178.77', 'host_port': 41297, 'image': '.pio\\build\\az-delivery-devkit-v4\\firmware.bin', 'host_ip': '0.0.0.0', 'auth': 'Nurminnen', 'esp_port': 3232, 'spiffs': False, 'debug': True, 'progress': True}
18:11:20 [INFO]: Starting on 0.0.0.0:41297
18:11:20 [INFO]: Upload size: 400640
Sending invitation to 192.168.178.77 ..........
18:13:00 [ERROR]: No response from the ESP
*** [upload] Error 1
The esp_ip in the log is that of the firewall/gateway, the IP of the device behind it is 192.178.199.40, but is of course not addressable.
Will I need to forward other ports as well? I was under the impression that only 3232 was used. The device can react on requests for port 502 (MODBUS/TCP) without problems, the port forwarding was configuered the same way for 3232. Additionally the device is cyclically gathering NTP time via port 123/UDP with 8888 as the callback port without problems, so outgoing traffic seems to be okay.
So you are addressing the Gateway instead? And 192.168.178.77 is the IP of the gatway?
Also, have you tried building the firmware with debug logs enable to see whether device gets any data at all? See docs, you shou do build_flags = -DCORE_DEBUG_LEVEL=3 or 4.
I tried both -DCORE_DEBUG_LEVEL=3 and =4, but with no noticeable differences. Where am I supposed to look for the debug output?
In another attempt I tried OTA upload within the same subnet, directly addressing the ESP with its IP address 192.168.199.40. The same happened, the upload timed out without any reaction on the invitation. As this subnet is sparsely active, the data LED on the W5500 module may be an indication of traffic going to the device. I saw it flicker each time the “Sending invitation” line got another dot added, so I am assuming the packets were actually sent to the device.
By the way: repeated builds are adding more and more ofthese error lines:
[Error 2] Das System kann die angegebene Datei nicht finden: 'C:\\Users\\Micha\\Documents\\PlatformIO\\Projects\\RS485-Bridge\\.pio\\build\\az-delivery-devkit-v4\\libFrameworkArduinoVariant.a'
Please manually remove the file `C:\Users\Micha\Documents\PlatformIO\Projects\RS485-Bridge\.pio\build\az-delivery-devkit-v4\libFrameworkArduinoVariant.a`
[Error 3] Das System kann den angegebenen Pfad nicht finden: 'C:\\Users\\Micha\\Documents\\PlatformIO\\Projects\\RS485-Bridge\\.pio\\build\\az-delivery-devkit-v4\\longcmd-1edf16a8dfab01cd43f3613a9b992520'
Please manually remove the file `C:\Users\Micha\Documents\PlatformIO\Projects\RS485-Bridge\.pio\build\az-delivery-devkit-v4\longcmd-1edf16a8dfab01cd43f3613a9b992520`
[Error 3] Das System kann den angegebenen Pfad nicht finden: 'C:\\Users\\Micha\\Documents\\PlatformIO\\Projects\\RS485-Bridge\\.pio\\build\\az-delivery-devkit-v4\\longcmd-6c0073e71529232f49235f36b0dbaf93'
Please manually remove the file `C:\Users\Micha\Documents\PlatformIO\Projects\RS485-Bridge\.pio\build\az-delivery-devkit-v4\longcmd-6c0073e71529232f49235f36b0dbaf93`
[Error 3] Das System kann den angegebenen Pfad nicht finden: 'C:\\Users\\Micha\\Documents\\PlatformIO\\Projects\\RS485-Bridge\\.pio\\build\\az-delivery-devkit-v4\\partitions.bin'
Please manually remove the file `C:\Users\Micha\Documents\PlatformIO\Projects\RS485-Bridge\.pio\build\az-delivery-devkit-v4\partitions.bin`
[Error 3] Das System kann den angegebenen Pfad nicht finden: 'C:\\Users\\Micha\\Documents\\PlatformIO\\Projects\\RS485-Bridge\\.pio\\build\\az-delivery-devkit-v4\\src'
Please manually remove the file `C:\Users\Micha\Documents\PlatformIO\Projects\RS485-Bridge\.pio\build\az-delivery-devkit-v4\src`
[Error 2] Das System kann die angegebene Datei nicht finden: 'C:\\Users\\Micha\\Documents\\PlatformIO\\Projects\\RS485-Bridge\\.pio\\build\\az-delivery-devkit-v4'
Please manually remove the file `C:\Users\Micha\Documents\PlatformIO\Projects\RS485-Bridge\.pio\build\az-delivery-devkit-v4`
Once I removed the directories as indicated, the next build ran without these errors.
Final observation: after having run for more than a week without problems, after adding the ArduinoOTA code yesterday (and not changing anything else) I had a complete freeze today without the ESP starting itself again. I had to cycle power to get it running again.
I am really stuck here - I almost for sure can sort out firewall issues, as other ports are forwarded without problems that are configured identically.
My main suspects at the moment are:
Wrong ArduinoOTA library - but there is only one available?!?
ArduinoOTA cannot cope with IP+port only, is missing its host name in the requests
needs to have the broadcast mDNS packets answered - that will not pass the firewall, so no one will do that
The device will have to be mounted into a DIN rail closet at the customer’s site, so going there, unmount it, flash it and mount it back in is not an option.
Try and sort things out when being on the same network, first. That you’re doing it with a W5500 Ethernet chip in the middle is a pretty important bit of information that wasn’t mentioned at first.
Does it also not work with the Arduino IDE? Have you made sure to confirm the functionality of the W5500 chip first with a test sketch? What code are you using to set up Ethernet + OTA? Does increasing the debug level make any ingoing packets visible?
I did not try that, I may have an outdated Arduino IDE installed somewhere but I never used it for the ESP32. The sketch I am working on has some 3000+ lines of code now, so I rather would write a test sketch instead. I honestly will try that as a last resort only, given the efforts.
Yes, it is working fine. I am doing NTP requests regularly and the ESP is properly reacting on TCP requests on port 502, so the module seems to work and the network settings are okay.
Excerpt from my code:
...
// SPI and Ethernet libs for W5500 Ethernet module
#include <SPI.h>
#include <IPAddress.h>
#include <Ethernet.h>
// W5500 reset pin
#define RESET_P GPIO_NUM_26
...
// OTA update interface
#include <ArduinoOTA.h>
...
// Server on port 502 (MODBUS)
EthernetServer server(IPport);
...
// A UDP instance to let us send and receive packets over UDP
EthernetUDP Udp;
...
void setup()
{
...
// SPI CS line is GPIO_NUM_5
Ethernet.init(GPIO_NUM_5);
// Hard boot W5500 module
WizReset();
...
// start the Ethernet connection:
bool connected = false;
if(IPDHCP) // DHCP required?
{
Trace(0, "Trying to get an IP address using DHCP\n");
if (Ethernet.begin(mac) == 0)
{
Trace(0, "Failed to configure Ethernet using DHCP\n");
}
else connected = true;
}
// Connection established?
if(!connected)
{
// No. Either DHCP was not required or did not work.
// initialize the Ethernet device not using DHCP:
Ethernet.begin(mac, Reverse(IPaddress), Reverse(IPDNS), Reverse(IPgateway), Reverse(IPmask));
// Check for Ethernet hardware present
if (Ethernet.hardwareStatus() == EthernetNoHardware)
{
Trace(0, "Ethernet shield was not found. Sorry, can't run without hardware. :(\n");
}
else
{
if (Ethernet.linkStatus() == LinkOFF)
{
Trace(0, "Ethernet cable is not connected.\n");
}
else connected = true;
}
}
// print local IP address:
lIP = Ethernet.localIP();
Trace(TRACE_BASIC, "My IP address: %u.%u.%u.%u\n", lIP[0], lIP[1], lIP[2], lIP[3]);
// Update server to configured port
if(IPport) server = EthernetServer(IPport);
// Now start it
server.begin();
...
// Set up UDP for NTP requests
setenv("TZ", NTP_TZ, 1);
tzset();
Udp.begin(localPort);
...
// Start ArduinoOTA service
ArduinoOTA.begin(lIP, "myBridge", "xxxx", InternalStorage);
}
...
void loop()
{
...
// Check for OTA updates
ArduinoOTA.poll();
...
}
Not at all, as I wrote above. The flag does not have a single effect, that is why I asked where to look for the additional output.
For the time being I give up now. I could not find a reason why the OTA invitation was not answered. The packets looked good and were sent to the W5500 module, but inside the ESP32 nothing was happening.
So you did try the minimal sketch with just enough added on to initialize the W5500 interface? I think the issue here is that when the OTA library listens on port 3232, it does so on the general WiFi (or hardware ethernet interface), but it doesn’t know anything about the W5500 chip attached via SPI? There may be some extensions that must be done to the network stack there. I’m sure people at Issues · espressif/arduino-esp32 · GitHub would know more.
The ArduinoOTA library is claiming to work with Ethernet.h, but you may be right and that is only meant for the “real Arduino” flavor. I will give your suggestion another try - it would be wonderful to have a working OTA…
Eh revisiting the code a bit, isn’t it a bit suspicious that the ArduinoOTA class has a member
WiFiUDP? …Maybe a hack with #include <Ethernet.h> and replacing WiFiUDP the EthernetUDP class (ref) works?
You should be able to do that as temporary hack in C:\Users\<user>\.platformio\packages\framework-arduinoespressif32\libraries\ArduinoOTA\src\ArduinoOTA.h, given that all the interface functions are the same the code should still work
@maxgerhardt: I changed the ArduinoOTA.h file as you proposed - compile went without issues, but no change, unfortunately.
I am somewhat irritated by the location of the Arduino.h file you hinted me to - there is another in C:\Users\<user>\.platformio\lib\ArduinoOTA_ID6178\src. Most probably this is the one installed with the PlatformIO Library Manager. I experimentally de-installed it, but then got a couple of errors when compiling:
src\main.cpp: In function 'void setup()':
src\main.cpp:1412:50: error: 'InternalStorage' was not declared in this scope
ArduinoOTA.begin(lIP, "myBridge", "xxxx", InternalStorage);
^
src\main.cpp: In function 'void loop()':
src\main.cpp:1747:14: error: 'class ArduinoOTAClass' has no member named 'poll'
ArduinoOTA.poll();
^
Compiling .pio\build\az-delivery-devkit-v4\lib547\ArduinoOTA\ArduinoOTA.cpp.o
*** [.pio\build\az-delivery-devkit-v4\src\main.cpp.o] Error 1
C:\Users\Micha\.platformio\packages\framework-arduinoespressif32\libraries\ArduinoOTA\src\ArduinoOTA.cpp: In member function 'void ArduinoOTAClass::begin()':
C:\Users\Micha\.platformio\packages\framework-arduinoespressif32\libraries\ArduinoOTA\src\ArduinoOTA.cpp:120:9: error: 'WiFi' was not declared in this scope
WiFi.macAddress(mac);
^
C:\Users\Micha\.platformio\packages\framework-arduinoespressif32\libraries\ArduinoOTA\src\ArduinoOTA.cpp: In member function 'void ArduinoOTAClass::_runUpdate()':
C:\Users\Micha\.platformio\packages\framework-arduinoespressif32\libraries\ArduinoOTA\src\ArduinoOTA.cpp:257:5: error: 'WiFiClient' was not declared in this scope
WiFiClient client;
^
C:\Users\Micha\.platformio\packages\framework-arduinoespressif32\libraries\ArduinoOTA\src\ArduinoOTA.cpp:258:10: error: 'client' was not declared in this scope
if (!client.connect(_ota_ip, _ota_port)) {
^
C:\Users\Micha\.platformio\packages\framework-arduinoespressif32\libraries\ArduinoOTA\src\ArduinoOTA.cpp:267:36: error: 'client' was not declared in this scope
while (!Update.isFinished() && client.connected()) {
^
C:\Users\Micha\.platformio\packages\framework-arduinoespressif32\libraries\ArduinoOTA\src\ArduinoOTA.cpp:326:9: error: 'client' was not declared in this scope
client.print("OK");
^
C:\Users\Micha\.platformio\packages\framework-arduinoespressif32\libraries\ArduinoOTA\src\ArduinoOTA.cpp:341:27: error: 'client' was not declared in this scope
Update.printError(client);
^
*** [.pio\build\az-delivery-devkit-v4\lib547\ArduinoOTA\ArduinoOTA.cpp.o] Error 1
I am suspecting that I unintentionally may have mixed up the ArduinoOTA from the core lib with that that I installed with the Library Manager.
By the way: a platformio.ini line as -upload_flags= -p 65280 seems not to be honored, as the debug out put still has the 3232 port: I tried the 65280 port as well, as it is in the code of the downloaded ArduinoOTA library. Not that it had any effect at all…
Seems like this is the library to go. If the library in the framework is troublesome (it unfortunately has the same name…), try doing a
; use bleeding edge version
lib_deps =
ArduinoOTA=https://github.com/jandrassy/ArduinoOTA.git
or temporarily remove it from the framework-arduinoespressif32\libraries folder. It can be brought back any time by simply deleting framework-arduinoespressif32 and recompiling the project (auto redownload).
Thanks again! I missed the bit with “one option per line” for -upload_flags - at least that is sorted out now.
But regarding the basic issue I am back at square one. The library (defined in lib_deps now) was downloaded and installed okay, the sketch compiles without errors or warnings, but still I am getting the notorious
13:03:06 [DEBUG]: Options: {'timeout': 10, 'esp_ip': '192.168.199.40', 'host_port': 49341, 'image': '.pio\\build\\az-delivery-devkit-v4\\firmware.bin', 'host_ip': '0.0.0.0', 'auth': 'xxxx', 'esp_port': 3232, 'spiffs': False, 'debug': True, 'progress': True}
13:03:06 [INFO]: Starting on 0.0.0.0:49341
13:03:06 [INFO]: Upload size: 403200
Sending invitation to 192.168.199.40 ..........
13:04:46 [ERROR]: No response from the ESP
*** [upload] Error 1
Basically the ArduinoOTA library you’ve linked is entirely incompatible with the ArduinoOTA that is provided by default in the Arduino ESP32 package. It uses an entirely different upload mechanism too: The external library opens a HTTP Web Server on a TCP port and does HTTP Basic authentication. It doesn’t work with the espota tool. I use the curl tool (curl for Windows) to push the sketch. espota uses a UDP advertisment custom TCP stream to do the update.
Note that that might not be the best thing to use this custom ArduinoOTA which differs so much from these tools. A modification or expansion request to the Arduino-ESP32 provided ArduinoOTA might still make sense.
@maxgerhardt: thank you very much for your time and effort to help me out!
I tend to refrain from adapting your proprietary solution and rather go to attempt getting the core ArduinoOTA modified. I already opened a bug report yesterday ( https://github.com/espressif/arduino-esp32/issues/4122), but that attracted no reaction yet.
For now I will deactivate the OTA code in my sketch and am hoping a modification of the core will happen in time before my rollout…
YAY! I cannot believe it was that easy in the end! Your information and tips were the clue - I brute-forcely changed all occurrences of WiFi objects in ArduinoOTA.h and ArduinoOTA.cpp with their Ethernet.h complements (leaving out WiFi.mac(), as Ethernet.h seems not to have that) and:
The code change to the core ArduinoOTA will be a few lines only, but I am afraid it will need a preprocessor #define before #including <ArduinoOTA.h> to switch between WiFi and Ethernet.
Great to hear! I think for now you can even make it cleaner by just copying it into a new library, e.g. called ArduinoEthernetOTA or something, with all instances of the old class name ArduinoOTA being changed to new ones. Also adapt the library.properties accordingly. Then put the lib into the lib/ folder. Then you don’t have to modify framework files and have a portable solution.
A really great thing would of course be if you could tell the original ArduinoOTA library which objects to use for the socket, server and mDNS service, just like the old lib does with its templated clases. Templates really make sense here.
There is Ethernet::MACAddress which you can use to save the MAC address into a given 6-byte buffer of yours.
Works like a charm in the local net; my firewall is rejecting the TCP part after the initial UDP communication, though. Another topic anyway, so thank you so much again for your help!
