ArduinoOTA over Firewall, ESP32: Invitation is failing

Me again… I am trying to integrate ArduinoOTA into my project to be able to update the code even if the device is on-site. My problem is, the device is in another network, behind a firewall. The firewall is configured to pass port 3232 to the device. But OTA updates are failing constantly, the invitation is never answered:

esptool.py v2.6
Advanced Memory Usage is available via "PlatformIO Home > Project Inspect"
RAM:   [=         ]   5.3% (used 28372 bytes from 532480 bytes)
Flash: [===       ]  30.6% (used 400518 bytes from 1310720 bytes)
Configuring upload protocol...
AVAILABLE: esp-prog, espota, esptool, iot-bus-jtag, jlink, minimodule, olimex-arm-usb-ocd, olimex-arm-usb-ocd-h, olimex-arm-usb-tiny-h, olimex-jtag-tiny, tumpa
CURRENT: upload_protocol = espota
Uploading .pio\build\az-delivery-devkit-v4\firmware.bin
18:11:20 [DEBUG]: Options: {'timeout': 10, 'esp_ip': '192.168.178.77', 'host_port': 41297, 'image': '.pio\\build\\az-delivery-devkit-v4\\firmware.bin', 'host_ip': '0.0.0.0', 'auth': 'Nurminnen', 'esp_port': 3232, 'spiffs': False, 'debug': True, 'progress': True}
18:11:20 [INFO]: Starting on 0.0.0.0:41297
18:11:20 [INFO]: Upload size: 400640
Sending invitation to 192.168.178.77 ..........
18:13:00 [ERROR]: No response from the ESP
*** [upload] Error 1

The esp_ip in the log is that of the firewall/gateway, the IP of the device behind it is 192.178.199.40, but is of course not addressable.

Will I need to forward other ports as well? I was under the impression that only 3232 was used. The device can react on requests for port 502 (MODBUS/TCP) without problems, the port forwarding was configuered the same way for 3232. Additionally the device is cyclically gathering NTP time via port 123/UDP with 8888 as the callback port without problems, so outgoing traffic seems to be okay.

So you are addressing the Gateway instead? And 192.168.178.77 is the IP of the gatway?

Also, have you tried building the firmware with debug logs enable to see whether device gets any data at all? See docs, you shou do build_flags = -DCORE_DEBUG_LEVEL=3 or 4.

1 Like

Yes, port 3232 is forwarded by the gateway to the ESP32, so I am addressing the gateway, IP-wise.

I will try the debug option, thanks for the hint.

I tried both -DCORE_DEBUG_LEVEL=3 and =4, but with no noticeable differences. Where am I supposed to look for the debug output?

In another attempt I tried OTA upload within the same subnet, directly addressing the ESP with its IP address 192.168.199.40. The same happened, the upload timed out without any reaction on the invitation. As this subnet is sparsely active, the data LED on the W5500 module may be an indication of traffic going to the device. I saw it flicker each time the “Sending invitation” line got another dot added, so I am assuming the packets were actually sent to the device.

By the way: repeated builds are adding more and more ofthese error lines:

[Error 2] Das System kann die angegebene Datei nicht finden: 'C:\\Users\\Micha\\Documents\\PlatformIO\\Projects\\RS485-Bridge\\.pio\\build\\az-delivery-devkit-v4\\libFrameworkArduinoVariant.a' 
Please manually remove the file `C:\Users\Micha\Documents\PlatformIO\Projects\RS485-Bridge\.pio\build\az-delivery-devkit-v4\libFrameworkArduinoVariant.a`
[Error 3] Das System kann den angegebenen Pfad nicht finden: 'C:\\Users\\Micha\\Documents\\PlatformIO\\Projects\\RS485-Bridge\\.pio\\build\\az-delivery-devkit-v4\\longcmd-1edf16a8dfab01cd43f3613a9b992520'
Please manually remove the file `C:\Users\Micha\Documents\PlatformIO\Projects\RS485-Bridge\.pio\build\az-delivery-devkit-v4\longcmd-1edf16a8dfab01cd43f3613a9b992520`
[Error 3] Das System kann den angegebenen Pfad nicht finden: 'C:\\Users\\Micha\\Documents\\PlatformIO\\Projects\\RS485-Bridge\\.pio\\build\\az-delivery-devkit-v4\\longcmd-6c0073e71529232f49235f36b0dbaf93'
Please manually remove the file `C:\Users\Micha\Documents\PlatformIO\Projects\RS485-Bridge\.pio\build\az-delivery-devkit-v4\longcmd-6c0073e71529232f49235f36b0dbaf93`
[Error 3] Das System kann den angegebenen Pfad nicht finden: 'C:\\Users\\Micha\\Documents\\PlatformIO\\Projects\\RS485-Bridge\\.pio\\build\\az-delivery-devkit-v4\\partitions.bin' 
Please manually remove the file `C:\Users\Micha\Documents\PlatformIO\Projects\RS485-Bridge\.pio\build\az-delivery-devkit-v4\partitions.bin`
[Error 3] Das System kann den angegebenen Pfad nicht finden: 'C:\\Users\\Micha\\Documents\\PlatformIO\\Projects\\RS485-Bridge\\.pio\\build\\az-delivery-devkit-v4\\src' 
Please manually remove the file `C:\Users\Micha\Documents\PlatformIO\Projects\RS485-Bridge\.pio\build\az-delivery-devkit-v4\src`
[Error 2] Das System kann die angegebene Datei nicht finden: 'C:\\Users\\Micha\\Documents\\PlatformIO\\Projects\\RS485-Bridge\\.pio\\build\\az-delivery-devkit-v4' 
Please manually remove the file `C:\Users\Micha\Documents\PlatformIO\Projects\RS485-Bridge\.pio\build\az-delivery-devkit-v4`

Once I removed the directories as indicated, the next build ran without these errors.

Final observation: after having run for more than a week without problems, after adding the ArduinoOTA code yesterday (and not changing anything else) I had a complete freeze today without the ESP starting itself again. I had to cycle power to get it running again. :thinking:

@maxgerhardt? Any other hints, please? :wink:

I am really stuck here - I almost for sure can sort out firewall issues, as other ports are forwarded without problems that are configured identically.

My main suspects at the moment are:

  • Wrong ArduinoOTA library - but there is only one available?!?
  • ArduinoOTA cannot cope with IP+port only, is missing its host name in the requests
  • needs to have the broadcast mDNS packets answered - that will not pass the firewall, so no one will do that

The device will have to be mounted into a DIN rail closet at the customer’s site, so going there, unmount it, flash it and mount it back in is not an option.

On a side note, Juraj Andrassi (maintainer of ArduinoOTA on Github) closed my issue with this rather disappointing comment:

platformio is not Arduino. they change things and lose compatibility. I don’t use platformio and I don’t plan to support it

So we are left to our own devices here.

Try and sort things out when being on the same network, first. That you’re doing it with a W5500 Ethernet chip in the middle is a pretty important bit of information that wasn’t mentioned at first.

Does it also not work with the Arduino IDE? Have you made sure to confirm the functionality of the W5500 chip first with a test sketch? What code are you using to set up Ethernet + OTA? Does increasing the debug level make any ingoing packets visible?

I did not try that, I may have an outdated Arduino IDE installed somewhere but I never used it for the ESP32. The sketch I am working on has some 3000+ lines of code now, so I rather would write a test sketch instead. I honestly will try that as a last resort only, given the efforts.

Yes, it is working fine. I am doing NTP requests regularly and the ESP is properly reacting on TCP requests on port 502, so the module seems to work and the network settings are okay.

Excerpt from my code:

...
// SPI and Ethernet libs for W5500 Ethernet module
#include <SPI.h>
#include <IPAddress.h>
#include <Ethernet.h>
// W5500 reset pin 
#define RESET_P GPIO_NUM_26
...
// OTA update interface
#include <ArduinoOTA.h>
...
// Server on port 502 (MODBUS)
EthernetServer server(IPport);
...
// A UDP instance to let us send and receive packets over UDP
EthernetUDP Udp;
...
void setup()
{
...
// SPI CS line is GPIO_NUM_5
  Ethernet.init(GPIO_NUM_5);
  // Hard boot W5500 module
  WizReset();
...
  // start the Ethernet connection:
  bool connected = false;
  if(IPDHCP) // DHCP required?
  {
    Trace(0, "Trying to get an IP address using DHCP\n");
    if (Ethernet.begin(mac) == 0) 
    {
      Trace(0, "Failed to configure Ethernet using DHCP\n");
    }
    else connected = true;
  }
  // Connection established?
  if(!connected)
  {
    // No. Either DHCP was not required or did not work.
    // initialize the Ethernet device not using DHCP:
    Ethernet.begin(mac, Reverse(IPaddress), Reverse(IPDNS), Reverse(IPgateway), Reverse(IPmask));
    // Check for Ethernet hardware present
    if (Ethernet.hardwareStatus() == EthernetNoHardware) 
    {
      Trace(0, "Ethernet shield was not found.  Sorry, can't run without hardware. :(\n");
    }
    else
    {
      if (Ethernet.linkStatus() == LinkOFF) 
      {
        Trace(0, "Ethernet cable is not connected.\n");
      }
      else connected = true;
    }
  }

  // print local IP address:
  lIP = Ethernet.localIP();
  Trace(TRACE_BASIC, "My IP address: %u.%u.%u.%u\n", lIP[0], lIP[1], lIP[2], lIP[3]);

  // Update server to configured port
  if(IPport) server = EthernetServer(IPport);
  // Now start it
  server.begin();
...
// Set up UDP for NTP requests
  setenv("TZ", NTP_TZ, 1);
  tzset();
  Udp.begin(localPort);
...
  // Start ArduinoOTA service
  ArduinoOTA.begin(lIP, "myBridge", "xxxx", InternalStorage);
}
...
void loop()
{
...
 // Check for OTA updates
  ArduinoOTA.poll();
...
}

Not at all, as I wrote above. The flag does not have a single effect, that is why I asked where to look for the additional output.

For the time being I give up now. I could not find a reason why the OTA invitation was not answered. The packets looked good and were sent to the W5500 module, but inside the ESP32 nothing was happening.

So you did try the minimal sketch with just enough added on to initialize the W5500 interface? I think the issue here is that when the OTA library listens on port 3232, it does so on the general WiFi (or hardware ethernet interface), but it doesn’t know anything about the W5500 chip attached via SPI? There may be some extensions that must be done to the network stack there. I’m sure people at Issues · espressif/arduino-esp32 · GitHub would know more.

The ArduinoOTA library is claiming to work with Ethernet.h, but you may be right and that is only meant for the “real Arduino” flavor. I will give your suggestion another try - it would be wonderful to have a working OTA…

Eh revisiting the code a bit, isn’t it a bit suspicious that the ArduinoOTA class has a member

WiFiUDP? …Maybe a hack with #include <Ethernet.h> and replacing WiFiUDP the EthernetUDP class (ref) works?

You should be able to do that as temporary hack in C:\Users\<user>\.platformio\packages\framework-arduinoespressif32\libraries\ArduinoOTA\src\ArduinoOTA.h, given that all the interface functions are the same the code should still work

@maxgerhardt: I changed the ArduinoOTA.h file as you proposed - compile went without issues, but no change, unfortunately.

I am somewhat irritated by the location of the Arduino.h file you hinted me to - there is another in C:\Users\<user>\.platformio\lib\ArduinoOTA_ID6178\src. Most probably this is the one installed with the PlatformIO Library Manager. I experimentally de-installed it, but then got a couple of errors when compiling:

src\main.cpp: In function 'void setup()':
src\main.cpp:1412:50: error: 'InternalStorage' was not declared in this scope
   ArduinoOTA.begin(lIP, "myBridge", "xxxx", InternalStorage);
                                                  ^
src\main.cpp: In function 'void loop()':
src\main.cpp:1747:14: error: 'class ArduinoOTAClass' has no member named 'poll'
   ArduinoOTA.poll();
              ^
Compiling .pio\build\az-delivery-devkit-v4\lib547\ArduinoOTA\ArduinoOTA.cpp.o
*** [.pio\build\az-delivery-devkit-v4\src\main.cpp.o] Error 1
C:\Users\Micha\.platformio\packages\framework-arduinoespressif32\libraries\ArduinoOTA\src\ArduinoOTA.cpp: In member function 'void ArduinoOTAClass::begin()':
C:\Users\Micha\.platformio\packages\framework-arduinoespressif32\libraries\ArduinoOTA\src\ArduinoOTA.cpp:120:9: error: 'WiFi' was not declared in this scope
         WiFi.macAddress(mac);
         ^
C:\Users\Micha\.platformio\packages\framework-arduinoespressif32\libraries\ArduinoOTA\src\ArduinoOTA.cpp: In member function 'void ArduinoOTAClass::_runUpdate()':
C:\Users\Micha\.platformio\packages\framework-arduinoespressif32\libraries\ArduinoOTA\src\ArduinoOTA.cpp:257:5: error: 'WiFiClient' was not declared in this scope
     WiFiClient client;
     ^
C:\Users\Micha\.platformio\packages\framework-arduinoespressif32\libraries\ArduinoOTA\src\ArduinoOTA.cpp:258:10: error: 'client' was not declared in this scope
     if (!client.connect(_ota_ip, _ota_port)) {
          ^
C:\Users\Micha\.platformio\packages\framework-arduinoespressif32\libraries\ArduinoOTA\src\ArduinoOTA.cpp:267:36: error: 'client' was not declared in this scope
     while (!Update.isFinished() && client.connected()) {
                                    ^
C:\Users\Micha\.platformio\packages\framework-arduinoespressif32\libraries\ArduinoOTA\src\ArduinoOTA.cpp:326:9: error: 'client' was not declared in this scope
         client.print("OK");
         ^
C:\Users\Micha\.platformio\packages\framework-arduinoespressif32\libraries\ArduinoOTA\src\ArduinoOTA.cpp:341:27: error: 'client' was not declared in this scope
         Update.printError(client);
                           ^
*** [.pio\build\az-delivery-devkit-v4\lib547\ArduinoOTA\ArduinoOTA.cpp.o] Error 1

I am suspecting that I unintentionally may have mixed up the ArduinoOTA from the core lib with that that I installed with the Library Manager.

By the way: a platformio.ini line as -upload_flags= -p 65280 seems not to be honored, as the debug out put still has the 3232 port: I tried the 65280 port as well, as it is in the code of the downloaded ArduinoOTA library. Not that it had any effect at all… :frowning:

12:29:00 [DEBUG]: Options: {'timeout': 10, 'esp_ip': '192.168.199.40', 'host_port': 26596, 'image': '.pio\\build\\az-delivery-devkit-v4\\firmware.bin', 'host_ip': '0.0.0.0', 'auth': 'Nurminnen', 'esp_port': 3232, 'spiffs': False, 'debug': True, 'progress': True}

Try to follow the exact sme syntax as here

https://docs.platformio.org/en/latest/platforms/espressif32.html#authentication-and-upload-options

Oh now that’s interesting. So there exists an external ArduinoOTA library with the same name… And it has much more explicit support for Ethernet OTA: ArduinoOTA/examples/OTEthernet/OTEthernet.ino at master · JAndrassy/ArduinoOTA · GitHub

Also it directly says that the ESP32 can only do it via WiFi: GitHub - JAndrassy/ArduinoOTA: Arduino library to upload sketch over network to Arduino board with WiFi or Ethernet libraries

Seems like this is the library to go. If the library in the framework is troublesome (it unfortunately has the same name…), try doing a

; use bleeding edge version
lib_deps = 
   ArduinoOTA=https://github.com/jandrassy/ArduinoOTA.git

or temporarily remove it from the framework-arduinoespressif32\libraries folder. It can be brought back any time by simply deleting framework-arduinoespressif32 and recompiling the project (auto redownload).

Thanks again! I missed the bit with “one option per line” for -upload_flags - at least that is sorted out now.

But regarding the basic issue I am back at square one. The library (defined in lib_deps now) was downloaded and installed okay, the sketch compiles without errors or warnings, but still I am getting the notorious

13:03:06 [DEBUG]: Options: {'timeout': 10, 'esp_ip': '192.168.199.40', 'host_port': 49341, 'image': '.pio\\build\\az-delivery-devkit-v4\\firmware.bin', 'host_ip': '0.0.0.0', 'auth': 'xxxx', 'esp_port': 3232, 'spiffs': False, 'debug': True, 'progress': True}
13:03:06 [INFO]: Starting on 0.0.0.0:49341
13:03:06 [INFO]: Upload size: 403200
Sending invitation to 192.168.199.40 ..........
13:04:46 [ERROR]: No response from the ESP
*** [upload] Error 1

Okay, that took me some time to figure out…

Basically the ArduinoOTA library you’ve linked is entirely incompatible with the ArduinoOTA that is provided by default in the Arduino ESP32 package. It uses an entirely different upload mechanism too: The external library opens a HTTP Web Server on a TCP port and does HTTP Basic authentication. It doesn’t work with the espota tool. I use the curl tool (curl for Windows) to push the sketch. espota uses a UDP advertisment custom TCP stream to do the update.

Finally issue Info: PlatformIO, Arduino Mega, build & upload OTA · Issue #29 · JAndrassy/ArduinoOTA · GitHub helped my get on the right track, although with some modifications.

I now have a fully working OTA setup with my ESP32 + W5500: GitHub - maxgerhardt/pio-esp32-ethernet-ota: Demo with a ESP32 + Ethernet + OTA

Note that that might not be the best thing to use this custom ArduinoOTA which differs so much from these tools. A modification or expansion request to the Arduino-ESP32 provided ArduinoOTA might still make sense.

@maxgerhardt: thank you very much for your time and effort to help me out!

I tend to refrain from adapting your proprietary solution and rather go to attempt getting the core ArduinoOTA modified. I already opened a bug report yesterday ( https://github.com/espressif/arduino-esp32/issues/4122), but that attracted no reaction yet.

For now I will deactivate the OTA code in my sketch and am hoping a modification of the core will happen in time before my rollout… :wink:

YAY! I cannot believe it was that easy in the end! Your information and tips were the clue - I brute-forcely changed all occurrences of WiFi objects in ArduinoOTA.h and ArduinoOTA.cpp with their Ethernet.h complements (leaving out WiFi.mac(), as Ethernet.h seems not to have that) and:

Configuring upload protocol...
AVAILABLE: esp-prog, espota, esptool, iot-bus-jtag, jlink, minimodule, olimex-arm-usb-ocd, olimex-arm-usb-ocd-h, olimex-arm-usb-tiny-h, olimex-jtag-tiny, tumpa
CURRENT: upload_protocol = espota
Uploading .pio\build\az-delivery-devkit-v4\firmware.bin
15:16:50 [DEBUG]: Options: {'timeout': 10, 'esp_ip': '192.168.199.40', 'host_port': 30605, 'image': '.pio\\build\\az-delivery-devkit-v4\\firmware.bin', 'host_ip': '0.0.0.0', 'auth': 'Nurminnen', 'esp_port': 3232, 'spiffs': False, 'debug': True, 'progress': True}
15:16:50 [INFO]: Starting on 0.0.0.0:30605
15:16:50 [INFO]: Upload size: 494768
Sending invitation to 192.168.199.40 
Authenticating...OK
15:16:50 [INFO]: Waiting for device...
Uploading: [============================================================] 100% Done...

15:16:57 [INFO]: Waiting for result...
15:16:58 [INFO]: Result: OK
15:16:58 [INFO]: Success
=================================================================================== [SUCCESS] Took 60.43 seconds ===================================================================================

The code change to the core ArduinoOTA will be a few lines only, but I am afraid it will need a preprocessor #define before #including <ArduinoOTA.h> to switch between WiFi and Ethernet.

Great to hear! I think for now you can even make it cleaner by just copying it into a new library, e.g. called ArduinoEthernetOTA or something, with all instances of the old class name ArduinoOTA being changed to new ones. Also adapt the library.properties accordingly. Then put the lib into the lib/ folder. Then you don’t have to modify framework files and have a portable solution.

A really great thing would of course be if you could tell the original ArduinoOTA library which objects to use for the socket, server and mDNS service, just like the old lib does with its templated clases. Templates really make sense here.

There is Ethernet::MACAddress which you can use to save the MAC address into a given 6-byte buffer of yours.

I named the lib ArduinoOTE :slight_smile:

Works like a charm in the local net; my firewall is rejecting the TCP part after the initial UDP communication, though. Another topic anyway, so thank you so much again for your help!