Good news 
It works now! I don’t know why but after one week of trial and error it works now…
I had to flash it first over serial and since then it works… This is very strange.
EDIT: As you can see in the schematic I forgot to put a decoupling capacitor between ESP 3.3V and GND pins. Although my power supply was stable, this missing capacitor caused the issue that ESP was constantly resetting. So if anyone has a similar issue: Put a decoupling cap close to the ESP32!