Very strange behavior when debugging using ESP-prog

I am using ESP-prog to upload and debug my ESP32 board.
But when I setup a break point, it often hit the line that is a few lines up or below.
When I clicked continue it sometimes hit a invisible break point.
Most strange of all, when debugging it crashes my board (stack overflow), when not debugging the same situation is ok.

Does any one know what could be the problem?

Is your build_type set to debug in platform.ini?

Yes, I tried debug build, it makes no difference.
I changed the USB cable re-wiring everything, the problem still exists.
The crash problem is that a thread I created is given too small stack size. It has nothing to do with the debugger.
Currently I almost gave up on the debugger. The step over function often step over random lines instead of just one.