Esp32 debug - not stopping at source

latonita · September 23, 2018, 4:21pm

Hi,

I cant get debugger working properly. it stopped once or twice at app_main, but now it never stops in source code so I can’t debug it step by step.
it tells that it stopped, but its not

my setup: ubuntu 18.04, vscode, esp32dev module, CJMCU-2232 breakout board.

FTDI AD0 -> ESP32 GPIO13 (TCK)
FTDI AD1 -> ESP32 GPIO12 (TDI)
FTDI AD2 -> ESP32 GPIO15 (TDO)
FTDI AD3 -> ESP32 GPIO14 (TMS)
FTDI AC2 -> ESP32 EN (RST)
FTDI GND -> ESP32 GND
FTDI 3V3 -> ESP32 3V3

platformio.ini:

[env:esp32dev]
platform = espressif32
board = esp32dev
framework = arduino
debug_tool = minimodule

here is log from debug window. in the end i clicked pause and it doesnt show me where it stopped. step over, step into not working as well.

Reading symbols from /home/user/Documents/PlatformIO/Projects/esp32-blink/.pioenvs/esp32dev/firmware.elf...
done.
PlatformIO Unified Debugger > http://bit.ly/pio-debug
PlatformIO: Initializing remote target...
Open On-Chip Debugger 0.10.0-dev (2018-06-04-09:50)
Licensed under GNU GPL v2
For bug reports, read
	http://openocd.org/doc/doxygen/bugs.html
adapter speed: 20000 kHz
esp32 interrupt mask on
force hard breakpoints
Info : tcl server disabled
Info : telnet server disabled
Info : ftdi: if you experience problems at higher adapter clocks, try the command "ftdi_tdo_sample_edge falling"
Info : clock speed 20000 kHz
Info : JTAG tap: esp32.cpu0 tap/device found: 0x120034e5 (mfg: 0x272 (Tensilica), part: 0x2003, ver: 0x1)
Info : JTAG tap: esp32.cpu1 tap/device found: 0x120034e5 (mfg: 0x272 (Tensilica), part: 0x2003, ver: 0x1)
Info : esp32: Debug controller was reset (pwrstat=0x5F, after clear 0x0F).
Info : esp32: Core was reset (pwrstat=0x5F, after clear 0x0F).
cpu0: Current bits set: BreakIn BreakOut RunStallIn
cpu1: Current bits set: none
Info : accepting 'gdb' connection from pipe
Info : Target halted. PRO_CPU: PC=0x40009394 (active)    APP_CPU: PC=0x00000000 
esp32: target state: halted
Info : Use core0 of target 'esp32'
Info : Target halted. PRO_CPU: PC=0x40091D07 (active)    APP_CPU: PC=0x00000000 
Info : Auto-detected flash size 4096 KB
Info : Using flash size 4096 KB
0x40009394 in ?? ()
Not implemented stop reason (assuming exception): undefined
Error: Too large number of threads 19205715!
Hardware assisted breakpoint 1 at 0x401207eb: file /home/user/.platformio/packages/framework-arduinoespressif32/cores/esp32/main.cpp, line 21.
Info : JTAG tap: esp32.cpu0 tap/device found: 0x120034e5 (mfg: 0x272 (Tensilica), part: 0x2003, ver: 0x1)
Info : JTAG tap: esp32.cpu1 tap/device found: 0x120034e5 (mfg: 0x272 (Tensilica), part: 0x2003, ver: 0x1)
JTAG tap: esp32.cpu0 tap/device found: 0x120034e5 (mfg: 0x272 (Tensilica), part: 0x2003, ver: 0x1)
cpu0: Current bits set: BreakIn BreakOut RunStallIn
JTAG tap: esp32.cpu1 tap/device found: 0x120034e5 (mfg: 0x272 (Tensilica), part: 0x2003, ver: 0x1)
cpu1: Current bits set: none
cpu0: Current bits set: BreakIn BreakOut RunStallIn
cpu1: Current bits set: none
Info : esp32: Debug controller was reset (pwrstat=0x5F, after clear 0x0F).
Info : esp32: Core was reset (pwrstat=0x5F, after clear 0x0F).
esp32: Debug controller was reset (pwrstat=0x5F, after clear 0x0F).
esp32: Core was reset (pwrstat=0x5F, after clear 0x0F).
Info : Target halted. PRO_CPU: PC=0x5000004B (active)    APP_CPU: PC=0x00000000 
Target halted. PRO_CPU: PC=0x5000004B (active)    APP_CPU: PC=0x00000000 
esp32: target state: halted
esp32: target state: halted
Info : esp32: Core was reset (pwrstat=0x1F, after clear 0x0F).
esp32: Core was reset (pwrstat=0x1F, after clear 0x0F).
Info : Target halted. PRO_CPU: PC=0x40000400 (active)    APP_CPU: PC=0x40000400 
Target halted. PRO_CPU: PC=0x40000400 (active)    APP_CPU: PC=0x40000400 
esp32: target state: halted
esp32: target state: halted
** Programming Started **
** Programming Started **
auto erase enabled
auto erase enabled
Info : Use core0 of target 'esp32'
Use core0 of target 'esp32'
Info : Target halted. PRO_CPU: PC=0x40091D07 (active)    APP_CPU: PC=0x400076DD 
Target halted. PRO_CPU: PC=0x40091D07 (active)    APP_CPU: PC=0x400076DD 
Info : Use core0 of target 'esp32'
Use core0 of target 'esp32'
Info : Target halted. PRO_CPU: PC=0x40091D07 (active)    APP_CPU: PC=0x400076DD 
Target halted. PRO_CPU: PC=0x40091D07 (active)    APP_CPU: PC=0x400076DD 
wrote 16384 bytes from file /home/user/.platformio/packages/framework-arduinoespressif32/tools/sdk/bin/bootloader_dio_40m.bin in 0.817486s (19.572 KiB/s)
wrote 16384 bytes from file /home/user/.platformio/packages/framework-arduinoespressif32/tools/sdk/bin/bootloader_dio_40m.bin in 0.817486s (19.572 KiB/s)
** Programming Finished **
** Programming Finished **
** Verify Started **
** Verify Started **
Info : Use core0 of target 'esp32'
Use core0 of target 'esp32'
Info : Target halted. PRO_CPU: PC=0x40091D07 (active)    APP_CPU: PC=0x400076E2 
Target halted. PRO_CPU: PC=0x40091D07 (active)    APP_CPU: PC=0x400076E2 
read 15088 bytes from file /home/user/.platformio/packages/framework-arduinoespressif32/tools/sdk/bin/bootloader_dio_40m.bin and flash bank 0 at offset 0x00001000 in 0.304486s (48.391 KiB/s)
contents match
read 15088 bytes from file /home/user/.platformio/packages/framework-arduinoespressif32/tools/sdk/bin/bootloader_dio_40m.bin and flash bank 0 at offset 0x00001000 in 0.304486s (48.391 KiB/s)
contents match
** Verified OK **
** Verified OK **
Info : JTAG tap: esp32.cpu0 tap/device found: 0x120034e5 (mfg: 0x272 (Tensilica), part: 0x2003, ver: 0x1)
Info : JTAG tap: esp32.cpu1 tap/device found: 0x120034e5 (mfg: 0x272 (Tensilica), part: 0x2003, ver: 0x1)
JTAG tap: esp32.cpu0 tap/device found: 0x120034e5 (mfg: 0x272 (Tensilica), part: 0x2003, ver: 0x1)
cpu0: Current bits set: BreakIn BreakOut RunStallIn
cpu1: Current bits set: BreakIn BreakOut RunStallIn
JTAG tap: esp32.cpu1 tap/device found: 0x120034e5 (mfg: 0x272 (Tensilica), part: 0x2003, ver: 0x1)
cpu0: Current bits set: BreakIn BreakOut RunStallIn
cpu1: Current bits set: BreakIn BreakOut RunStallIn
Info : esp32: Debug controller was reset (pwrstat=0x5F, after clear 0x0F).
Info : esp32: Core was reset (pwrstat=0x5F, after clear 0x0F).
esp32: Debug controller was reset (pwrstat=0x5F, after clear 0x0F).
esp32: Core was reset (pwrstat=0x5F, after clear 0x0F).
Info : Target halted. PRO_CPU: PC=0x5000004B (active)    APP_CPU: PC=0x00000000 
Target halted. PRO_CPU: PC=0x5000004B (active)    APP_CPU: PC=0x00000000 
esp32: target state: halted
esp32: target state: halted
Info : esp32: Core was reset (pwrstat=0x1F, after clear 0x0F).
esp32: Core was reset (pwrstat=0x1F, after clear 0x0F).
Info : Target halted. PRO_CPU: PC=0x40000400 (active)    APP_CPU: PC=0x40000400 
Target halted. PRO_CPU: PC=0x40000400 (active)    APP_CPU: PC=0x40000400 
esp32: target state: halted
esp32: target state: halted
** Programming Started **
** Programming Started **
auto erase enabled
auto erase enabled
Info : Use core0 of target 'esp32'
Use core0 of target 'esp32'
Info : Target halted. PRO_CPU: PC=0x40091D07 (active)    APP_CPU: PC=0x400076E2 
Target halted. PRO_CPU: PC=0x40091D07 (active)    APP_CPU: PC=0x400076E2 
Info : Use core0 of target 'esp32'
Use core0 of target 'esp32'
Info : Target halted. PRO_CPU: PC=0x40091D07 (active)    APP_CPU: PC=0x400076DD 
Target halted. PRO_CPU: PC=0x40091D07 (active)    APP_CPU: PC=0x400076DD 
wrote 4096 bytes from file /home/user/Documents/PlatformIO/Projects/esp32-blink/.pioenvs/esp32dev/partitions.bin in 0.618787s (6.464 KiB/s)
wrote 4096 bytes from file /home/user/Documents/PlatformIO/Projects/esp32-blink/.pioenvs/esp32dev/partitions.bin in 0.618787s (6.464 KiB/s)
** Programming Finished **
** Programming Finished **
** Verify Started **
** Verify Started **
Info : Use core0 of target 'esp32'
Use core0 of target 'esp32'
Info : Target halted. PRO_CPU: PC=0x40091D07 (active)    APP_CPU: PC=0x400076DD 
Target halted. PRO_CPU: PC=0x40091D07 (active)    APP_CPU: PC=0x400076DD 
read 3072 bytes from file /home/user/Documents/PlatformIO/Projects/esp32-blink/.pioenvs/esp32dev/partitions.bin and flash bank 0 at offset 0x00008000 in 0.261835s (11.458 KiB/s)
contents match
** Verified OK **
read 3072 bytes from file /home/user/Documents/PlatformIO/Projects/esp32-blink/.pioenvs/esp32dev/partitions.bin and flash bank 0 at offset 0x00008000 in 0.261835s (11.458 KiB/s)
contents match
** Verified OK **
Info : JTAG tap: esp32.cpu0 tap/device found: 0x120034e5 (mfg: 0x272 (Tensilica), part: 0x2003, ver: 0x1)
Info : JTAG tap: esp32.cpu1 tap/device found: 0x120034e5 (mfg: 0x272 (Tensilica), part: 0x2003, ver: 0x1)
JTAG tap: esp32.cpu0 tap/device found: 0x120034e5 (mfg: 0x272 (Tensilica), part: 0x2003, ver: 0x1)
cpu0: Current bits set: BreakIn BreakOut RunStallIn
JTAG tap: esp32.cpu1 tap/device found: 0x120034e5 (mfg: 0x272 (Tensilica), part: 0x2003, ver: 0x1)
cpu1: Current bits set: BreakIn BreakOut RunStallIn
cpu0: Current bits set: BreakIn BreakOut RunStallIn
cpu1: Current bits set: BreakIn BreakOut RunStallIn
Info : esp32: Debug controller was reset (pwrstat=0x5F, after clear 0x0F).
Info : esp32: Core was reset (pwrstat=0x5F, after clear 0x0F).
esp32: Debug controller was reset (pwrstat=0x5F, after clear 0x0F).
esp32: Core was reset (pwrstat=0x5F, after clear 0x0F).
Info : Target halted. PRO_CPU: PC=0x5000004B (active)    APP_CPU: PC=0x00000000 
Target halted. PRO_CPU: PC=0x5000004B (active)    APP_CPU: PC=0x00000000 
esp32: target state: halted
esp32: target state: halted
Info : esp32: Core was reset (pwrstat=0x1F, after clear 0x0F).
esp32: Core was reset (pwrstat=0x1F, after clear 0x0F).
Info : Target halted. PRO_CPU: PC=0x40000400 (active)    APP_CPU: PC=0x40000400 
Target halted. PRO_CPU: PC=0x40000400 (active)    APP_CPU: PC=0x40000400 
esp32: target state: halted
esp32: target state: halted
** Programming Started **
** Programming Started **
auto erase enabled
auto erase enabled
Info : Use core0 of target 'esp32'
Use core0 of target 'esp32'
Info : Target halted. PRO_CPU: PC=0x40091D07 (active)    APP_CPU: PC=0x400076DD 
Target halted. PRO_CPU: PC=0x40091D07 (active)    APP_CPU: PC=0x400076DD 
Info : Use core0 of target 'esp32'
Use core0 of target 'esp32'
Info : Target halted. PRO_CPU: PC=0x40091D07 (active)    APP_CPU: PC=0x400076E2 
Target halted. PRO_CPU: PC=0x40091D07 (active)    APP_CPU: PC=0x400076E2 
wrote 8192 bytes from file /home/user/.platformio/packages/framework-arduinoespressif32/tools/partitions/boot_app0.bin in 0.686053s (11.661 KiB/s)
wrote 8192 bytes from file /home/user/.platformio/packages/framework-arduinoespressif32/tools/partitions/boot_app0.bin in 0.686053s (11.661 KiB/s)
** Programming Finished **
** Programming Finished **
** Verify Started **
** Verify Started **
Info : Use core0 of target 'esp32'
Use core0 of target 'esp32'
Info : Target halted. PRO_CPU: PC=0x40091D07 (active)    APP_CPU: PC=0x400076DD 
Target halted. PRO_CPU: PC=0x40091D07 (active)    APP_CPU: PC=0x400076DD 
read 8192 bytes from file /home/user/.platformio/packages/framework-arduinoespressif32/tools/partitions/boot_app0.bin and flash bank 0 at offset 0x0000e000 in 0.291061s (27.486 KiB/s)
contents match
read 8192 bytes from file /home/user/.platformio/packages/framework-arduinoespressif32/tools/partitions/boot_app0.bin and flash bank 0 at offset 0x0000e000 in 0.291061s (27.486 KiB/s)
** Verified OK **
contents match
** Verified OK **
Info : JTAG tap: esp32.cpu0 tap/device found: 0x120034e5 (mfg: 0x272 (Tensilica), part: 0x2003, ver: 0x1)
Info : JTAG tap: esp32.cpu1 tap/device found: 0x120034e5 (mfg: 0x272 (Tensilica), part: 0x2003, ver: 0x1)
cpu0: Current bits set: BreakIn BreakOut RunStallIn
cpu1: Current bits set: BreakIn BreakOut RunStallIn
JTAG tap: esp32.cpu0 tap/device found: 0x120034e5 (mfg: 0x272 (Tensilica), part: 0x2003, ver: 0x1)
JTAG tap: esp32.cpu1 tap/device found: 0x120034e5 (mfg: 0x272 (Tensilica), part: 0x2003, ver: 0x1)
cpu0: Current bits set: BreakIn BreakOut RunStallIn
cpu1: Current bits set: BreakIn BreakOut RunStallIn
Info : esp32: Debug controller was reset (pwrstat=0x5F, after clear 0x0F).
Info : esp32: Core was reset (pwrstat=0x5F, after clear 0x0F).
esp32: Debug controller was reset (pwrstat=0x5F, after clear 0x0F).
esp32: Core was reset (pwrstat=0x5F, after clear 0x0F).
Info : Target halted. PRO_CPU: PC=0x5000004B (active)    APP_CPU: PC=0x00000000 
Target halted. PRO_CPU: PC=0x5000004B (active)    APP_CPU: PC=0x00000000 
esp32: target state: halted
esp32: target state: halted
Info : esp32: Core was reset (pwrstat=0x1F, after clear 0x0F).
esp32: Core was reset (pwrstat=0x1F, after clear 0x0F).
Info : Target halted. PRO_CPU: PC=0x40000400 (active)    APP_CPU: PC=0x40000400 
Target halted. PRO_CPU: PC=0x40000400 (active)    APP_CPU: PC=0x40000400 
esp32: target state: halted
esp32: target state: halted
** Programming Started **
** Programming Started **
auto erase enabled
auto erase enabled
Info : Use core0 of target 'esp32'
Use core0 of target 'esp32'
Info : Target halted. PRO_CPU: PC=0x40091D07 (active)    APP_CPU: PC=0x400076DD 
Target halted. PRO_CPU: PC=0x40091D07 (active)    APP_CPU: PC=0x400076DD 
Info : Use core0 of target 'esp32'
Use core0 of target 'esp32'
Info : Target halted. PRO_CPU: PC=0x40091D07 (active)    APP_CPU: PC=0x400076DD 
Target halted. PRO_CPU: PC=0x40091D07 (active)    APP_CPU: PC=0x400076DD 
wrote 688128 bytes from file /home/user/Documents/PlatformIO/Projects/esp32-blink/.pioenvs/esp32dev/firmware.bin in 5.302926s (126.722 KiB/s)
wrote 688128 bytes from file /home/user/Documents/PlatformIO/Projects/esp32-blink/.pioenvs/esp32dev/firmware.bin in 5.302926s (126.722 KiB/s)
** Programming Finished **
** Programming Finished **
** Verify Started **
** Verify Started **
Info : Use core0 of target 'esp32'
Use core0 of target 'esp32'
Info : Target halted. PRO_CPU: PC=0x40091D07 (active)    APP_CPU: PC=0x400076DD 
Target halted. PRO_CPU: PC=0x40091D07 (active)    APP_CPU: PC=0x400076DD 
read 686224 bytes from file /home/user/Documents/PlatformIO/Projects/esp32-blink/.pioenvs/esp32dev/firmware.bin and flash bank 0 at offset 0x00010000 in 2.643653s (253.490 KiB/s)
contents match
read 686224 bytes from file /home/user/Documents/PlatformIO/Projects/esp32-blink/.pioenvs/esp32dev/firmware.bin and flash bank 0 at offset 0x00010000 in 2.643653s (253.490 KiB/s)
contents match
** Verified OK **
** Verified OK **
Info : JTAG tap: esp32.cpu0 tap/device found: 0x120034e5 (mfg: 0x272 (Tensilica), part: 0x2003, ver: 0x1)
Info : JTAG tap: esp32.cpu1 tap/device found: 0x120034e5 (mfg: 0x272 (Tensilica), part: 0x2003, ver: 0x1)
JTAG tap: esp32.cpu0 tap/device found: 0x120034e5 (mfg: 0x272 (Tensilica), part: 0x2003, ver: 0x1)
cpu0: Current bits set: BreakIn BreakOut RunStallIn
cpu1: Current bits set: BreakIn BreakOut RunStallIn
JTAG tap: esp32.cpu1 tap/device found: 0x120034e5 (mfg: 0x272 (Tensilica), part: 0x2003, ver: 0x1)
cpu0: Current bits set: BreakIn BreakOut RunStallIn
cpu1: Current bits set: BreakIn BreakOut RunStallIn
Info : esp32: Debug controller was reset (pwrstat=0x5F, after clear 0x0F).
Info : esp32: Core was reset (pwrstat=0x5F, after clear 0x0F).
esp32: Debug controller was reset (pwrstat=0x5F, after clear 0x0F).
esp32: Core was reset (pwrstat=0x5F, after clear 0x0F).
Info : Target halted. PRO_CPU: PC=0x5000004B (active)    APP_CPU: PC=0x00000000 
Target halted. PRO_CPU: PC=0x5000004B (active)    APP_CPU: PC=0x00000000 
esp32: target state: halted
esp32: target state: halted
Info : esp32: Core was reset (pwrstat=0x1F, after clear 0x0F).
esp32: Core was reset (pwrstat=0x1F, after clear 0x0F).
Info : Target halted. PRO_CPU: PC=0x40000400 (active)    APP_CPU: PC=0x40000400 
Target halted. PRO_CPU: PC=0x40000400 (active)    APP_CPU: PC=0x40000400 
esp32: target state: halted
esp32: target state: halted
PlatformIO: Initialization completed
PlatformIO: Resume the execution to `debug_init_break = thb app_main`
Info : Target halted. PRO_CPU: PC=0x40007A2E (active)    APP_CPU: PC=0x400076DD 
Target halted. PRO_CPU: PC=0x40007A2E (active)    APP_CPU: PC=0x400076DD 

Program received signal 
SIGINT, Interrupt.
0x40007a2e in ?? ()

latonita · September 23, 2018, 5:18pm

Maybe I don’t understand something. It started working again after i flashed another project and returned back to try esp32-blink again… What is proper procedure to have stable results?

Question number 2. I noticed step over/step into not working properly sometimes. either not going over and going into some assembler code instead or jumping over several lines.
just to be sure i dont have any optimizations turned on i tried this, hopefully this will help

build_unflags = 
  -Os 
  -Og
build_flags=
  -O0

ivankravets · September 25, 2018, 7:50pm

Could you reproduce this issue again?

Could you remove these lines? PlatformIO builds project in debug mode automatically for you.

latonita · September 26, 2018, 11:15pm

Yes i can reproduce this issue.
However I cant find consistency - sometimes it is stopping, sometimes not. For example its been non functional since yesterday.

I removed build flags and unflags. no changes to behaviour.

Here is what helps me - i work on bigger project, if im lucky - i can debug it. if debug fails to stop at breakpoint - i have to close project, open another one (for example simple esp32-blink), debug it (most likely it will stop at app_main properly), then close and reopen my project. debug mode will start working with 80% probability.
p.s. restarting vscode doesnt help

ivankravets · October 4, 2018, 4:24pm

Could you try debug_load_mode = preload? It slower but use another method to flash a debugging firmware.

latonita · October 7, 2018, 1:01pm

Tried this. no changes. back to situation when debugger not working at all.
And I have same issue, like here ESP32 & PIO Unified Debugger - #64 by ivankravets
I run blink example. just added serial output there, no other changes.
I start debug with or without preload setting.

it stops in app_main, when I do F10 or F11 it stops in ASM code …rtc_slow_clock
when stopped in app_main i can set bp in loopTask, it stops there
then i press F10 - it goes and stops in 'loop` (but i was on ‘setup’!)
next step F10 - it stops at IPAddress.cpp IPAddress::printTo…
next F10 - small error message with red cross in bottom right corner.
…
something is broken either in opencd or opencd interfacing module… i’m on kubuntu 18.04

one question i have arised - is it possible to disable debugging for core0, and leave it for core1, where arduino task is running?

ivankravets · October 16, 2018, 10:20pm

Could you try the latest release of OpenOCD for ESP32 from Releases · espressif/openocd-esp32 · GitHub?
Please download it and unpack to $USER_HOME/.platformio/packages/tool-openocd-esp32. Please keep package.json file without changes.

Does it work better?

latonita · October 17, 2018, 9:06pm

got latest openocd from their releases page
replaced .platformio/packages/tool-openocd-esp32 with #1
same result

ivankravets · October 19, 2018, 11:39am

We experience the same issue and already reported to ESP team. We will back here soon with updates.

ivankravets · October 22, 2018, 1:59pm

See my comment here

Does it work for you now?