Hm… Maybe it needs a 1200 bps touch (serial port open) to get into actual bootloader mode? Can you add
board_upload.use_1200bps_touch = yes
board_upload.wait_for_upload_port = yes
board_upload.require_upload_port = yes
to the environment while removing upload_port?