Hello PIO’s
In investigation to secure an upcoming IoT product, I’m trying to use MAC Address info while uploading the firmware to the devices, to log it into an excel sheet for an example.
an info while uploading contains MAC address of the device. However, How can I get it automatically via a script file? for an example.
Thanks in advance.
Hm I don’t think there’s a way to intercept the upload, that would have to be built into the core, but there are other ways:
You can also get just the MAC via an esptool.py read_mac command.
>esptool.py read_mac
esptool.py v3.0
Found 2 serial ports
Serial port COM9
Connecting......
Detecting chip type... ESP32
Chip is ESP32-D0WDQ6 (revision 0)
Features: WiFi, BT, Dual Core, Coding Scheme None
Crystal is 40MHz
MAC: 24:0a:c4:04:XX:XX
Uploading stub...
Running stub...
Stub running...
MAC: 24:0a:c4:04:XX:XX
Hard resetting via RTS pin...
Via advanced scrpiting and a env.AddPreAction("upload", before_upload) call you can call the read_mac function before upload, log that, and then just let PIO upload normally. So, two esptool.py runs (where the first one should be really quick).
You can also redirect the upload tool from the normal esptool.py script into your own script (that accepts the same parameters). In your script your then execute esptool.py using the subprocess library while also getting the output text at the same time. You can then parse that and store it wherever.
Hi, Thanks @maxgerhardt for the helping answer.
I’d prefer the 3rd one for just a simple reason: I have a custom board that doesn’t control RST and bootstrapping pins.
To your knowledge, Can I use one esptool.py call to also upload file system in one call? This will make it really helpful.
All esptool.py gets is is a list of binaries with the addresses where to put them (and preceeding misc. info).
So a pio run -t uploadfs (Filesystem upload) builds the spiffs.bin partition file (from the data/) directory and then invokes
“[…]python.exe” “[…]esptool.py” --chip esp32 --port “COM18” --baud 460800 --before default_reset --after hard_reset write_flash -z --flash_mode dio --flash_size detect 2686976 .pio\build\esp32_spiffs\spiffs.bin
Whereas decimal 2686976 is 0x290000, the address of the SPIFFS partition in the default partition table.
And a firmware upload looks e.g. like
“[…]\python.exe” “[…]esptool.py” --chip esp32 --port “COM18” --baud 460800 --before default_reset --after hard_reset write_flash -z --flash_mode dio --flash_freq 40m --flash_size detect 0x1000 […]framework-arduinoespressif32\tools\sdk\bin\bootloader_dio_40m.bin 0x8000 […].pio\build\esp32_spiffs\partitions.bin 0xe000 […]\framework-arduinoespressif32\tools\partitions\boot_app0.bin 0x10000 .pio\build\esp32_spiffs\firmware.bin
You can see that a list of of files and addresses are given, with the bootloader at 0x1000, followed by the partition table at 0x8000, a little bit of stub code that makes app0 the active boot partition at 0xe000 and finally the app0 firmware image at 0x10000.
This list can be arbitrary, so one could also add the address and path to the prebuilt SPIFFS partition file there and it would upload it in one go.
So for an automated firmware flashing process, this is also what you’d probably want. Collect the esptool.py invocations once (using verbose upload and verbose filesystem upload / pio run -t upload -v and pio run -t uploadfs -v), then collect all the needed files in one folder, craft an adapted esptool.py command that flashes all files at once, and then wrap that invocation in some other program or script that also extracts the readout MAC address.
However, on a sidenote, note that the upload via the UART bootloader is still slower than what would be possible over a JTAG adapter and openocd. The baud rate for the upload is probably 921600bps (can go a bit faster if you’ve got a good serial adapter) but still JTAG can go to up to 20 or 26MHz, beeting the sub-1MBps UART speed.
Wow, Thank you very much @maxgerhardt for this wonderful answer!
It’s clear to me what I need to do, I’m now working on ESP8266, So JTAG could be not available as upload mechanism.
Because I’m gonna flash-write every device with different ID (essentially differentiated via an integer number) I’m wondering whether there’s a possible way to inject that integer directly to the binary.
This seems out of the topic’s scope, and potentially out of PlatformIO support. But I’d appreciate if you could direct me to the way.
Thank you again for your kind and informative, yet helpful answer!
DISCLAIMER: The first section of this text are just some thoughts that I had in the beginning, the lower half explains a much better way.
If you have a final firmware and know the exact location where that ID integer is stored, you can modify the .bin file directly. The tricky thing is knowing where the ID is. Especially with regards to compiler optimization, the value might already be propagated into the function code to produce optimized results.
If you think e.g. of
const int my_id = 123; //this is e.g. changed for every build
int get_id() {
return my_id;
}
On low optimization levels like -O0 the const int my_id would be a 4-byte variable living in flash (since its constant) at some address, and the get_id() function would read from that address in flash to return the value. With e.g. the addition of a .map file, it’s easy to find what exactly that address is, since it lists the address of each symbol (address map).
But on higher optimization levels the value might (very likely) be inlined so the the function code of get_id() is directly optimized to return 123; in thet instruction code. That is significantly harder to spot (since there might also be lots of other callers where constant propagation has also already happened) and patch, but possible. But one really doesn’t want to go that way – if the firmware and code can be built in such a way that the value is only read from one known or predictable memory address and it feeds all of its callers without a compile-time optimized value propagation.
Sadly GCC doesn’t have a nice way of e.g. placing a variable at a known location, that must be done with a modification to a linker script. But what it is certainly possible is a scheme like:
platformio.ini )0x40200000 and is 1044464 Bytes big, so it ends at 0x402feff0. Let’s say e.g. we use the last 4 bytes for the ID. With esptool.py we can write an arbitrary blob to an arbitrary address. If we write 4 bytes starting at 0x402fefec, we fill the last 4 bytes of the sketch space with our value. Then we can write a very simple C(++) function that reads from this constant, known address and returns the ID. The compiler can then not optimize away anything because it doesn’t know what will be placed at that address and is forced to always read from this value.int get_id_in_flash() {
int* id_read_addr = (int*) 0x402fefec;
//we can read 4-byte aligned address directly without memcpy
return *id_read_addr;
}
And now that I thought about this for a while, there is also a much much easier way to do it: Use the “EEPROM” (in flash emulated). This is basically like the method above but uses a pre-existing and pre-allocated section of flash.
The Arduino-ESP8266 has a EEPROM library that manages (very primitively) a section in flash, called eeprom. See e.g.
All the EEPROM library does is directly read the flash contents given the address (indexed from 0, so the first byte of EEPROM is “address=0” mapping to e.g. physical 0x405FB000 per above.
Then we have read/write functions and helpers to get multi-byte values.
So, if one just writes to the known EEPROM start address, per above example linker script 0x405FB000, some arbitrary bytes, e.g. a 4 byte integer, the code can just do
//init, only called once.
//our EEPROM area is only 4 bytes long.
//triggers reading flash into internal (4-byte) buffer
EEPROM.begin(4);
//read value stored during production at EEPROM index 0
uint32_t node_id = 0;
EEPROM.get(0, node_id);
//use node_id variable..
//optional: we can read the node ID into a RAM variable once and then
//EEPROM.end();
//to release the internal 4-byte buffer.
I haven’t tested it, but it should be really easy…
PS: The EEPROM.get(0, node_id); is ofc also only a fancy way to read from the internal buffer that has initially been read from flash, similiar to what the above self-made C++ function does with a direct pointer access.
Wow, Great information and solutions as always!
While I was reading your first section, I got a similar idea of the second section, but using FS, The script writes to a file directly and upload it with the File system (requires a FS rebuild).
But while EEPROM is available, It’s also a good solution doesn’t require a FS rebuild.
Now I think the whole idea is completed, I’ll start developing it at some point in the near future.
Thanks again @maxgerhardt for your helpful answers 
Hi @maxgerhardt, I’m now testing this approach using esp flash tool, but I can’t get the number right in the program:
I’ve written 5 inside node-id.bin file (I don’t know whether it’s valid format or not):
and this is what I get through serial monitor:
Node_ID=6293661
A
Any idea what’s wrong with the current isituation
The address is missing 0x at the beginning right?
What’s the exact binary content of the node-id.bin file? (e.g. screenshot of HexDump Online: show contents of a binary file in hex)
What is the exact code to read the Node_ID?
Thanks, I’ve corrected, and added 0x to the beginning.
This is the output of HexDump:
file name: node_id.bin
mime type:
0000-0001: 35 5
The code to read it:
void initEEPROM()
{
//init, only called once.
//our EEPROM area is only 4 bytes long.
//triggers reading flash into internal (4-byte) buffer
EEPROM.begin(4);
//read value stored during production at EEPROM index 0
uint32_t node_id = 0;
EEPROM.get(0, node_id);
//use node_id variable..
Serial.printf("Node_ID=%u\n",node_id);
//optional: we can read the node ID into a RAM variable once and then
//EEPROM.end();
//to release the internal 4-byte buffer.
}
Ah so you save it as an ASCII string? Hm I wouldn’t recommend that, since ASCII strings are variable length and you have to detect the 0 at the end… if you store it directly as the binary representation, constant-width, and directly in the right endianness (little-endian, “backwards”) you will have much less trouble.
So if replace the node_id.bin file with hexadecimal content
05 00 00 00
it will read is little-endian number 5, without any code changes.
The code you have now reads a 4-byte value (uint32_t) from that memory location but you only prepared 1 byte of character data to it – you would have to change uint32_t node_id = 0; to char node_id = 0; and print it with %c if you’d want it to work with that .bin file. For larger than 1 character node-ids that will ofc not work, so you would need to build additional logic for reading multiple bytes and detecting the end…
So as said, better to use constant-width binary integer. After reading it out, you can still directly convert it to a String if you need it.
Thanks, That’s one step forward, didn’t have an experience about hex files before.
Now I’ve the following HexDump:
file name: node_id.bin
mime type:
0000-0004: 05 00 00 00 ....
But the value read isn’t correct yet:
Node_ID=4294967295
I’ve double checked the ld.script I’m using, It’s eagle.flash.4m1m.ld.
I’ve tried to read the address directly using a pointer:
uint32_t get_id_in_flash() {
uint32_t* id_read_addr = (uint32_t*) 0x405FB000;
//we can read 4-byte aligned address directly without memcpy
return *id_read_addr;
}
and called it just before EEPROM.begin(4):
uint32_t value = get_id_in_flash();
Serial.printf("get_id()=%u\n",value);
I got those two different values:
get_id()=8290304
Node_ID=4294967295
Node_ID is the result of EEPROM.get(0,node_id)
** There’s no effect in using int* instead of uint32_t*.
One thing I can definitely see is that you’re selecting a flash size of 16Mbit in the flasher tool, so that’s 2 MByte. Is that correct in the first place?
The example address I’ve posted is for a 4MByte flash. What happens if you flash it to address 0x403FB000 instead?
per
Sorry for that, I’ve corrected that implicitly without updating the status.
But your last hint really helped!!!
get_id()=8290304
Node_ID=5
platformio.ini:
[platformio]
default_envs = nodemcuv2
[common]
framework = arduino
monitor_speed = 115200
lib_deps =
https://github.com/philbowles/PangolinMQTT
https://github.com/philbowles/H4
https://github.com/philbowles/h4plugins
https://github.com/philbowles/ESPAsyncWebServer
https://github.com/philbowles/ESPAsyncTCP-master
upload_speed = 921600
[env:nodemcuv2]
framework = ${common.framework}
platform = espressif8266
board = nodemcuv2
monitor_speed = ${common.monitor_speed}
upload_speed = ${common.upload_speed}
build_flags =
-D BEARSSL_SSL_BASIC
-DNDEBUG
-D PIO_FRAMEWORK_ARDUINO_LWIP2_HIGHER_BANDWIDTH_LOW_FLASH
; build_type = debug
board_build.flash_mode = dio
board_build.filesystem = littlefs
board_build.ldscript = eagle.flash.4m1m.ld
lib_deps = ${common.lib_deps}
ESPAsyncUDP
I’m wondering what could make it different to default EEPROM address of eagle.flash.4m1m.ld?
Uh hm okay so the chip is supposed to be 4Mbyte chip. But then 0x405FB000 should be correct. Can you check with esptool.py what it says about the flash capacity?
Something like
python C:\Users\<user>\.platformio\packages\tool-esptoolpy\esptool.py flash_id
(you might need to replace python with PIO’s python version, e.g. C:\Users\<user>\.platformio\penv\Scripts\python.exe)
Hi, This is the output:
esptool.py v3.0
Found 1 serial ports
Serial port COM3
Connecting....
Detecting chip type... ESP8266
Chip is ESP8266EX
Features: WiFi
Crystal is 26MHz
MAC: 5c:cf:7f:8c:0d:f8
Uploading stub...
Running stub...
Stub running...
Manufacturer: e0
Device: 4016
Detected flash size: 4MB
Hard resetting via RTS pin...
Hi @maxgerhardt, I’m very thankful for your help.
What do you think about this strangeness? where EEPROM of compiled 4m1m accesses the 2m1m EEPROM address?
Is it wrong compiler linking, or maybe running wrong ld script?.. What do you think? where should I investigate?
Hi #maxgerhrdt, I’d like to inform you with an issue raised in Arduino Core about the subject:
It’s my pleasure to inform you that the issue is solved, the flash is mapped to 0x40200000, so EEPROM starts after 0x3fb000 and results in the correct address of 0x405fb000.