Redirect ESP32 log messages to SD card

Hey, I use ESP32 with arduino framework. For remote debugging, I want to store system logs like errors or warnings to the SD card together with my own log statements. Therefore, I created a callback function, which should be called when ESP_LOGE or ESP_LOGW is used, by exploiting esp_log_set_vprintf(sdCardLogOutput). However, the callback function is never called and only the ESP_LOGE is printed to the terminal via UART.

This is my code example:

#include <Arduino.h>
#include "FS.h"
#include <SD.h>
#include "SPI.h"
#include "esp_log.h"

#define TAG "myApp"
#define LOG_LEVEL ESP_LOG_INFO
#define MY_ESP_LOG_LEVEL ESP_LOG_WARN

File logFile;

int sdCardLogOutput(const char *format, va_list args)
{
	Serial.println("Callback running");
	char buf[128];
	int ret = vsnprintf(buf, sizeof(buf), format, args);
	if (logFile)
	{
		logFile.println(buf);
		logFile.flush();
	}
	return ret;
}

void setup()
{
	Serial.begin(9600);
	delay(1000);
	while (!Serial)
	{
	}
	if (SD.begin())
	{
		Serial.println("Successfully initialized SD card");
	}
	else
	{
		Serial.println("SD card failed");
	}
	logFile = SD.open("/log.txt", FILE_WRITE, true);
	if (!logFile)
	{
		Serial.println("Failed to open log file");
	}
	else
	{
		Serial.println("Setting log levels and callback");
		esp_log_level_set("*", MY_ESP_LOG_LEVEL);
		esp_log_level_set(TAG, LOG_LEVEL);
		esp_log_set_vprintf(sdCardLogOutput);
	}
}

void loop()
{
	ESP_LOGE(TAG, "Error message");
	ESP_LOGW(TAG, "Warning message");
	delay(1000);
}

Besides, this is my platformIO.ini file:

[env:firebeetle32]
platform = espressif32
board = firebeetle32
framework = arduino
build_flags = -DUSE_ESP32_LOG

Any idea why

  • My callback is never invoked

  • Only the error message is printed but not the warning message although my log levels should allow for this

There’s exactly 0 references to a USE_ESP32_LOG macro in the entirety of Arduino-ESP32.

https://github.com/search?q=repo%3Aespressif%2Farduino-esp32%20USE_ESP32_LOG&type=code

The is however USE_ESP_IDF_LOG. That would make Arduino use the esp_log_.. system.

But to get that to compile you have to go follow the ESP_LOGE macro to get to esp32-hal-log.h and uncomment the missing

and define the maximum log level (this checks it at the macro level, the inner log function will check it against the log level of the tag)

So in total I have a build_flags of

build_flags= -DUSE_ESP_IDF_LOG -DLOG_LOCAL_LEVEL=5

with a minimal source code of

#include <Arduino.h>
#undef TAG // from Arduino.h

#define TAG "myApp"
#define LOG_LEVEL ESP_LOG_INFO
#define MY_ESP_LOG_LEVEL ESP_LOG_WARN

int sdCardLogOutput(const char *format, va_list args)
{
    Serial.println("Callback running");
    char buf[128];
    int ret = vsnprintf(buf, sizeof(buf), format, args);
    Serial.print(buf);
    return ret;
}

void setup()
{
    Serial.begin(115200);
    delay(1000);
    while (!Serial)	{	}
    Serial.println("Setting log levels and callback");
    esp_log_level_set("*", MY_ESP_LOG_LEVEL);
    esp_log_level_set(TAG, LOG_LEVEL);
    esp_log_set_vprintf(sdCardLogOutput);
}

void loop()
{
    ESP_LOGE(TAG, "Error message");
    ESP_LOGW(TAG, "Warning message");
    ESP_LOGI(TAG, "Info message");
    ESP_LOGV(TAG, "Verbose message"); // must not print, for our tag we set it to info
    delay(1000);
}

and that continuously outputs

Callback running
E (79794) myApp: Error message
Callback running
W (79794) myApp: Warning message
Callback running
I (79794) myApp: Info message

Note that this is not what you may want. You may want to work with the ArduHAL logging system used by default. I’ll leave that research to you.

1 Like

Thanks for the elaborate answer.

First what do I want in a little more detail:

  • I want to redirect errors produced by libraries to the SD card for remote debugging, e.g. [E][WiFiSTA.cpp:232] begin(): SSID too long or missing!

  • I want to store my “self made” print statements for debugging on the SD card, too, e.g. ESP_LOGI(ESP_LOG_INFO, TAG, "This is info");

Approach 1: USE_ESP_IDF_LOG
I was able to reproduce your code and forward the messages to the SD card.However, I get flooded with warnings regarding redefinition of log_x functions during compilation. I was not able to test if the libary calls get forwarded, too, but I believe they will be.

Approach 2: ArduHAL
Using these build_flags=-DCONFIG_ARDUHAL_ESP_LOG -DCORE_DEBUG_LEVEL=5 I am not getting warnings, but the messages are only printed to the serial port, but not to the SD card.

Besides, I get compile Warning for both build flags stating the options are unknown:
Warning! Ignore unknown configuration option -dconfig_arduhal_esp_log -dcore_debug_level in section [env:firebeetle32]

Another disadvantage of approach 1 is messing with the library header files, which can always leads to errors during development when different people work on a project. So could you advice me on how to proceed to use the ArduHal approach redirecting the prints to the SD card?

Are you sure you’ve made the modificaiton in esp32-hal-log.h with commenting in the

//#ifndef TAG
//#define TAG "ARDUINO"
//#endif

block so that it becomes

#ifndef TAG
#define TAG "ARDUINO"
#endif

?

Yes I uncommented everything otherwise I would not compile at all. I just retried.

[env:firebeetle32]
platform = espressif32
board = firebeetle32
framework = arduino
build_flags= -DUSE_ESP_IDF_LOG -DLOG_LOCAL_LEVEL=5
#ifdef USE_ESP_IDF_LOG
#ifndef TAG
#define TAG "ARDUINO"
#endif
#define log_n(format, ...) myLog(ESP_LOG_NONE, format, ##__VA_ARGS__)
#else
In file included from C:/Users/Slevin/.platformio/packages/framework-arduinoespressif32/cores/esp32/esp32-hal.h:76,
                 from C:/Users/Slevin/.platformio/packages/framework-arduinoespressif32/cores/esp32/Arduino.h:36,
                 from src/main.cpp:1:
C:/Users/Slevin/.platformio/packages/framework-arduinoespressif32/cores/esp32/esp32-hal-log.h:195: warning: "log_n" redefined
 #define log_n(format, ...) myLog(ESP_LOG_NONE, format, ##__VA_ARGS__)

This is one of the many all similar looking warnings (not errors).

No keep that line commented.

So, I came a big step forward.

In esp32-hal-log.h I have now:

#ifdef USE_ESP_IDF_LOG
#ifndef TAG
#define TAG "myAPP"
#endif
//#define log_n(format, ...) myLog(ESP_LOG_NONE, format, ##__VA_ARGS__)
#else

My platformio.ini looks like:

[env:firebeetle32]
platform = espressif32
board = firebeetle32
framework = arduino
build_flags= -DUSE_ESP_IDF_LOG -DCORE_DEBUG_LEVEL=5

Note that I am now using CORE_DEBUG_LEVEL instead of LOG_LOCAL_LEVEL

With the CORE_DEBUGL_LEVEL constant I can set the logging level at compilation. Afterwards, I can use esp_log_level_set(TAG, LOG_LEVEL) to change the logging level for different TAGs during runtime.

It is important to know, that you cannot set higher logging levels than defined with CORE_DEBUG_LEVEL. If you do, esp_log_level_set() does nothing - at least from my experience.

The following code example can be used to get familiar with the logging capabilities:

#include <Arduino.h>
#include "FS.h"
#include <SD.h>
#include "SPI.h"
#include "esp_log.h"
#include "esp32-hal-log.h"

#define LOG_LEVEL ESP_LOG_WARN
#define MY_ESP_LOG_LEVEL ESP_LOG_INFO

File logFile;

int sdCardLogOutput(const char *format, va_list args)
{
	Serial.println("Callback running");
	char buf[128];
	int ret = vsnprintf(buf, sizeof(buf), format, args);
	if (logFile)
	{
		logFile.print(buf);
		logFile.flush();
	}
	return ret;
}

void setup()
{
	Serial.begin(9600);
	delay(1000);
	while (!Serial)
	{
	}
	if (SD.begin())
	{
		Serial.println("Successfully initialized SD card");
	}
	else
	{
		Serial.println("SD card failed");
	}
	logFile = SD.open("/log.txt", FILE_APPEND);
	if (!logFile)
	{
		Serial.println("Failed to open log file");
	}
	else
	{
		Serial.println("Setting log levels and callback");
		esp_log_level_set("TEST", LOG_LEVEL);
		esp_log_level_set(TAG, MY_ESP_LOG_LEVEL);
		esp_log_set_vprintf(sdCardLogOutput);
	}
}

void loop()
{
	for (int i = 0; i < 100; i++)
	{
		ESP_LOGE(TAG, "Error message");
		ESP_LOGW(TAG, "Warning message");
		ESP_LOGI("TEST", "Info message");
		ESP_LOGD(TAG, "Verbose message"); // must not print, for our tag we set it to info
	}
}

Thank you for your help Max.
1 Like

Hey,
I have the same issue,
I found some workaround without changing a core file, creating a custom command that calls a custom function

void esp_log_vprintf(esp_log_level_t level, const char* format, ...);

#define CFORMAT(level, latter, format, ...) esp_log_vprintf(ESP_LOG_INFO, ARDUHAL_LOG_FORMAT(latter, format), ##__VA_ARGS__)
#define CESP_LOGE(tag, format, ...)  log_e("[%s] " format, tag, ##__VA_ARGS__); CFORMAT(ESP_LOG_ERROR, E, "[%s] " format, tag, ##__VA_ARGS__)
#define CESP_LOGW(tag, format, ...)  log_w("[%s] " format, tag, ##__VA_ARGS__); CFORMAT(ESP_LOG_WARN, W, "[%s] " format, tag, ##__VA_ARGS__)
#define CESP_LOGI(tag, format, ...)  log_i("[%s] " format, tag, ##__VA_ARGS__); CFORMAT(ESP_LOG_INFO, I, "[%s] " format, tag, ##__VA_ARGS__)
#define CESP_LOGD(tag, format, ...)  log_d("[%s] " format, tag, ##__VA_ARGS__); CFORMAT(ESP_LOG_DEBUG, D, "[%s] " format, tag, ##__VA_ARGS__)
#define CESP_LOGV(tag, format, ...)  log_v("[%s] " format, tag, ##__VA_ARGS__); CFORMAT(ESP_LOG_VERBOSE, V, "[%s] " format, tag, ##__VA_ARGS__)
void esp_log_vprintf(esp_log_level_t level, const char* format, ...) {
  va_list arg;
  va_start(arg, format);
  char buf[1024];  // Buffer for storing log message
  int len = vsnprintf(buf, sizeof(buf), format, arg);
  va_end(arg);
  
  // Write log message to SPIFFS
  File logFile = SPIFFS.open("/logs.txt", "a");
  if (!logFile) {
    _PL("Failed to open log file");
  }

  logFile.write((const uint8_t*)buf, len);
  logFile.close();
}

Thank you for your follow up.

But I have a few questions.

  • The callback function esp_log_vprintf() is triggered by calling CESP_LOGI() but also by logging messages generated by core functions using ESP_LOGI()?

  • How do you set the global logging level?

  • Can you set different logging levels for different tags?

Thank you for your questions,

  • The esp_log_vprintf() is called by CESP_LOGI() as a command, so the callback work only by direct call to CESP_LOGI(),
    unfortunately, I didn’t find a way to use the callback from the core functions, for example, BLE (it is just a workaround, of course, the best solution is a callback from the <esp_log.h>)
  • you can add a condition if(level <= CORE_DEBUG_LEVEL)
void esp_log_vprintf(esp_log_level_t level, const char* format, ...) {
  if(level <= CORE_DEBUG_LEVEL) {
    va_list arg;
    va_start(arg, format);
    char buf[1024];  // Buffer for storing log message
    int len = vsnprintf(buf, sizeof(buf), format, arg);
    va_end(arg);
    
    // Write log message to SPIFFS
    File logFile = SPIFFS.open("/logs.txt", "a");
    if (!logFile) {
      _PL("Failed to open log file");
    }
  
    logFile.write((const uint8_t*)buf, len);
    logFile.close();
  }
}
  • you can implement it with some map
std::map<String, int> tagToLevelMap = {
  {"main", 3},
  {"file1", 4},
};

 `if(level <= CORE_DEBUG_LEVEL || level <= tagToLevelMap[tag]) `

Thanks for this thread! I had a similar problem and it helped!

I would like to share my solution that doesn’t require esp32-hal-log.h modifications.
It’s possible to define TAG via compiler flags:

build_flags = -DUSE_ESP_IDF_LOG -DLOG_LOCAL_LEVEL=5 -DTAG="\"ARDUINO\""
2 Likes

Thank you very much for this simple solution. Now I can dynamically set log levels per module with esp_log_level_set(), which didn’t work with the mapped Arduino logging.

Only - now the colors in the VS Code Terminal are gone. I used to have DCONFIG_ARDUHAL_LOG_COLORS=1 set in platform.ini, however that has now no effect anymore.

Does anyone know how to turn color codes back on? Thank you!

Peter

CONFIG_LOG_COLORS
Can restore colors

1 Like

Fantastic and simple explanation as usual, thank you so much @maxgerhardt! I have re-purposed this to log to MQTT and I’m very happy with the result.

This one discussion clears up so much about using ESP_LOGx style logging that I couldn’t find anywhere else. I have converted almost totally from Serial.print now.