PlatformIO Community

Is it possible to output LOGS to a SPIFFS file that's inside the ESP32?


#1

Hello !

Do you think it’s possible to output logs like ESP_LOGV("TAG", "Verbose"); to a SPIFFS file that’s stored in the ESP’s flash memory ?

I would like to display log messages to a webpage hosted on the ESP32.
And I want to do that by writing them first in a SPIFFS file.
I was thinking of some sort of function that works using the first in, last out principle after 3 chunks of logs.
1 chunk of log would be comprised of a number of logs resulted from one passing through the void loop() function, for example.

The following lines of code enable debug messages to be outputted on the Serial Monitor.

  esp_log_level_set("*", ESP_LOG_VERBOSE);  
  ESP_LOGE("TAG", "Error");
  ESP_LOGW("TAG", "Warning");
  // ESP_LOGI("TAG", "Info");
  ESP_LOGD("TAG", "Debug");
  ESP_LOGV("TAG", "Verbose");

The following line is inside platformio.ini :

build_flags = -DCORE_DEBUG_LEVEL=ARDUHAL_LOG_LEVEL_DEBUG

Let’s say I have a html page events.html and a HTTP GET with a URL inside the void loop(). For example, I would like to check logs for the HTTP GET request. Or let’s say that the ESP crashes out of nowhere and I would like to access some logs and see what went wrong.

#include <Arduino.h>
#include <WiFi.h>
#include <ESPAsyncWebServer.h>

AsyncWebServer server(80);


void setup() {
	Serial.begin(115200);
	delay(2000);
  if (!SPIFFS.begin(true)) {
    Serial.println("An Error has occurred while mounting SPIFFS");
    return;
  }
  File events = SPIFFS.open("/events.txt", FILE_WRITE);
  if(!events) Serial.println("Couldn't create /events.txt");
  	
	WiFi.begin("mySSID","password");
	
	server.on("/events", HTTP_GET, [](AsyncWebServerRequest *request){
		request->send(SPIFFS, "/events.html", "text/html");
	});
	
	.
	.
	.
	
	events.close();
}

void loop() {
  if ((WiFi.status() == WL_CONNECTED)) { //Check the current connection status
 
    HTTPClient http;
 
    http.begin("http://jsonplaceholder.typicode.com/comments?id=10"); //Specify the URL
    int httpCode = http.GET();                                        //Make the request
 
    if (httpCode > 0) { //Check for the returning code
        Serial.println();
        Serial.printf("[HTTP] GET... code: %d\n", httpCode);
        if(httpCode == HTTP_CODE_OK) {
            String payload = http.getString();
            Serial.println(payload);
            Serial.println();
          }
      } else {
          Serial.printf("[HTTP] GET... failed, error: %s\n", http.errorToString(httpCode).c_str());
        }
 
    http.end(); //Free the resources
  }
 
  delay(10000);
 
}

I am guessing that the answer has something to do with the following functions:

/**
 * @brief Set function used to output log entries
 *
 * By default, log output goes to UART0. This function can be used to redirect log
 * output to some other destination, such as file or network. Returns the original
 * log handler, which may be necessary to return output to the previous destination.
 *
 * @param func new Function used for output. Must have same signature as vprintf.
 *
 * @return func old Function used for output.
 */
vprintf_like_t esp_log_set_vprintf(vprintf_like_t func);

/**
 * @brief Function which returns timestamp to be used in log output
 *
 * This function is used in expansion of ESP_LOGx macros.
 * In the 2nd stage bootloader, and at early application startup stage
 * this function uses CPU cycle counter as time source. Later when
 * FreeRTOS scheduler start running, it switches to FreeRTOS tick count.
 *
 * For now, we ignore millisecond counter overflow.
 *
 * @return timestamp, in milliseconds
 */
uint32_t esp_log_timestamp(void);

I’ve been reading here but it’s still ambiguous to me.

I haven’t had any experience with logs and I am still reading about it. I thought that by posting here I will have a bigger change of someone guiding me towards my answer.
Thank you !


#2

You might install something like this? Write the output into an interemediate buffer, then write it to a SPIFFS File object.

The standard library actually provides vfprintf() directly writes to a FILE* (no big intermediate buffer) but I couldn’t find a method to get the SPIFFS class to give me a FILE* instead of a File.

Note that the va_end (args); is not needed because the ESP-IDF function already does that (see here)

#include <stdarg.h>
#include <stdio.h>

//log file
static File spiffsLogFile;
//static buffer in which log output is written
static char log_print_buffer[512];

void setup_spiffs_logging() {
	//open file in WRITE and APPEND mode to not overwrite old content.
	//you might want to choose only WRITE (or a previous delete call)
	//to remove the old log
	spiffsLogFile = SPIFFS.open("/spiffs_log.txt", FILE_WRITE FILE_APPEND);
}

void spiffs_logging_cleanup() {
	spiffsLogFile.close();
}

int vprintf_into_spiffs(const char* szFormat, va_list args) {
	//write evaluated format string into buffer
	int ret = vsnprintf (log_print_buffer, sizeof(log_print_buffer), szFormat, args);

	//output is now in buffer. write to file.
	if(ret >= 0) {
		//debug output
		printf("[Writing to SPIFFS] %.*s", ret, log_print_buffer);
		spiffsLogFile.write((uint8_t*) log_print_buffer, (size_t) ret);
		//to be safe in case of crashes: flush the output
		spiffsLogFile.flush();
	}
	return ret;
}

void setup(){
  Serial.begin(115200);

  setup_spiffs_logging();
  //install new logging function
  esp_log_set_vprintf(&vprintf_into_spiffs);
  //..
}

#3

I think I did something wrong because it writes nothing in the file.

#include <Arduino.h>
#include <WiFi.h>
#include <HTTPClient.h>
#include <ESPAsyncWebServer.h>
#include <stdarg.h>
#include <stdio.h>
#include <SPIFFS.h>

const char* ssid = "SSID";
const char* password =  "PASSWORD";

AsyncWebServer server(80);

static File spiffsLogFile;
static char log_print_buffer[512];

void setup_spiffs_logging() {
	//open file in WRITE and APPEND mode to not overwrite old content.
	//you might want to choose only WRITE (or a previous delete call)
	//to remove the old log
	spiffsLogFile = SPIFFS.open("/spiffs_log.txt", FILE_WRITE FILE_APPEND);
}

void spiffs_logging_cleanup() {
	spiffsLogFile.close();
}

int vprintf_into_spiffs(const char* szFormat, va_list args) {
	//write evaluated format string into buffer
	int ret = vsnprintf (log_print_buffer, sizeof(log_print_buffer), szFormat, args);

	//output is now in buffer. write to file.
	if(ret >= 0) {
		//debug output
		printf("[Writing to SPIFFS] %.*s", ret, log_print_buffer);
		spiffsLogFile.write((uint8_t*) log_print_buffer, (size_t) ret);
		//to be safe in case of crashes: flush the output
		spiffsLogFile.flush();
	}
	return ret;
}

void setup() {
 
  Serial.begin(115200);
  delay(1000);

  if (!SPIFFS.begin(true)) {
    Serial.println("An Error has occurred while mounting SPIFFS");
    return;
  }

  setup_spiffs_logging();
  //install new logging function
  esp_log_set_vprintf(&vprintf_into_spiffs);

  delay(1000);
  WiFi.begin(ssid, password);
  delay(1000);

  while (WiFi.status() != WL_CONNECTED) {
    delay(1000);
    Serial.println("Connecting to WiFi..");
  }
 
  Serial.println("Connected to the WiFi network");
  Serial.println(WiFi.localIP());

  server.on("/events", HTTP_GET, [](AsyncWebServerRequest *request){
    if(SPIFFS.exists("/spiffs_log.txt")) {
      request->send(SPIFFS, "/spiffs_log.txt", "text/plain");
    } else {
      request->send(200, "text/plain", "LOGS not found ! Restarting in 5 seconds..");
      delay(5000);
      ESP.restart();
    }
    
    });

  server.begin();

  File file2 = SPIFFS.open("/spiffs_log.txt");

  if(!file2){
      Serial.println("Failed to open file for reading");
      return;
  }

  Serial.println("File Content:");

  while(file2.available()){

      Serial.write(file2.read());
  }

  file2.close();
  Serial.println("End of file content");
}
 
void loop() {
  if ((WiFi.status() == WL_CONNECTED)) { //Check the current connection status
 
    HTTPClient http;
 
    http.begin("http://jsonplaceholder.typicode.com/comments?id=10"); //Specify the URL
    int httpCode = http.GET();                                        //Make the request
 
    if (httpCode > 0) { //Check for the returning code
        Serial.println();
        Serial.printf("[HTTP] GET... code: %d\n", httpCode);
        if(httpCode == HTTP_CODE_OK) {
            String payload = http.getString();
            Serial.println(payload);
            Serial.println();
          }
      } else {
          Serial.printf("[HTTP] GET... failed, error: %s\n", http.errorToString(httpCode).c_str());
        }
 
    http.end(); //Free the resources
  }
 
  delay(10000);
 
}

I don’t have any file (/spiffs_log.txt) content on the Serial Monitor nor at http://myIP/events.

I don’t know what I did wrong…

When should I call spiffs_logging_cleanup() to close the file ?

The only output I get is:

Connecting to WiFi..
Connected to the WiFi network
*myIP*
File Content:
End of file content

[HTTP] GET... code: 200
[
  {
    "postId": 2,
    "id": 10,
    "name": "eaque et deleniti atque tenetur ut quo ut",
    "email": "Carmen_Keeling@caroline.name",
    "body": "voluptate iusto quis nobis reprehenderit ipsum amet nulla\nquia quas dolores velit et non\naut quia necessitatibus\nnostrum quaerat nulla et accusamus nisi facilis"
  }
]

[HTTP] GET... code: 200
[
  {
    "postId": 2,
    "id": 10,
    "name": "eaque et deleniti atque tenetur ut quo ut",
    "email": "Carmen_Keeling@caroline.name",
    "body": "voluptate iusto quis nobis reprehenderit ipsum amet nulla\nquia quas dolores velit et non\naut quia necessitatibus\nnostrum quaerat nulla et accusamus nisi facilis"
  }
]

And the URL response gets repeated every 10 seconds.


#4

Yeah we have a few problems here. First of all, the SPIFFS class has a base path of /spiffs/ so the paths should include that too…

Next the problem is when one File object is opened for writing and reading, when reading it, we have to seek() back to position 0 and seek back to the end afterwards, the code didn’t do that. I tried implementing that but then all read() calls gave a -1 and I couldn’t find out why. So instead, I just open() and close() the file on every log write call.

Next, to trigger writing to the log, we have to call functions like these

  //trigger some text
  esp_log_level_set("TAG", ESP_LOG_DEBUG);
  //write into log
  esp_log_write(ESP_LOG_DEBUG, "TAG", "text!\n");

Which goes to the base problem: The Arduino-ESP32 implementation does not use the esp_log_write() functions, but it’s own logging system where the log output function cannot be altered!

Looking to esp32-hal-log.h

int log_printf(const char *fmt, ...);

#define ARDUHAL_SHORT_LOG_FORMAT(letter, format)  ARDUHAL_LOG_COLOR_ ## letter format ARDUHAL_LOG_RESET_COLOR "\r\n"
#define ARDUHAL_LOG_FORMAT(letter, format)  ARDUHAL_LOG_COLOR_ ## letter "[" #letter "][%s:%u] %s(): " format ARDUHAL_LOG_RESET_COLOR "\r\n", pathToFileName(__FILE__), __LINE__, __FUNCTION__

#if ARDUHAL_LOG_LEVEL >= ARDUHAL_LOG_LEVEL_VERBOSE
#define log_v(format, ...) log_printf(ARDUHAL_LOG_FORMAT(V, format), ##__VA_ARGS__)
#define isr_log_v(format, ...) ets_printf(ARDUHAL_LOG_FORMAT(V, format), ##__VA_ARGS__)
#else
#define log_v(format, ...)
#define isr_log_v(format, ...)
#endif

So depending on the log level, the framework will set log_v (and related functions) to log_printf() for normal executions and ets_printf() for when inside an ISR. And log_printf() is implemented statically in esp32-hal-uart.c

int log_printf(const char *format, ...)
{
    if(s_uart_debug_nr < 0){
        return 0;
    }
    static char loc_buf[64];
    char * temp = loc_buf;
    int len;
    va_list arg;
    va_list copy;
    va_start(arg, format);
    va_copy(copy, arg);
    len = vsnprintf(NULL, 0, format, arg);
    va_end(copy);
    if(len >= sizeof(loc_buf)){
        temp = (char*)malloc(len+1);
        if(temp == NULL) {
            return 0;
        }
    }
    vsnprintf(temp, len+1, format, arg);
#if !CONFIG_DISABLE_HAL_LOCKS
    if(_uart_bus_array[s_uart_debug_nr].lock){
        xSemaphoreTake(_uart_bus_array[s_uart_debug_nr].lock, portMAX_DELAY);
        ets_printf("%s", temp);
        xSemaphoreGive(_uart_bus_array[s_uart_debug_nr].lock);
    } else {
        ets_printf("%s", temp);
    }
#else
    ets_printf("%s", temp);
#endif
    va_end(arg);
    if(len >= sizeof(loc_buf)){
        free(temp);
    }
    return len;
}

Since this uses a static ets_printf() from the library we can’t change that. So, we can’t redirect the the debug output like

[D][WiFiGeneric.cpp:342] _eventCallback(): Event: 0 - WIFI_READY

but only our own calls with esp_log_write().

I’d suggest to open an issue on https://github.com/espressif/arduino-esp32 for such a feature request.

Here is the full code

#include <Arduino.h>
#include <WiFi.h>
#include <HTTPClient.h>
#include <ESPAsyncWebServer.h>
#include <stdarg.h>
#include <stdio.h>
#include <SPIFFS.h>

const char* ssid = "SSID";
const char* password =  "PW";

AsyncWebServer server(80);

static char log_print_buffer[512];

int vprintf_into_spiffs(const char* szFormat, va_list args) {
	//write evaluated format string into buffer
	int ret = vsnprintf (log_print_buffer, sizeof(log_print_buffer), szFormat, args);

	//output is now in buffer. write to file.
	if(ret >= 0) {
		File spiffsLogFile = SPIFFS.open("/spiffs/spiffs_log.txt", FILE_APPEND);
		//debug output
		//printf("[Writing to SPIFFS] %.*s", ret, log_print_buffer);
		spiffsLogFile.write((uint8_t*) log_print_buffer, (size_t) ret);
		//to be safe in case of crashes: flush the output
		spiffsLogFile.flush();
		spiffsLogFile.close();
	}
	return ret;
}

void setup() {

  Serial.begin(115200);
  delay(1000);

  if (!SPIFFS.begin(true)) {
    Serial.println("An Error has occurred while mounting SPIFFS");
    return;
  }

  //install new logging function
  esp_log_set_vprintf(&vprintf_into_spiffs);

  //trigger some text
  esp_log_level_set("TAG", ESP_LOG_DEBUG);
  //write into log
  esp_log_write(ESP_LOG_DEBUG, "TAG", "text!\n");

  delay(1000);
  WiFi.begin(ssid, password);
  delay(1000);

  while (WiFi.status() != WL_CONNECTED) {
    delay(1000);
    Serial.println("Connecting to WiFi..");
  }

  Serial.println("Connected to the WiFi network");
  Serial.println(WiFi.localIP());

  server.on("/events", HTTP_GET, [](AsyncWebServerRequest *request){
    if(SPIFFS.exists("/spiffs_log.txt")) {
      request->send(SPIFFS, "/spiffs_log.txt", "text/plain");
    } else {
      request->send(200, "text/plain", "LOGS not found ! Restarting in 5 seconds..");
      delay(5000);
      ESP.restart();
    }

    });

  server.begin();

  File file2 = SPIFFS.open("/spiffs_log.txt");

  if(!file2){
      Serial.println("Failed to open file for reading");
      return;
  }

  Serial.println("File Content:");

  while(file2.available()){

      Serial.write(file2.read());
  }

  file2.close();
  Serial.println("End of file content");


  Serial.println("End of file content");
}

void loop() {
  /*if ((WiFi.status() == WL_CONNECTED)) { //Check the current connection status

    HTTPClient http;

    http.begin("http://jsonplaceholder.typicode.com/comments?id=10"); //Specify the URL
    int httpCode = http.GET();                                        //Make the request

    if (httpCode > 0) { //Check for the returning code
        Serial.println();
        Serial.printf("[HTTP] GET... code: %d\n", httpCode);
        if(httpCode == HTTP_CODE_OK) {
            String payload = http.getString();
            Serial.println(payload);
            Serial.println();
          }
      } else {
          Serial.printf("[HTTP] GET... failed, error: %s\n", http.errorToString(httpCode).c_str());
        }

    http.end(); //Free the resources
  }

  delay(10000);
*/
}

This example does readout the correct contents as writen via esp_log_write

File Content:
text!
End of file content

#5

This is the code:

#include <Arduino.h>
#include <WiFi.h>
#include <HTTPClient.h>
#include <ESPAsyncWebServer.h>
#include <stdarg.h>
#include <stdio.h>
#include <SPIFFS.h>

const char* ssid = "SSID";
const char* password =  "password";

AsyncWebServer server(80);

static char log_print_buffer[512];

int vprintf_into_spiffs(const char* szFormat, va_list args) {
	//write evaluated format string into buffer
	int ret = vsnprintf (log_print_buffer, sizeof(log_print_buffer), szFormat, args);

	//output is now in buffer. write to file.
	if(ret >= 0) {
    if(!SPIFFS.exists("/LOGS.txt")) {
      File writeLog = SPIFFS.open("/LOGS.txt", FILE_WRITE);
      if(!writeLog) Serial.println("Couldn't open spiffs_log.txt"); 
      delay(50);
      writeLog.close();
    }
    
		File spiffsLogFile = SPIFFS.open("/LOGS.txt", FILE_APPEND);
		//debug output
		//printf("[Writing to SPIFFS] %.*s", ret, log_print_buffer);
		spiffsLogFile.write((uint8_t*) log_print_buffer, (size_t) ret);
		//to be safe in case of crashes: flush the output
		spiffsLogFile.flush();
		spiffsLogFile.close();
	}
	return ret;
}

void setup() {
 
  Serial.begin(115200);
  delay(1000);
  if (!SPIFFS.begin(true)) {
    Serial.println("An Error has occurred while mounting SPIFFS");
    return;
  }
  esp_log_set_vprintf(&vprintf_into_spiffs);
  //install new logging function
  //trigger some text
  esp_log_level_set("TAG", ESP_LOG_DEBUG);
  //write into log
  esp_log_write(ESP_LOG_DEBUG, "TAG", "text!\n");

  delay(1000);
  WiFi.begin(ssid, password);
  delay(1000);
 
  while (WiFi.status() != WL_CONNECTED) {
    delay(1000);
    Serial.println("Connecting to WiFi..");
  }
 
  Serial.println("Connected to the WiFi network");
  Serial.println(WiFi.localIP());

  server.on("/events", HTTP_GET, [](AsyncWebServerRequest *request){
    if(SPIFFS.exists("/LOGS.txt")) {
      request->send(SPIFFS, "/LOGS.txt", "text/plain");
    } else {
      request->send(200, "text/plain", "LOGS not found ! Restarting in 5 seconds..");
      delay(5000);
      ESP.restart();
    }
    
    });

  server.begin();

  File file2 = SPIFFS.open("/LOGS.txt");

  if(!file2){
      Serial.println("Failed to open file for reading");
      return;
  }

  Serial.println("File Content:");

  while(file2.available()){

      Serial.write(file2.read());
  }

  file2.close();

  Serial.println("End of file content");
}

void loop() {
  if ((WiFi.status() == WL_CONNECTED)) { //Check the current connection status
 
    HTTPClient http;
 
    http.begin("http://jsonplaceholder.typicode.com/comments?id=10"); //Specify the URL
    int httpCode = http.GET();                                        //Make the request
 
    if (httpCode > 0) { //Check for the returning code
        Serial.println();
        Serial.printf("[HTTP] GET... code: %d\n", httpCode);
        if(httpCode == HTTP_CODE_OK) {
            String payload = http.getString();
            Serial.println(payload);
            Serial.println();
          }
      } else {
          Serial.printf("[HTTP] GET... failed, error: %s\n", http.errorToString(httpCode).c_str());
        }
 
    http.end(); //Free the resources
  }
 
  delay(10000);
 
}

And it outputs:

Connected to the WiFi network
192.168.1.107
File Content:
text!
text!
End of file content

So it ouputs text! twice…
Shouldn’t it output debug messages ?


#6

You are appending to the log. Does another line appear when you reset the MCU?

see


#7

I understand that I append. That’s the point, isn’t it ? If it was FILE_WRITE it would’ve overwritten what was already in the file.

No. This is the whole output. Running it multiple times, adds to the number of text ! inside the file.

Connected to the WiFi network
192.168.1.107
File Content:
text!
text!
End of file content

I’ve read that multiple times but I didn’t want to believe it.

From what I’ve understand there there is no way to redirect the debug messages to a SPIFFS File, right ? And if that’s the case, than surely there is another way of debugging an ESP.

What is the solution to debugging an ESP 32 without a Serial Monitor ? Let’s say I install this ESP somewhere where it’s inconvenient to access. How is anybody supposed to debug the ESP if it crashes out of nowhere ?

Thank you !