Reading data from SD card back to a structure

meathome2017 · August 22, 2020, 5:12pm

Sorry guys I seem to be stuck again.
I have managed to write data from my data structure to the SD card successfully.
But I cannot get the ESP32 to read that data from the SD card and put it back into the structure.
I’ve looked at numerous articles and tried all sorts of different ways but to no avail.

In theory it looks so easy,
There are two parts to the code SD1 writes data to the SD card
SDW1 is my hopeless attempt to read the data back into the structure.

sorry to disturb your weekend.

#include <M5Stack.h>
#include <TaskScheduler.h>
#include "FS.h"
#include "SD.h"
#include "SPI.h"

const int chipSelect = 4;
File dataFile;

 typedef struct Board1 { 
        byte   sensornumber;        // Sensor number provided by e.g. Sensor=3
        String Time;                //Time
        float  Temperature;         // DHT Temperature
        float  Humidity;            // DHT Humidity
        float  BMP_Temperature;     // BMP Temperature
        float  BMP_Pressure;        // Bmp Pressure
        int Light;                  // Light 
    } Board1;

  struct Board1 Board_1;  
  //void SD1();
  void SDW1();
//###############################################################################
void setup() {
    Serial.begin(115200);
   
  M5.begin();
  
  M5.Lcd.fillScreen(DARKGREEN);
  M5.Lcd.setCursor(10, 10);
  M5.Lcd.setTextColor(WHITE);
  M5.Lcd.setTextSize(3);
  M5.Lcd.println("Starting!");
  M5.update();
// **dummy data**
  Board_1.BMP_Pressure=1010;
  Board_1.BMP_Temperature=26.6;
  Board_1.Humidity=77;
  Board_1.Light=555;
  Board_1.sensornumber=44;
  Board_1.Temperature=25.5;
  Board_1.Time="10:20:30";
  //SD1();
  SDW1();
}
//###############################################################################
void loop() {
  
}
//###############################################################################
void SD1(){
  auto filename="/Data1.csv";   
  auto myFile = SD.open(filename, FILE_APPEND);
  if(myFile){
    myFile.print(Board_1.sensornumber);
    myFile.print(",");
    myFile.print(Board_1.Time);
    myFile.print(",");
    myFile.print(Board_1.Temperature);
    myFile.print(",");
    myFile.print(Board_1.Humidity);
    myFile.print(",");
    myFile.print(Board_1.BMP_Temperature);
    myFile.print(",");
    myFile.print(Board_1.BMP_Pressure);
    myFile.print(",");
    myFile.println(Board_1.Light);
    myFile.close();
    Serial.println("Done");
    }
  else{
    Serial.println("Error opening Data1.csv");
  }
}
//#########################################################################
void SDW1(){

 if (!SD.begin(chipSelect)) {
    Serial.println("Card failed, or not present");
    return;
  }
  delay(2000);
  Serial.println("card initialized.");
  dataFile = SD.open("/Data1.csv", FILE_READ);  
  delay(2000);
  
  while (dataFile.available()){
    Serial.write(dataFile.read());
  }
    
    
  
    dataFile.close();
    
}

normandunbar · August 22, 2020, 8:20pm

Evening @meathome2017.

I’m looking at your code from the viewpoint of someone not familiar with your board or the SD card stuff, so apologies if I’m way off base.

When you write, you write each individual member of the Board1 structure. This makes me think that you cannot write the structure as a whole? Does the SD object have a member function which writes a byte array or charr array? If so, you could do something similar to:

myfile.print((char *)&Board_1, sizeof(struct Board1);

That would coerce the struct to a char array and write it out.

When you read back from the SD card, you are not reading back the individual fields of the struct, so I’m not sure what is being read back. You could read back the struct fields separately, into the struct, then Serial.write them individually.

Alternatively, if there is a method to write the struct as a byte array, you should also have a correspoding read function.

dataFile.read((char *)&Board_1, sizeof(struct Board1);

Then you could Serial.write the structure fields.

Curently you write individual fields, but read everything without the read knowing what data type it is supposed to be reading.

HTH

Cheers,
Norm.

PS. My weekend is not disturbed. I’ve had a couple of glasses of wine with my wife.

meathome2017 · August 23, 2020, 11:30am

Thank you Norman for your reply and the ideas that you suggested.
A couple of little points one I am very new to all of this and do not really know if it’s possible to write directly to the structure or not. Regarding your idea I can read it character by character but as the length of the string change it is not reliable enough to reload into the structure.
I’ve tried using
dataFile.read((char *)&Board_1, sizeof(struct Board1);
but I to be honest I do not know how to get it back into the structure,

I’m sure it must be possible, is just my lack of knowledge enabling me to put it together
any further suggestions from anybody would be much appreciated.

normandunbar · August 23, 2020, 4:02pm

Hi @meathome2017.

Ok, let’s see how good a teacher I am!

The examples for using SD card files with the Arduino SD card library (from adafruit) look to be incomplete in that you can write various data types to the disk file, but you can only read back one byte, or an array of bytes. You cannot do this:

int fred = 666;
...
dataFile.write(fred);
...

Then, to read it back:

int barney = 0;
...
dataFile.read(barney);
...

You can do this though:

dataFile.read(&barney, sizeof(barney));

you cannot read an int, for example, when you wrote one to the card. This is bad if you ask me. So, here are some helper functions you might find useful:

readByte(File dataFile, byte *data) {
    dataFile.read(data, sizeof(byte);
}

readShort(File dataFile, short *data) {
    dataFile.read(data, sizeof(short);
}

readInt(File dataFile, int *data) {
    dataFile.read(data, sizeof(int);
}

readLong(File dataFile, long *data) {
    dataFile.read(data, sizeof(long);
}

readFloat(File dataFile, float *data) {
    dataFile.read(data, sizeof(float);
}

readText(File dataFile, char *data, int dataSize) {
    dataFile.read(data, dataSize);
}

The above is untested, I don’t use SD cards. It should work though.

When you write to the Sd card, you do the following to write each field of the Board_1 struct to the disk file:

...
myFile.print(Board_1.Temperature);
myFile.print(",");
myFile.print(Board_1.Humidity);
myFile.print(",");
myFile.print(Board_1.BMP_Temperature);
...

You access each field in the struct, one by one, and write the textual representation of the byte, int and floats as text, to the SD card. Each is separated with a comma, so I presume you are writing some sort of CSV file?

The String (Board_1.Time) will be written in text format as well. I suspect it will be written with a trailing byte of 0 (zero) to terminate the text. So, given your example data, you would expect to see something like this in the SD card’s file:

44,10:20:30,25.5,77,26.6,1010,555

I say, expect because, on second reading of your code, you have initialised the Board1 struct in the setup() function, but then called SDW1() which reads some data off of the SD Card - you have not written the data out yet. (Or have you done so previously as I see SD1() commented out in setup()? I assume that you have done so.)

Looking at the documentation for the Arduino SD library, SD - Arduino Reference, the read() function takes two forms, the first returns a single character, the second read(buf, len) which reads len bytes into a buffer at address buf.

I agree,and I missed this first time, that the String would probably cause problems. So, to read the structure back, I would do this using the helper functions above:

readByte(dataFile, Board_1.sensornumber);
dataFile.read();

// This should work, but test it!
char tempTime[10];                    // Temp buffer, bigger than data to be read back.
readText(dataFile, tempTime, 8);      // Wrote "hh:mm:ss" plus a terminator.
Board_1.Time = tempTime;
dataFile.read();

readFloat(dataFile, &Board_1.Temperature);
dataFile.read();

readFloat(dataFile, &Board_1.Humidity);
dataFile.read();

readFloat(dataFile, &Board_1.BMP_Temperature);
dataFile.read();

readFloat(dataFile, &Board_1.BMP_Pressure);
dataFile.read();

readInt(dataFile, &Board_1.Light);
dataFile.read()             // The newline character?

dataFile.close();

Reading a String requires reading some text into a char buffer, then converting that to a String. It might be possible to read a string back directly using readText() but I never use String variables in my code as they are brutal in that they do a lot of allocating heap space, then deallocating and so on. This can quickly lead to out of memory problems and weird crashes. Copying a string, to make it longer, for example, needs enough space to have both copies in RAM at the same time.

Using the code above, you read each individual field back and also, read the comma between each one - datFile.read() returns a single character, so that accounts for the comma which we are ignoring.

Now, the other way to do this would be something like this for the write:

myFile.write((char *)&Board_1, sizeof(Board_1));

This one line will write the total number of bytes in the struct to the SD card. We have to convert the address of the start of the Board_1 variable to a char * (pointer to a char) We get the address of any variable using the & operator. So, the bit in brackets, (char *) is a cast which says, whatever comes after me, convert it to the same type as me. In this case, *Take the address of the structure named Board_1 and convert it to the data type char *. Simple.

sizeof returns the length, in bytes of the variable it references. If there are packing bytes involved (to align the floats, for example, on an address divisible by 4 or 8 or whatever) then they are included. That would write the whole struct to the SD card.

Reading it back as an array of bytes is just the opposite:

dataFile.read((char *)&Board_1, sizeof(Board_1));

This of course, assumes that reading a String from disk is the opposite of writing one, and internally, the String will be constructed correctly.

HTH

Cheers,
Norm.

meathome2017 · August 23, 2020, 6:51pm

Thank you ever so much Norman for your in depth teaching, many apologies for putting you through writing half a book. I think you deserve another glass of wine or two, while I sit down and try to absorb what you are saying and see if the compiler says yes!!!

I’m going to need a couple of days to try and sort this but thank you the immense amount of help you have given me.

I certainly come back and let you know how I get on.

Enjoy your evening.

Paul

meathome2017 · August 23, 2020, 7:49pm

Good evening again
I seem to be getting an error on the compiler
in that it does not seem to understand “readByte”
it reports that identifier “readByte” is undefined
the same error is being reported for “readInt” & readFloat

I must be missing something very obvious perhaps you can point me in the right direction.


File file = SD.open("/Data1.csv"); 
if(!file){
  Serial.println("Failed to open file for reading");
  return;
}
else
{
  Serial.println("File is open file for reading");
}
readByte (file , Board_1.sensornumber);
dataFile.read();




    Serial.println(Board_1.BMP_Pressure );
    Serial.println(Board_1.BMP_Temperature );
    Serial.println(Board_1.Humidity );
    Serial.println(Board_1.Light );
    Serial.println(Board_1.sensornumber );
    Serial.println(Board_1.Temperature );
    Serial.println(Board_1.Time );
    delay(250);    
}

normandunbar · August 23, 2020, 8:03pm

Hi @meathome2017 (Paul),

I’ve already written one published Arduino book, I have another in progress! Another “half” was no problem.

If readByte() etc are undefined, you need to copy my code above into your sketch. The best place is above any code which calls it.

However I’ve made an error above! It should be:

void readByte(File dataFile, byte *data) {
    dataFile.read(data, sizeof(byte);
}

void readShort(File dataFile, short *data) {
    dataFile.read(data, sizeof(short);
}

void readInt(File dataFile, int *data) {
    dataFile.read(data, sizeof(int);
}

void readLong(File dataFile, long *data) {
    dataFile.read(data, sizeof(long);
}

void readFloat(File dataFile, float *data) {
    dataFile.read(data, sizeof(float);
}

void readText(File dataFile, char *data, int dataSize) {
    dataFile.read(data, dataSize);
}

I missed the void.

You could write them another way to return the value read, but the above “just” works too.

Cheers,
Norm.

Give it a try and let me know how it goes.

(Don’t worry about asking more questions - I’m happy to help.)

Cheers,
Norm.

meathome2017 · August 23, 2020, 8:40pm

Sorry Norman one or two more errors now showing

I will also reload the code in case there’s something obvious I’m doing wrong?

#include <M5Stack.h>
#include <TaskScheduler.h>
#include "FS.h"
#include "SD.h"
#include "SPI.h"

void readByte(File dataFile, byte *data); 
void readShort(File dataFile, short *data);
void readInt(File dataFile, int *data);
void readLong(File dataFile, long *data);
void readFloat(File dataFile, float *data);
void readText(File dataFile, char *data, int dataSize);

const int chipSelect = 4;
File dataFile;

 typedef struct Board1 { 
        byte   sensornumber;        // Sensor number provided by e.g. Sensor=3
        String Time;                //Time
        float  Temperature;         // DHT Temperature
        float  Humidity;            // DHT Humidity
        float  BMP_Temperature;     // BMP Temperature
        float  BMP_Pressure;        // Bmp Pressure
        int Light;                  // Light 
    } Board1;

  struct Board1 Board_1;  
  //void SD1();
  void SDW1();
//###############################################################################
void setup() {
    Serial.begin(115200);
   
  M5.begin();
  
  M5.Lcd.fillScreen(DARKGREEN);
  M5.Lcd.setCursor(10, 10);
  M5.Lcd.setTextColor(WHITE);
  M5.Lcd.setTextSize(3);
  M5.Lcd.println("Starting!");
  M5.update();
  /*
  Board_1.BMP_Pressure=1010;
  Board_1.BMP_Temperature=26.6;
  Board_1.Humidity=77;
  Board_1.Light=555;
  Board_1.sensornumber=44;
  Board_1.Temperature=25.5;
  Board_1.Time="10:20:30";
  */
  //SD1();
  SDW1();
}
//###############################################################################
void loop() {
  
}
//###############################################################################
void SD1(){
  auto filename="/Data1.csv";   
  auto myFile = SD.open(filename, FILE_APPEND);
  if(myFile){
    myFile.print(Board_1.sensornumber);
    myFile.print(",");
    myFile.print(Board_1.Time);
    myFile.print(",");
    myFile.print(Board_1.Temperature);
    myFile.print(",");
    myFile.print(Board_1.Humidity);
    myFile.print(",");
    myFile.print(Board_1.BMP_Temperature);
    myFile.print(",");
    myFile.print(Board_1.BMP_Pressure);
    myFile.print(",");
    myFile.println(Board_1.Light);
    myFile.close();
    Serial.println("Done");
    }
  else{
    Serial.println("Error opening Data1.csv");
  }
}
//#########################################################################
void SDW1(){

dataFile = SD.open("/Data1.csv", FILE_READ); 
if(!dataFile){
  Serial.println("Failed to open file for reading");
  return;
}
else
{
  Serial.println("File is open file for reading");
}

//dataFile.read((uint8_t *)&Board_1, sizeof(Board_1));

auto readByte(dataFile , Board_1.sensornumber);
dataFile.read();




    Serial.println(Board_1.BMP_Pressure );
    Serial.println(Board_1.BMP_Temperature );
    Serial.println(Board_1.Humidity );
    Serial.println(Board_1.Light );
    Serial.println(Board_1.sensornumber );
    Serial.println(Board_1.Temperature );
    Serial.println(Board_1.Time );
    delay(250);
    
}
//####################################################################################
void readByte(File dataFile, byte *data) {
    dataFile.read(data, sizeof(byte));
}

void readShort(File dataFile, short *data) {
    dataFile.read(data, sizeof(short));
}

void readInt(File dataFile, int *data) {
    dataFile.read(data, sizeof(int));
}

void readLong(File dataFile, long *data) {
    dataFile.read(data, sizeof(long));
}

void readFloat(File dataFile, float *data) {
    dataFile.read(data, sizeof(float));
}

void readText(File dataFile, char *data, int dataSize) {
    dataFile.read(data, dataSize);
}

normandunbar · August 23, 2020, 9:34pm

Oh dear oh dear! I did say it was untested, however, it would help if I typed it in correctly. I blame the wine!

My apologies, for I am an idiot! My helper functions provide pointers to the different data types, when the pointer should be a char pointer - as I explained in my “half a book” post above. I need to follow my own instructions!

void readByte(File dataFile, byte *data) {
    dataFile.read((char *)data, sizeof(byte));
}

void readShort(File dataFile, short *data) {
    dataFile.read((char *)data, sizeof(short));
}

void readInt(File dataFile, int *data) {
    dataFile.read((char *)data, sizeof(int));
}

void readLong(File dataFile, long *data) {
    dataFile.read((char *)data, sizeof(long));
}

void readFloat(File dataFile, float *data) {
    dataFile.read((char *)data, sizeof(float));
}

void readText(File dataFile, char *data, int dataSize) {
    dataFile.read(data, dataSize);
}

The readText() function is fine, the rest need to have (char *) pasted/typed in front of the data variable name in the calls to dataFile.read().

Apologies. (Again.)

Hopefully, I’ve got the datatype for the dataFile parameters correct. You used auto in your code, I found File in the Adafruit docs. If you get further errors complaining about the dataFile parameters, then I’ve got the wrong information.

Cheers,
Norm.

meathome2017 · August 24, 2020, 11:37am

Hi Norman
I still have a lot of errors sorry to say.

“dataFile.read”
no instance of overloaded function “fs::File::read” matches the argument list – argument types are: (char *, unsigned long) – object type is: fs::File

and at the end of the helper
invalid conversion from ‘char*’ to ‘uint8_t* {aka unsigned char*}’ [-fpermissive]

Thanks for the help

normandunbar · August 24, 2020, 3:43pm

Hello again!

I’m not able to fully test this because I do not have any SD card interfaces for my Arduinos, however, I do have the built in SD library, supplied by Adafruit, in my Arduino IDE - I didn’t install it, it was “just” there! I have been assuming that this is the one you are using?

This code compiles ok. It’s the example SD sketch named “ReadWrite” and shows how to write data to an SD card and read it back. The reading is a bit iffy though, as it’s just dumping each character read to Serial - there’s no structure (pun intended!) to the returned data. The text written was originally “Testing 1 2 3” - a simple case of picking your test data to make it look good methinks!

Anyway, I’ve amended the code slightly. I’ll explain as I go along. See the comments beginning with //ND they mark my code additions.

/*
  SD card read/write

  This example shows how to read and write data to and from an SD card file
  The circuit:
   SD card attached to SPI bus as follows:
 ** MOSI - pin 11
 ** MISO - pin 12
 ** CLK - pin 13
 ** CS - pin 4 (for MKRZero SD: SDCARD_SS_PIN)

  created   Nov 2010
  by David A. Mellis
  modified 9 Apr 2012
  by Tom Igoe

  This example code is in the public domain.

*/

#include <SPI.h>
#include <SD.h>


// ND - I've added these two examples here, exactly as I explained in a previous post.
void readByte(File dataFile, byte *data) {
    dataFile.read((char *)data, sizeof(byte));
}

void readInt(File dataFile, int *data) {
    dataFile.read((char *)data, sizeof(int));
}
// ND Ends - End of my additions. More later.


File myFile;

void setup() {
  // Open serial communications and wait for port to open:
  Serial.begin(9600);
  while (!Serial) {
    ; // wait for serial port to connect. Needed for native USB port only
  }


  Serial.print("Initializing SD card...");

  if (!SD.begin(4)) {
    Serial.println("initialization failed!");
    while (1);
  }
  Serial.println("initialization done.");

  // open the file. note that only one file can be open at a time,
  // so you have to close this one before opening another.
  myFile = SD.open("test.txt", FILE_WRITE);

  // if the file opened okay, write to it:
  if (myFile) {
    Serial.print("Writing to test.txt...");

    // ND - Write a byte, and an iint to the SD card. I deleted the original text written.
    myFile.print(0xA5);
    myFile.print ((int)12345);
    // ND Ends - End of this bit. More later!
    
    // close the file:
    myFile.close();
    Serial.println("done.");
  } else {
    // if the file didn't open, print an error:
    Serial.println("error opening test.txt");
  }

  // re-open the file for reading:
  myFile = SD.open("test.txt");
  if (myFile) {
    Serial.println("test.txt:");


    // ND - Define a byte and int variable
    byte fred;
    int  wilma;
    
    // ND - Read back a byte and an int from the SD card.
    readByte(myFile, &fred);
    readInt(myFile, &wilma);
    // ND Ends. End of all my additions, nothing to follow.

    // close the file:
    myFile.close();
  } else {
    // if the file didn't open, print an error:
    Serial.println("error opening test.txt");
  }
}

void loop() {
  // nothing happens after setup
}

It would be interesting to know if you are able to compile this and test it to see if it does write and read correctly. Thanks.

One thing you can do, when you have written some data to the SD card, eject it and plug it nto your laptop/desktop. Can you see the file(s) on the card? if so, can you open them and see the data? The data I’ve written above won’t be readable in taext mode though. They will be written as binary data, the internal representation of a byte and an int, and not as the plain text digits that these “numbers” represent.

HTH

Cheers,
Norm.

meathome2017 · August 24, 2020, 5:35pm

Good evening Norman I have exactly the same errors on your helper files as I did before. Question are we compiling for the same board ESP32?
I just copied and pasted the code as you supplied no changes.

Thank you for your persistence

normandunbar · August 24, 2020, 6:25pm

Ah! I’m compiling for an Arduino Uno. I’m using, as I mentioned, the SD library from Adafruit, which is built in to the Arduino IDE.

I don’t have an ESP32 (yet!).

Which library are you using?
Is there documentation on line that I can read?
Do you know what/how the read() function is defined?

At least it’s not complaining about the dataFile parameter!

Cheers,
Norm.

normandunbar · August 24, 2020, 6:29pm

I think I’ve worked it out!

Change the cast to char * to a cast to uint8_t * instead.

void readByte(File dataFile, byte *data) {
    dataFile.read((uint8_ *)data, sizeof(byte));
}

Youll need it in every funtion.

Cheers,
Norm.

meathome2017 · August 24, 2020, 7:11pm

Well Norman it seems that Fred and Wilma are working well together.
First programme that compiles and runs and in the Test.txt file is 16512345

Now I need to build upon this base I presume?

normandunbar · August 24, 2020, 9:08pm

Hooray for Fred and Wilma!

Really? In text format? And in exactly that layout? No linefeeds etc between the 165 and 12345? Weird. You should have seen one byte followed by 2 or 4 for the int. All in binary. I’m surprised it reads back.

What happens if you add Serial.println(fred, HEX); and Serial.println(wilma); just above myFile.close();? Do you see A5 and 12345?

Cheers,
Norm.

meathome2017 · August 25, 2020, 1:50pm

Sorry to say no, I get :
49
842085686

if that helps.

normandunbar · August 25, 2020, 2:47pm

This is getting weirder and weirder!

0xA5, the byte, is decimal 165. You posted above that the text file shows “16512345”, which makes no sense as you appear to be seeing the textual/ascii representation of the byte and the int. We should be seeing the binary/internal representation.

Reading it back should be the same as what was written. 0xA5, and 12345.

0x49 is 73 decimal and nothing like 0xA5. 84208586 is defintely not 12345 either. This is bugging me now! I’m considering getting hold of an SD card interface and debugging this problem. Problem is it takes ages for deliveries to arrive in the wilds of Yorkshire!

I’m a bit stumped now.

Maybe we need to write a linefeed between the two values when we are writing?

I’ll have a think.

Cheers,
Norm.

normandunbar · August 25, 2020, 4:18pm

Ordered! I refuse to be beaten by a machine!

https://www.amazon.co.uk/gp/product/B077MB17JB/ref=ppx_yo_dt_b_asin_title_o00_s00?ie=UTF8&psc=1

Cheers,
Norm.

meathome2017 · August 25, 2020, 6:55pm

It is very kind of you Norman to put yourself out like this thank you again for your kindness.
It is very frustrating I must admit this seems to be so easy to store the information on the device but at the moment appears to be virtually impossible to read it back.
As for reading the chip I just opened it up on my laptop and use notepad to look at the file, and I was a bit perplexed to find it still in a readable format. It’s bound to be something stupidly obvious.

Let me know how you get on.
Paul