I have an ESP32 board and I use the adruino framework in VScode in my platformIO project.
I want to download/upload a file from/to an S3 bucket. The bucket and its content should be private (blocked to public). Since the file size is approx. between 1MB (download) and 10MB(upload) I would like to use HTTPS. The 10MB logfiles are stored on an SD card.
I have found several hints on what could be used to achieve this:
I have already spent at least 10h of research and a little try n error testing some exampels without success and still not knowing what is the most promising approach to accomplish my goals. I already use MQTT to connect to AWS IoT bi-directional which works fine but I am struggling to set up the correct header to get access to my S3 bucket.
Could someone either
-
point me in the right direction, which libraries/Frameworks (e.g. AWS FreeRTOS) I should include into my project or
-
provide me with a tutorial on how to configure my HTTPS header correctly (especially on how to authenticate correctly to S3 using my AWS Access/Secret key and/or certificates) to access a non-public S3 bucket to download/upload form and to.
So, I would like to use the WiFiClientSecure library and not the HTTPClient library, because the latter one is somehow limited or I simply do not understand how to send my “self-made” HTTPS request header using it.
This is my code:
WiFiClientSecure ota_client;
ota_client.setCACert(AWS_CERT_CA);
String bucket_url = "https://" + bucket_name + "." + service + "." + region + "." + host + "/" + object;
// Example bucket_url: "https://test_bucket.s3.eu-central-1.amazonaws.com/firmware.bin"
ota_client.connect(bucket_url.c_str(), 443);
ota_client.print(http_header);
// Wait for the response
while (ota_client.connected() && !ota_client.available());
// Read and print the response
while (ota_client.available())
{
String line = ota_client.readStringUntil('\r');
Serial.print(line);
}
The AWS_CERT_CA is the root certificate I am also using for connecting to the AWS IoT MQTT broker.
The http_header I am trying to send has been built following these AWS documentations:
sig-v4-header-based-auth
GetObject
create-string-to-sign
This is my canonical request:
GET
/firmware.bin
content-type:application/octet-stream
host:test_bucket.s3.amazonaws.com
x-amz-date:20230308T151538Z
host;x-amz-date
e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
This is my string to sign (hash has been altered):
AWS4-HMAC-SHA25
20230308T151538Z
20230308/eu-central-1/s3/aws4_request
08f718a3ffa27f18fad718eab178f3564f1a1c2f389540cc36f17e329304bc45
This is my final header (content has been adapted regarding AWS ACCESS key and signature):
GET /firmware.bin HTTP/1.1
Host: test_bucket.s3.eu-central-1.amazonaws.com
Date: Wed, 08 Mar 2023 15:15:38 GMT
Authorization: AWS4-HMAC-SHA25 Credentials=AKIAZOKIXQT4EXAMPLE/20230308/eu-central-1/s3/aws4_request,SignedHeaders=host;x-amz-date,Signature=0d4e215b75606f5e8ff45f8b87cac62d7c9a2e58b6cead7fdc34886a7417154f
The errors I ger are:
[ 33426][E][WiFiGeneric.cpp:1438] hostByName(): DNS Failed for https://test_bucket.s3.eu-central-1.amazonaws.com/firmware.bin
[ 33430][E][WiFiClientSecure.cpp:135] connect(): start_ssl_client: -1
So I believe at this point something with the certificate but also with the URL is wrong. I checked the firmware.bin URL in AWS and it is exactly the same as the one I use for the connection. Now I hope someone can help me with a few things:
- Is the certificate I am using the correct one - it was generated when I created a “Thing” in AWS IoT? If not how can I generate a proper one to connect to S3?
- Does my general signing approach look correct or can you see any issues?
- To create hashs, I use my AWS SECRET ACCESS key, is this correct? It is not mentioned in the third link I provided.
- Is there something wrong in my general approach to connect to S3 via HTTPS?
Any help would be great!
No. WiFiClientSecure is a wrapper around Just TLS. When you use the connect() method, it expects a hostname. Not the full URL. So you’d just give it test_bucket.s3.eu-central-1.amazonaws.com.
Make sure this http_header string ends in two \r\n linebreaks, otherwise the HTTP server won’t recognize that the request has ended.
Thank you for the response.
Make sure this http_header string ends in two \r\n linebreaks, otherwise the HTTP server won’t recognize that the request has ended.
Thanks for the hint, I added them both. Besides, in the AWS documentation they always say:
Create a string by concatenating the following strings, separated by newline characters.
Do you think they mean using \r\n or simply \n?
It’s CRLF (\r\n). But if your your HTTP request has a message body then a final CRLF CRLF is not needed, only between header end and message body start, just like RFC 2616 says
Ok, thank you for the hint I will update my code.
Regarding the conncetion, I still get the same error:
[ 44752][E][WiFiGeneric.cpp:1438] hostByName(): DNS Failed for test_bucket.s3.eu-central-1.amazonaws.com/firmware.bin
[ 44780][E][WiFiClientSecure.cpp:135] connect(): start_ssl_client: -1
This is still using the full URL. Can you show the updated code in full?
Oh, my bad, I forgot to remove the object string at the end of the bucket URL. Here is my code.
String bucket_url = bucket_name + "." + service + "." + region + "." + host;
Serial.println("Bucket URL: ");
Serial.println(bucket_url);
WiFiClientSecure ota1_client;
ota1_client.setCACert(AWS_CERT_CA);
ota1_client.connect(bucket_url.c_str(), 443);
ota1_client.print(http_header);
// Wait for the response
while (ota1_client.connected() && !ota1_client.available());
// Read and print the response
while (ota1_client.available())
{
String line = ota1_client.readStringUntil('\r');
Serial.print(line);
}
ota1_client.stop();
And now I am finally receiving a response (altered at sensitive location).
Bucket URL:
test_bucket.s3.eu-central-1.amazonaws.com
HTTP/1.1 400 Bad Request
x-amz-request-id: 1JTPQC3C4ET8JTXB
x-amz-id-2: qVT0RKD/ScBqX51/TdjtT8B9w1/A+01pkYFalSr1yJ9/jJc4Pan5OV+Q226wVtPQHN/X+qSBjtk=
Content-Type: application/xml
Transfer-Encoding: chunked
Date: Wed, 08 Mar 2023 17:38:02 GMT
Server: AmazonS3
Connection: close
20f
<?xml version="1.0" encoding="UTF-8"?>
<Error><Code>InvalidArgument</Code><Message>Unsupported Authorization Type</Message><ArgumentName>Authorization</ArgumentName><ArgumentValue>AWS4-HMAC-SHA25 Credentials=AKIAZOKIXQT4DEXAMPLE/20230308/eu-central-1/s3/aws4_request,SignedHeaders=host;x-amz-date,Signature=0cea32460ea4345096d06dc8a35346aadb83d4ceeb1e712cb8185bda197dd</ArgumentValue><RequestId>1JTPQC3C4ET8JTXB</RequestId><HostId>qVT0RKD/ScBqX51/TdU8KlI0O/A+01pkYFalSr1yJ9/jJc4Pan5OV+Q226wVtPQHN/X+qSBjtk=</HostId></Error>
0
Well, it’s at least something. Now you probably have to corect the authentication headers or the authentication token you used already ran out?
Yes I have to double and triple check the documentation to ensure request signing. Thank you for your help so far.
