Get/Upload/Delete files from AWS-S3 bucket using Python

Using Python, we can upload the files & get the content of the files and update the existing files and also download the files from the S3 bucket.

AWS- Amazon Web Services provides a service called S3 service which is used to store the files and data that you like. S3 indicates a Simple Storage Service.

The First Step is to create a bucket in the AWS-S3 service. You can keep bucket as private or public depending upon your necessity.

After creating an S3 bucket, establish AWS-EC2 service that communicates with the S3 bucket.

Establishing EC2 communication with S3 bucket:

Launch AWS-EC2 instance in the AWS console and select the necessary volume and AMI accordingly. After launching AWS instance, go to Actions and look for Instance Settings, and then click Attach/Replace IAM Role. Add the IAM role that consists of the S3 service Full Access Policy.

First, create an IAM role that consists of S3 Full access by adding permissions or else by attaching a policy in the AWS-IAM service.

So, therefore EC2 instance will communicate with the S3 bucket with full access through the S3 bucket. (We can create IAM role by while launching AWS-EC2 instance also.)

The use of launching the AWS instance and S3 bucket communication is that we need to use AWS Secret Key and Secret Access Key in our code.

But we need to install python awscli in our instance by logging into our server. (By using ssh -i pem_file ubuntu@ip_address)

Configure the AWS S3 IAM role credentials after installing python awscli by giving the following command.

aws configure

It asks for the secret key and secret access key and region. Provide it accordingly, so that it will be safe in our server environment and then export the credentials in our server by giving the commands like

export AWS_SECRET_KEY = ‘ABC’
export AWS_SECRET_ACCESS_KEY = ‘DEF’

Now we can start writing our logic to perform file operations.

Prerequisites:

pip install boto3==1.9.232
pip install botocore==1.12.232
pip install awscli9.232

Initialize the S3 Manager Class and Establish the Connection:

Initializing the class S3Manager class to establish a connection with the S3 bucket.

Get the Pre Signed URL:

Get the Presigned URL for the specified bucket.

Download the Folder from the S3 bucket:

Download the specified folder from the S3 bucket by mentioning in the source path.

Mention the source path by specifying the folder path like

Eg: source = “folder/”
download_folder_from_s3(source)

Get the Content from the S3 bucket:

Get the content from the S3 bucket by mentioning the key. (s3 path)

Mention the S3 key to download the respective file and gets its content.

Eg: s3_key = “folder/file_name”
get_content_from_file(s3_key)

Upload the file to S3 bucket:

Upload the file content to the respective path by mentioning the destination path.

Mention the content of the file and the destination path of the file like:

Eg: content = { “data”: “Abc” }, destination =    “folder/file_name.json”
put_key_content(content=content, destination=destination)

This way, we can upload and download the files by calling the respective methods in our Python Files.

We can also perform necessary operations through the terminal.

Run AWS command line to download folder from S3:

aws s3 sync s3://bucket_name destination_path

Run AWS command line to Remove folder from S3:

aws s3 rm s3://bucket_name/ --recursive

Run AWS command line to Upload folder from S3:

aws s3 cp source_path s3://bucket_name/folder_name/ --recursive

References:

Using high-level (s3) commands with the AWS CLI