Simple Storage Service (S3)

Icon

S3 Icon

About

Introduction

• Amazon Simple Storage Service (Amazon S3) is an object storage service that offers industry-leading scalability, data availability, security, and performance.

• Use cases include

  • Backup and storage

  • Disaster Recovery (DR)

  • Archive

  • Hybrid Cloud Storage

  • Media and application hosting

  • Data lakes and big data analytics

  • Software delivery

  • Static website hosting

• Stores files (object) in buckets (directory like) and these buckets have globally (across all region) unique names.

Buckets

• Buckets are region level entities.

  • Bucket naming convention is as follows,

    • No uppercase

    • No underscore

    • Names can be 3 - 63 characters long.

    • Must start with lower case letter or number.

    • Must not start with xn-- prefix.

    • Must not end with -s3alias suffix.

    • Must not be IP address.

Objects

• Object files have key, which is a full path.

  • Key example, s3://my-bucket/file.txt

  • Key is composed of prefix (s3://bucket-name/folder-path..) and object name (file name).

  • Keys are nothing but very long path names with slashes.

• Max size of objects are 5 TB. There is no concept of directories within buckets.

• Object values are contents of the body.

  • Files bigger than 5 GB must be uploaded with multi-part upload feature.

• Objects have metadata (key-value pairs)

• Supports versioning of objects.

• Have tags which can be used for security/lifecycle.

• S3 Presigned URL enables the owner of the object view the object using this URL as they contain the owner details as token encoded in this URL.

• Public URL associated with S3 Object can only be used to access publicly if no access policy explicitly denies the object and bucket level settings is explicitly set to allow it.

Security

User based

• IAM policy.

Resource based

Bucket policy

• Bucket wide rules from S3 Console.

• Allows cross account based access, grant public access or encrypt object when uploading.

Note bucket and its object will only be public if it is not blocked by Block Public Access settings at account level.

• JSON based policy.

• Sample policy example,

      {
          "Version":"2012-10-17",
          "Statement":[
              {
                  "Principal":{
                      "AWS":"arn:aws:iam::111122223333:role/JohnDoe"
                  },
                  "Effect":"Allow",
                  "Action":[
                      "s3:GetObject",
                      "s3:GetObjectVersion"
                  ],
                  "Resource":"arn:aws:s3:::DOC-EXAMPLE-BUCKET/*",
                  "Condition":{
                      "StringEquals":{
                      "s3:ExistingObjectTag/environment":"production"
                      }
                  }
              }
          ]
      }

• Also has block all public access settings at bucket level.

• At account level also we can set access policy to make sure none of the buckets are ever publicly accessible.

Object Access Control List

• Can contol access at object level.

Bucket Access Control List

• Can contol access at bucket level.

Note: An IAM principal can access S3 object, if the user IAM permission ALLOW it OR the resource policy allows it, and there's not explicit DENY.

Encryption

• Can enable encryption using encryption keys.

Static Web Hosting

• S3 can host static websites and make them publicly accessible.

  • A typical S3 URL might look like below

        http://bucket-name.s3-website-aws-region.amazonaws.com
        http://bucket-name.s3-website.aws-region.amazonaws.com

Note : Ensure public read access is set explicitly on S3 bucket.

Versioning

• Bucket level setting that needs to be enabled to access this feature.

• Every upload for same key will version the object when this feature is enabled.

• Deleting a version of the object is permanent and cannot be undone.

• Versioning along with delete markers (added to a versioned object) helps to restore the file against accidental deletion.

• Enables roll back to previous version.

• A file that is not versioned prior to enabling this feature will have version "null".

• Allows suspending the versioning, though this does not delete previous versions, that were made when this feature was enabled.

• Deleting a delete marker version of the file (the file which was already deleted) makes it possible to restore the previous version of file.

Replication

• Replication rules control how replication will be done.

• Must enable versioning in source and destination buckets.

  • These buckets can be in same account or different accounts.

• There are two types of replication.

  • Cross Region Replication (CRR)

  • Same Region Replication (SRR)

• Only possible to replicate new objects once replication is enabled.

• To replicate existing object before replication was enabled one must use S3 Batch Replication feature.

  • This replicates existing as well as objects that failed replication.

• Can replicate delete markers from source to destination, by default it is disabled but can be enabled.

• Deleting a file with a version ID (i.e a version of an object) are not replicated (this features thus avoid malicious deletes).

• However, if existing object is versioned (i.e object created before replication) and a new version is added this new version of object along with previous version will be replicated.

• Copying is asynchronous.

• Buckets can be in different AWS account.

• Replication works only if proper IAM permission are provided to S3 service.

• Does not support replication chaining.

Storage Classes

Introduction

• Can move between classes manually or using S3 Lifecycle configurations.

Durability

• Provides 11 9's of durability guarantee for all storage classes.

  • This means if you store 1,000,000 objects in S3, you will loose a single object once every 10,000 years.

Availability

• Depending on storage class availability varies.

• Most storage class provides 99.99% availability (about 53 minutes not available).

Comparison

S3 Storage Tier Comparison

Reference

• S3 Storage

• Multipart Upload

• Policy Examples

• Replication