Managing data retention in S3 Storage

S3 Storage is a distributed data storage system where each file (object) is kept in a dedicated container called a bucket. Data is accessed via the S3 API, which ensures compatibility with AWS tools and other S3-compatible platforms.

---

The Lifecycle feature in S3 storage allows you to automate actions on objects after a specified period of time. The most common use case is automatically deleting files after a certain number of days or on a specific date. This helps optimize storage usage by cleaning up temporary data, backups, and other files with a limited retention period.

---

Currently, the storage supports:

1. Expiration (automatic deletion of objects after a retention period) Deletes objects after a specified number of days from upload or on a specific date.

2. Prefix Limits a rule’s scope to a specific path (for example, only to the /logs/ folder).

3. Limitations

   • The minimum value is 1 day.
   • The deletion date must be set to 00:00 UTC.

---

Deleting All Objects After 30 Days

Create a lifecycle.json file:

{
  "Rules": [
    {
      "ID": "DeleteAfter30Days",
      "Prefix": "",
      "Status": "Enabled",
      "Expiration": {
        "Days": 30
      }
    }
  ]
}

Apply the configuration to your bucket:

aws s3api put-bucket-lifecycle-configuration \
  --bucket your-bucket-name \
  --lifecycle-configuration file://lifecycle.json

Check the current configuration:

aws s3api get-bucket-lifecycle-configuration \
  --bucket your-bucket-name

Delete the lifecycle rule:

aws s3api delete-bucket-lifecycle \
  --bucket your-bucket-name

---

Deleting Files in a Specific Folder

If you want to delete files only within a specific directory, such as logs/, you can use the Prefix parameter.

{
  "Rules": [
    {
      "ID": "DeleteLogsAfter7Days",
      "Prefix": "logs/",
      "Status": "Enabled",
      "Expiration": {
        "Days": 7
      }
    }
  ]
}

After applying this rule, only objects stored under the logs/... path will be deleted.

---

Deleting Files on a Specific Date

If you want all objects to be deleted on a specific date, use the Date parameter.

{
  "Rules": [
    {
      "ID": "DeleteAtMidnight",
      "Prefix": "",
      "Status": "Enabled",
      "Expiration": {
        "Date": "2025-12-01T00:00:00Z"
      }
    }
  ]
}

---