Managing data retention in S3 Storage

S3 storage is a distributed data storage system where every file is kept in a specialized container called a bucket. Data access is managed via the S3 API, ensuring full compatibility with AWS tools and other industry-standard services.

A lifecycle in S3 is a "set-and-forget" mechanism that automates actions on objects over time. The most common use case is the automatic deletion of files after a specific number of days or on a pre-defined date. This helps optimize your storage footprint by automatically purging temporary data, logs, and aging backups.

Supported features

1. Expiration — removes objects after a set number of days from upload or on a specific calendar date.
2. Prefix — narrows the rule's scope to a specific path (e.g., only targeting the logs/ folder).
3. Constraints:
   • Minimum value — 1 day.
   • Execution time — deletions are processed at 00:00 UTC.

Deleting all objects after 30 days

To set up an automated cleanup for your bucket, follow these steps:

Step 1. Creating a configuration file Create a local file named lifecycle.json with the following configuration:

{
  "Rules": [
    {
      "ID": "DeleteAfter 30Days",
      "Prefix": "",
      "Status": "Enabled",
      "Expiration": {
        "Days": 30
      }
    }
  ]
}

Step 2. Applying the configuration to the bucket Use the AWS CLI utility to upload the settings to your storage by running the aws s3api put-bucket-lifecycle-configuration command:

aws s3api put-bucket-lifecycle-configuration \
  --bucket your-bucket-name \
  --lifecycle-configuration file://lifecycle.json

Step 3. Verifying the active rules To confirm the configuration was applied successfully, run the aws s3api get-bucket-lifecycle-configuration command:

aws s3api get-bucket-lifecycle-configuration \
  --bucket your-bucket-name

Deleting files in a specific folder

If you only want to purge a specific directory—such as logs/—define the Prefix parameter.

{
  "Rules": [
    {
      "ID": "DeleteLogsAfter7Days",
      "Prefix": "logs/",
      "Status": "Enabled",
      "Expiration": {
        "Days": 7
      }
    }
  ]
}

Once applied, this rule will only affect objects located within the logs/... path.

Deleting files on a specific date

To set a hard deadline for data retention, use the Date parameter.

{
  "Rules": [
    {
      "ID": "DeleteAtMidnight",
      "Prefix": "",
      "Status": "Enabled",
      "Expiration": {
        "Date": "2026-12-01T00:00:00Z"
      }
    }
  ]
}

Key insights for efficient management

To ensure your lifecycle rules behave as expected, keep these technical nuances in mind:

• Execution delay: lifecycle rules are typically evaluated once a day at 00:00 UTC. However, the physical removal of data can take up to 24 hours after the expiration period has passed.

• Checking expiration status: you can see exactly when an object is scheduled for deletion by checking its metadata. Run the head-object command and look for the x-amz-expiration header in the response.

• Virtual folders: S3 is a flat storage system. "Folders" are just prefixes in file names. If a rule deletes all objects with a specific prefix, the "folder" will effectively disappear from the console list.