Guides

Architecture & Data Flow

TierPilot's design goal is simple to state: the components that can see your data run in your account; the component that leaves your account can only see anonymized aggregates. This page describes the pieces and the exact data that crosses the boundary.

Components

In your account: the Collector

A serverless pipeline (Lambda + Athena over your access logs) deployed by the connector stack. On a daily schedule it:

  1. Parses new S3 server access log objects for the monitored buckets.
  2. Joins them with the S3 Inventory metadata snapshot.
  3. Aggregates per object: read counts over rolling windows, last-access recency, size, age, current storage class.
  4. Tokenizes every object key using HMAC-SHA256 with a salt generated in your account at install time and stored in your AWS Secrets Manager. The salt never leaves your account, making tokens irreversible to us.
  5. Ships the aggregate rows to the TierPilot API over TLS.

In our cloud: the Optimizer

Receives the anonymized aggregates and runs the cost model: for each object token, it prices every storage class against the observed access pattern — storage rate, expected retrieval fees, transition request cost, and minimum-duration risk — and emits a transition plan for tokens where the switch has provable positive ROI with a safety margin.

In your account: the Operator

A Lambda exposing a small authenticated API (the Operations API). It receives the transition plan — a list of (object_token, target_storage_class) pairs — resolves tokens back to keys locally using the in-account salt, validates each move against your guardrails (excluded prefixes, tier floors), and executes the transitions. Every S3 call it makes lands in your CloudTrail like any other API activity in your account.

What crosses the boundary

The complete schema of a record sent to TierPilot's servers:

{
  "account_ref":  "acct_7c21...",      // your TierPilot account id
  "bucket_ref":   "bkt_e01a...",       // salted token, not the bucket name
  "object_ref":   "obj_9f3ba2e1c8",    // salted token, not the key
  "size_bytes":   52428800,
  "storage_class": "STANDARD",
  "age_days":     412,
  "reads_7d":     0,
  "reads_30d":    0,
  "reads_180d":   2,
  "last_access_days_ago": 97,
  "content_type_class": "binary"       // coarse bucket: text/image/binary/archive
}

What never crosses the boundary: object keys, object contents, bucket names, prefixes, tags, ACLs, requester identities, or IP addresses from your access logs.

Failure modes

  • TierPilot's API is unreachable: the collector retries with backoff; aggregates queue in your account. No optimization runs are missed, only delayed.
  • The operator receives an invalid or replayed plan: plans are signed and single-use; the operator rejects anything unsigned, expired, or violating your guardrails, and reports the rejection.
  • You delete the stack mid-run: in-flight transitions complete or fail atomically per object via the S3 API; nothing is left in an inconsistent state.