Cutting AWS costs

What the job is

An AWS cost program lowers what the organization pays for the same work, then keeps it there. It has three parts: knowing who spends what, using less, and paying less for what is used.

The FinOps Foundation calls the practice FinOps. It defines FinOps as "an operational framework and cultural practice which maximizes the business value of technology, enables timely data-driven decision making, and creates financial accountability through collaboration between engineering, finance, and business teams."

Its framework has three phases: Inform, Optimize and Operate. Inform means examining cost, usage and efficiency data. Optimize means finding ways to improve efficiency and value. Operate means putting those changes into practice.

Cut once or change how spending works

The first decision is scope. A one-time cut removes idle resources, rightsizes the obvious cases and buys commitments. Assign an owner to each account and workload so the savings hold.

A standing program assigns spend to teams, reviews it on a fixed cycle and sets rules for new resources. The FinOps Foundation's principles include "Everyone takes ownership for their technology usage" and "FinOps should be enabled centrally". One way to apply them: a small central team, and engineering owners who act on what it finds.

Engineering teams decide architecture. A program that finance runs alone can buy commitments but cannot change architecture.

The levers

Savings Plans do not apply to Spot usage or to usage already covered by Reserved Instances. Reserved Instances apply first, then EC2 Instance Savings Plans, then Compute Savings Plans.

The order of the work

1. Get the data. Turn on Cost Explorer. It shows up to 13 months of history and forecasts 18 months ahead. Use the AWS Cost and Usage Report or Data Exports for line-item detail.
2. Allocate the spend. Agree on a tagging standard: owner, application, environment, cost center. Activate the tags in the Billing console. User-defined and AWS-generated tags must each be activated before they appear in Cost Explorer. Tags can take up to 24 hours to appear.
3. Set alerts. Configure AWS Budgets and Cost Anomaly Detection, with alerts to each owner. Anomaly detection can take up to 24 hours after the usage occurs.
4. Remove waste. Delete unattached volumes, old snapshots, idle load balancers and unused Elastic IP addresses. Stop non-production instances outside working hours.
5. Rightsize. Opt in to Compute Optimizer. Move EBS gp2 volumes to gp3. Resize instances, databases and Lambda memory settings with the owners' agreement.
6. Commit. Buy Savings Plans or Reserved Instances against the usage left after rightsizing, not before. Start with a share of the steady baseline and add more as usage proves stable.
7. Change architecture. Move suitable workloads to Spot, add gateway VPC endpoints, and review data transfer paths. Plan for this step to need the most engineering time.

What takes longer than expected

Allocation is slow. Tags only work on resources that carry them. Resources created before the standard carry none until someone adds them. Management account users can backfill activation status for up to 12 months, but backfill adds no tag to a resource that never had one.

Rightsizing waits on owners. Each change needs someone to accept the performance risk, test it and schedule it. Compute Optimizer's default 14-day window can miss monthly batch peaks. Ask about month-end and quarter-end loads before resizing.

The detail plans leave out

• NAT gateways. Data processing is charged for every gigabyte through the gateway, on top of an hourly charge. AWS's US East (Ohio) example uses $0.045 per hour and $0.045 per GB. Traffic to S3 can go through a gateway VPC endpoint instead, which has no hourly or data processing charge.
• Public IPv4 addresses. AWS charges $0.005 per hour for each public IPv4 address, in use or idle.
• Data transfer out. The first 100 GB per month to the internet is free, pooled across services and Regions except China and GovCloud. Above that, rates are tiered by aggregate usage across EC2, S3 and other services.
• Dedicated Instances. AWS charges $2 per hour in each Region with at least one running. Savings Plans do not discount that fee.
• EKS. Compute Savings Plans apply to EC2 and Fargate usage. Every EKS cluster also has a fee per cluster per hour.

Commitments and the contract

A Savings Plan is a contract for a dollar amount per hour, for 365 days or 1,095 days. AWS states it cannot be canceled during the term. Each hour's commitment is used in that hour or lost. Usage above it is charged at On-Demand rates.

There is one exit. A Savings Plan with an hourly commitment of $100 or less, bought within the last 7 days and in the same calendar month, can be returned for a full refund of upfront charges. Returns are subject to a quota.

Standard Reserved Instances can be sold on the Reserved Instance Marketplace. Convertible RIs and RIs for RDS or ElastiCache cannot. A seller needs a US bank account, the RI must have been active for 30 days with at least one month left, AWS takes a 12% fee on the upfront price, and an account can sell no more than $50,000 in RIs over its lifetime.

In an AWS Organization, Savings Plans apply first to the buying account, then to other accounts if sharing is turned on. Decide who buys and who receives the discount before the first purchase, so finance can charge it back.

Skills and operations

The work needs three skills in one room: someone who reads the bill and the Cost and Usage Report, engineers who know the workloads, and someone in finance who owns the forecast.

Give each account and each major workload a named owner who receives its alerts and its monthly cost. Review commitment coverage and utilization on a fixed schedule and before each purchase.

Settle these first

• What share of spend can be assigned to an owner today?
• Which tag keys are mandatory, and who enforces them?
• Who approves a Savings Plan or RI purchase, and against what forecast?
• Is Savings Plans discount sharing on across the Organization, and who gets the credit?
• Which workloads could tolerate a two-minute interruption?
• What is the target: a lower bill, or a lower cost per transaction or customer?

Questions

Should we buy Savings Plans or Reserved Instances?

AWS puts Compute Savings Plans at up to 66% off On-Demand, like Convertible RIs, and EC2 Instance Savings Plans at up to 72%, like Standard RIs. Savings Plans apply without manual exchanges. Standard RIs can be resold on the RI Marketplace; Savings Plans cannot be canceled.

Can we get out of a Savings Plan?

Only within a short window. A plan with an hourly commitment of $100 or less, bought in the last 7 days and in the same calendar month, can be returned. After that it runs to the end of its 1- or 3-year term.

How much warning does a Spot Instance get?

Two minutes, except with hibernation, which starts at once. EC2 sends the notice as an EventBridge event and in instance metadata. AWS recommends checking for it every 5 seconds. Notices are best effort.

Can tags be applied to past costs?

Partly. Management account users can backfill tag activation for up to 12 months. Costs only show tag values for periods when the resource actually carried the tag.

Is moving from gp2 to gp3 disruptive?

Not if the instance supports Elastic Volumes. Elastic Volumes can change the volume type without detaching the volume or restarting the instance, if the instance supports it. A 1 TiB volume can take up to six hours to finish modifying.