Capacity Advisor and Upgrade Scenarios

1. The real problem: calm averages, dangerous peaks

A service can average about 45% CPU and still suffer twice-weekly latency spikes while memory headroom collapses. If the average is the only signal, downsizing may look reasonable even though the next burst would make capacity worse. This lesson separates convenient numbers from decision-grade evidence.

2. Lesson boundary

This lesson owns Capacity Evidence & Upgrade Scenario Dossier v1. It consumes performance/provenance evidence without redesigning the adjacent performance-lens outcome. It does not define provider adapter policy and it does not execute a production resize, autoscaling change, or architecture migration. The output is a bounded recommendation.

3. Capacity Advisor mental model

The decision chain is: demand baseline → saturation signals → freshness → bottleneck hypothesis → headroom → scenario set → cost/risk comparison → recommendation state → separate execution authorization. When a decisive link is weak, preserve INSUFFICIENT_DATA, STALE_EVIDENCE, NOT_COMPARABLE, DEFERRED, or UNKNOWN rather than forcing a clean answer.

4. Demand window and baseline

The baseline records workload type, observation window, throughput, latency, CPU, memory, demand growth, units, and timestamps. An average without a window is not decision-grade. Interactive workloads need windows that include known bursts; monthly batch workloads cannot be judged safely from an eight-day view if their main peak lies outside it.

5. Saturation and headroom

High utilization is not identical to saturation. Saturation matters when a resource approaches its limit and service signals such as latency or throughput degrade with it. Headroom must be computed for the resource tied to the bottleneck; healthy CPU headroom does not cancel memory saturation.

6. Burst and seasonality

Burst profile records frequency, duration, and magnitude. Seasonality tests whether the current observation window covers the real workload cycle. Current Compute Engine rightsizing documentation uses eight days of history and warns that brief or infrequent spikes can be missed, making the observation window a real constraint.

7. Bottleneck identity

Before comparing upgrades, state what the bottleneck is and why. If peak memory is high, latency breaks in the same period, and CPU remains low, low CPU is not proof of safe downsizing; the evidence points more strongly to memory pressure. If the bottleneck cannot be supported, stop the recommendation.

8. Autoscaling and resource requests

For HPA resource-utilization targets, utilization is calculated against resource requests. If the relevant request is absent, utilization can be undefined for that metric. Therefore “HPA is enabled” is not complete capacity evidence; metric semantics and requests must be valid first.

9. Freshness and recommendation age

Each recommendation retains last_refresh_time or an equivalent timestamp beside the newest demand event. Google Cloud Recommender documents that recommendation content reflects resource state at the last refresh time. A material spike after that point makes the recommendation stale for this decision, even if it is still visible in a console.

10. Comparable scenario set

Compare at least NO_CHANGE, SCALE_OUT, and SCALE_UP against the same evidence package; add an architecture-change option only when needed. Each scenario records the change, expected capacity, expected cost, constraints, risk, and rollback. Scenarios computed from different demand or cost versions are NOT_COMPARABLE.

11. Cost, risk, and rollback

The cheapest scenario is not automatically the right one. It must address the bottleneck, deliver required capacity, respect constraints, and have a usable rollback condition. Cost evidence also ages; stale scenario pricing produces DEFERRED until the comparison is recomputed.

12. Decision states and authorization

RECOMMEND means evidence supports proposing a scenario; OBSERVE means no change is justified yet; INSUFFICIENT_DATA means decisive evidence is missing; STALE_EVIDENCE means critical evidence is old; NOT_COMPARABLE means scenarios lack a common basis; REJECTED means no candidate satisfies the requirement. None of these states silently grants execution authorization.

13. Worked baseline

The baseline averages 45% CPU, but memory peaks at 92% while latency breaches the SLO. Bursts are covered, the recommendation is fresh, and the memory bottleneck is supported. Among the comparable scenarios, SCALE_UP is the lowest-cost option that restores required headroom and has rollback evidence, so the result is RECOMMEND, not automatic execution.

14. Negative cases that block shortcuts

Four shortcuts must fail: a stale recommendation after a newer spike; HPA utilization without resource requests; changed scenarios without recomputation; and missing provenance. Each produces a bounded non-success state instead of coercing a dashboard or vendor label into a decision.

15. Transfer: monthly batch workload and deliverable

Transfer the advisor to a batch workload with one monthly peak. Because that peak lies outside an eight-day recommender window, the interactive-service conclusion cannot be copied. The dossier emits INSUFFICIENT_DATA, asks for COLLECT_BURST_EVIDENCE, and schedules a recheck after the monthly cycle is covered. The final artifact keeps evidence, scenario comparison, recommendation, and execution authorization separate.