Kubernetes and Kafka: Problem Fit and Not-Now Decision

Two famous technologies, two different problems, no mandatory adoption

A product runs three containerized services on one managed host, deploys a few times per week, and emits a modest stream of domain events. Two consumers receive those events through a transactional outbox and managed queue. The current service objective is met, there is no replay-driven analytics or regulatory replay requirement, and one application team owns the system. A proposal says the next release “needs Kubernetes and Kafka to scale.”

That sentence hides two decisions. Kubernetes addresses management of containerized workloads and services. Kafka addresses publishing, retaining, and processing streams of events. Both have substantial capabilities, but a capability fact is not evidence that this product has the matching problem today. [K8SOV] [KAFKAINTRO]

Your deliverable is Kubernetes/Kafka Use-Not-Now Decision Matrix v1. It contains one independent row per technology. Each row records the problem class, current symptom, measured trigger, fixture-specific threshold, availability or event requirements, simpler alternative, operating owner, skill readiness, failure surface, migration cost, rollback path, evidence trace, decision, review date, and stop or pivot condition.

The allowed outcomes include USE, PILOT, NOT_NOW, UNKNOWN, EVIDENCE_INCOMPLETE, OWNER_MISSING, REVIEW_REQUIRED, HOLD, PARTIAL, and DISPUTED. NOT_NOW is not a polite guess. It is complete only when decisive evidence is present, the adoption trigger is not met, and a simpler alternative meets the current need. Missing evidence requires a typed incomplete state.

Plan about 45 active minutes. You will separate the problem classes, inspect the baseline evidence, reject prestige-driven shortcuts, run a deterministic offline lab, and recompute both rows for a system with many workloads, multiple teams, and a hard replay requirement.

This lesson is for engineers, technical leads, architects, and product owners who must defend a bounded decision about Kubernetes or Kafka—including a defensible decision not to adopt yet.

This is not a Kubernetes or Kafka installation, configuration, migration, capacity-planning, benchmarking, or production approval guide. It does not deploy a cluster or broker, choose a vendor, or absorb the adjacent Redis/Valkey, Temporal, search, API, or microservices lessons.

Prerequisite probe: can you distinguish an observed fact, an inference from that fact, and a design decision that requires local authority? Can you attach a claim to visible evidence without turning a tool description into permission to change architecture?

If that boundary is fragile, use this bounded bridge: write FACT beside what the fixture or primary documentation directly supports; write INFERENCE beside a conclusion drawn from several facts; write DESIGN beside a threshold, owner assignment, adoption gate, or rollback rule chosen for this product. A design decision must name its authority and cannot borrow authority from vendor documentation. This bridge supplies only the evidence habit needed here; it does not claim completion of FND-003.

Separate the problems before naming a technology

Answer these eight checks before taking the fast path:

1. Can you state the Kubernetes problem as a workload-management gap rather than “we use containers”?
2. Can you state the Kafka problem as a durable event-streaming gap rather than “we have events”?
3. Can you provide a measured current condition and a fixture-specific threshold for each row?
4. Can you prove the simpler alternative misses the same declared requirement?
5. Can you distinguish Kafka ordering inside one topic-partition from a global order across every event?
6. Can you name the team that owns on-call response, upgrades, capacity, recovery, and schema or policy changes?
7. Can you show a bounded rollback path rather than saying migration is reversible?
8. Can you preserve an incomplete state when trigger, owner, or recovery evidence is missing?

Eight of eight supports the experienced route. Four to seven means use the repair prompts. Zero to three means keep the field dictionary visible through the lab. Every route retains the same evidence bar, negative cases, transfer task, and source boundary.

Self-check: replace “enterprise scale,” “cloud native,” “future-proof,” and “event driven” with an observable condition. If the sentence loses its meaning, it was a slogan rather than evidence.

Mental model: capability, trigger, and decision are distinct

Use two independent chains:

KUBERNETES → container-workload management problem → current placement/release/availability evidence → declared trigger → managed-host or scheduler alternative → owner + skills + control-plane failure surface → migration + rollback → bounded state

KAFKA → durable event-streaming problem → event rate + retention + replay + ordering + consumer evidence → declared trigger → outbox/queue/direct-integration alternative → owner + skills + broker/partition failure surface → migration + rollback → bounded state

The Kubernetes overview describes a portable, extensible, open-source platform for managing containerized workloads and services with declarative configuration and automation. It describes capabilities such as desired-state management, scaling, failover, service discovery, and load balancing. Those are platform facts. They do not prove that three services on one managed host need a cluster. [K8SOV]

The Apache Kafka introduction describes an event-streaming platform that publishes and subscribes to streams, stores them durably, and processes them. Events are organized in topics; topics are partitioned for distributed scale; consumers read events in order within a given topic-partition. Those facts do not prove that every application event, queue, or background job needs Kafka. [KAFKAINTRO]

The ordered text above is the complete fallback for asset://AWR-001/AWR-001-ASSET-FLOW. If the diagram is unavailable, the decision dependency remains intact.

Evidence contract for two independent decisions

Kubernetes row

Record the containerized workloads, teams, deployment frequency, release coupling, placement constraints, scaling pattern, availability objective, actual incidents, and current host or scheduler behavior. A positive state must identify a workload-management capability that closes a measured gap—not merely count containers. It must also show why the managed host, deployment automation, or a smaller scheduler cannot meet the same need.

Make the operating burden visible: control-plane and worker lifecycle, networking and policy, identity, secrets, storage integration, observability, upgrades, compatibility, capacity, incident response, and who is on call. This lesson does not claim every Kubernetes deployment uses the same topology or burden. It requires the proposed topology’s obligations to be evidence rather than an omitted footnote.

Kafka row

Record current and peak event rate, producer count, independent consumer groups, retention duration, replay need, ordering scope, delivery and availability objective, schema-evolution boundary, consumer-lag tolerance, and what the current outbox, queue, or direct integration fails to provide. Event volume alone is not enough; a modest volume can still have a hard replay need, while a high volume may be handled by another bounded service.

Ordering must name its scope. Kafka preserves order as written and read within a topic-partition; it does not silently give one global order across unrelated partitions. [KAFKAINTRO] A positive state must bind the partition/key strategy to the required order and expose retention, replay, consumer, and failure assumptions.

Shared fields

Both rows require a technology-specific measured trigger, fixture-specific threshold, simpler alternative and its measured result, operational owner, skill readiness, failure surface, migration cost, rollback path, evidence reference, FACT/DESIGN authority trace, learner rationale, review date, and stop or pivot trigger. The rows may reach different states. A bundled “Kubernetes plus Kafka” verdict is invalid.

Building Kubernetes/Kafka Use-Not-Now Decision Matrix v1

Step 1 — Split the proposal. Rewrite “we need Kubernetes and Kafka” as two falsifiable problem statements. For Kubernetes: what workload placement, rollout, scaling, isolation, or recovery outcome is missed? For Kafka: what publish, retention, replay, ordering, processing, or independent-consumer outcome is missed?

Step 2 — Establish the current path. Record what the managed host, CI/CD process, database outbox, managed queue, or direct integration already delivers. Include measured behavior and declared requirements. A comparison without a current baseline cannot establish improvement.

Step 3 — Define a local trigger. A trigger combines an observable condition and a threshold chosen for this fixture. “Twenty workloads and three teams,” for example, is synthetic exercise data—not a universal Kubernetes rule. “Four consumer groups or regulatory replay” is also a local design trigger. Tag both as DESIGN, state who owns them, and retain the evidence that will activate review.

Step 4 — Test capability fit. Map only the capability that answers the problem. Kubernetes desired-state controllers may fit workload-management evidence; they do not replace application correctness, CI/CD, observability, or data systems. Kafka retention and partitioned consumption may fit replay and independent-consumer evidence; they do not turn every CRUD notification into a streaming-platform need. [K8SOV] [KAFKAINTRO]

Step 5 — Challenge the simpler alternative. For Kubernetes, test managed container hosting, stronger deployment automation, autoscaling offered by the current platform, and explicit runbooks. For Kafka, test a transactional outbox with a managed queue, database change processing with a bounded contract, or direct integration. Compare against the same availability, replay, ordering, and recovery requirement.

Step 6 — Price ownership and failure. Name the real owner and skill gap. Describe new failure surfaces: cluster upgrades, networking and policy, scheduling and storage for Kubernetes; broker availability, partitions, rebalancing, retention, schema evolution, lag, and poison events for Kafka. Avoid invented dollar figures. A qualitative cost can still be auditable when its activities and staffing assumptions are explicit.

Step 7 — Design migration and rollback together. A bounded Kubernetes pilot preserves deployable artifacts for the current host and defines traffic cutback. A bounded Kafka pilot can dual-publish only when duplicate effects, ordering, and reconciliation are controlled; it defines how to return consumers to the outbox/queue path. “We can always roll back” without state reconciliation is not a rollback plan.

Step 8 — Choose the weakest supported state. PILOT is appropriate only for a complete, bounded experiment with success, stop, owner, and rollback evidence. USE requires validated adoption approval beyond these synthetic fixtures. NOT_NOW requires trigger absence plus a sufficient alternative. Unknown, conflicting, stale, or ownerless evidence remains typed.

Step 9 — Bind reconsideration. Every row carries a review date and measurable reopen condition. The decision can change when the evidence changes; it should not drift because a platform becomes fashionable.

Worked example: one system, two evidence-backed NOT_NOW decisions

All numbers and thresholds below are synthetic fixture data. They teach decision structure, not industry sizing rules.

Kubernetes. The product has three independently packaged services, one team, and about three deployments per week. The managed container host meets the declared 99.5% service objective, and the evidence contains no incident attributable to placement, scheduler recovery, or release coordination. The fixture’s review trigger is at least twenty independent workloads with three teams, or an earlier measured placement or availability gap. The application team can operate the current host; it has no Kubernetes on-call rotation or cluster-upgrade runbook. The result is NOT_NOW: evidence is complete enough to show that the measured orchestration trigger is absent and the current alternative meets the requirement.

Kafka. The system emits 40 events per second with a peak of 120, has two consumers, needs only 24 hours for operational retry, and has no regulatory or analytical replay requirement. Ordering is required only per business aggregate. A transactional outbox plus managed queue meets the declared delivery objective. There is no broker and partition operations owner or runbook. The result is also NOT_NOW: the durable streaming trigger is absent and the simpler alternative is sufficient.

Now increase the service count to eight while keeping one team, low deployment frequency, and no placement or availability pressure. That mutation does not cross the declared Kubernetes trigger. More containers may increase attention, but it does not by itself prove a cluster problem.

Next add a hard seven-year replay/audit requirement and six independent consumer groups while keeping event volume moderate. Volume did not create the trigger; retention, replay, and consumers did. Kafka must be recomputed. If owner, skill, schema, reconciliation, migration, and rollback evidence are complete, the row may become PILOT. If the owner is absent, the correct state is OWNER_MISSING, not a positive verdict.

The structured example is asset://AWR-001/AWR-001-ASSET-TABLE; its JSON and locale fallbacks retain the decisive fields.

Common failure modes

1. Prestige becomes a requirement. “Serious systems use Kubernetes and Kafka” replaces evidence. Repair: name one missed local outcome and its measurement.
2. The two decisions are coupled. A Kubernetes trigger automatically turns Kafka positive, or vice versa. Repair: recompute independent rows.
3. Containers imply Kubernetes. Packaging is mistaken for an orchestration gap. Repair: inspect workload count, teams, releases, placement, availability, and current host behavior.
4. Events imply Kafka. Domain notifications are mistaken for durable streaming. Repair: inspect rate, retention, replay, ordering, consumer independence, and current queue limits.
5. Ordering is overstated. Partition ordering becomes a claim of global order. Repair: state the topic, key, partition, and consumer boundary. [KAFKAINTRO]
6. Thresholds masquerade as vendor facts. Synthetic local triggers are presented as official guidance. Repair: label them DESIGN and preserve local authority.
7. A feature list replaces comparison. The current path is never measured. Repair: test the simpler alternative against the same requirement.
8. Operations disappear. On-call, upgrades, capacity, recovery, schemas, policies, and observability have no owner. Repair: use OWNER_MISSING until a real team accepts the obligation.
9. PILOT means quiet adoption. No stop criteria or rollback exists. Repair: bound traffic, duration, evidence, reconciliation, and cutback.
10. Unknown becomes NOT_NOW. Missing evidence is treated as trigger absence. Repair: use UNKNOWN, EVIDENCE_INCOMPLETE, PARTIAL, or another supported state.
11. Baseline becomes an oracle. The transfer system copies both NOT_NOW labels. Repair: hide the labels and recompute from changed evidence.
12. Capability becomes authority. Official documentation is treated as product approval. Repair: separate external FACT from MAKNOW DESIGN and human adoption authority.

Experienced bridge: do not collapse platform capability into product need

If you have operated Kubernetes, compress the introduction but retain the evidence: desired workload state, scheduling or placement need, rollout and scaling pattern, availability gap, current-platform comparison, cluster lifecycle, storage/network/policy boundary, owner, and rollback. Familiarity may reduce training cost; it does not create the product trigger.

If you have operated Kafka, retain the event contract: producer and consumer identities, rate and burst, retention, replay, ordering scope, partition/key assumptions, delivery semantics, schema ownership, lag tolerance, current queue comparison, owner, migration, reconciliation, and rollback. Prior deployment experience does not make Kafka the default transport.

Use a reversal test for each row: what precise observation would move NOT_NOW to PILOT, and what observation would stop the pilot? If the answer is “growth” or “complexity,” make it measurable. If the answer is “we already run it elsewhere,” include the shared platform’s actual service boundary, capacity, support agreement, and incremental cost rather than assuming them.

Problem fit still does not select a distribution, cloud vendor, managed service, topology, operator, partition count, storage class, security architecture, or rollout plan. Those are subsequent decisions with separate evidence and authority.

Measure the simpler alternative before adding a platform

Managed container host versus Kubernetes

A managed container host with CI/CD, health checks, platform autoscaling, and explicit runbooks may meet the needs of a small product with one team. Kubernetes can provide a consistent declarative control plane across containerized workloads, but a proposed cluster also introduces lifecycle, networking, policy, storage, observability, capacity, compatibility, and incident obligations. [K8SOV] The decision is not “simple versus professional.” It is whether the added control plane closes a measured orchestration gap at an acceptable ownership cost.

Transactional outbox and managed queue versus Kafka

An outbox can bind database changes to event publication, while a managed queue can provide bounded delivery and retry for a small number of consumers. Kafka adds durable event streams, repeated reading, partitioned scale, and stream-processing options. [KAFKAINTRO] It also requires an explicit event model, retention and schema governance, partition/key decisions, lag monitoring, recovery, consumer behavior, and operational ownership. Choose it because those semantics close evidenced gaps—not because the word “event” appears in the architecture.

Managed service versus self-managed operation

A managed offering may reduce some infrastructure work, but it does not remove application-level partitioning, schemas, permissions, observability, cost, incident coordination, or migration risk. Likewise, an existing shared platform may lower incremental burden only when capacity, tenancy, service objectives, support ownership, and chargeback are visible. Treat “managed” and “already available” as evidence to inspect, not automatic approval.

The smaller path is sufficient only when it meets the same declared requirement. Rejecting a tool without measuring its alternative is as weak as adopting it without a trigger.

Guided and independent practice

Use the offline lab at lab://AWR-001/AWR-001-LAB1. It needs a standard Python interpreter. All records are synthetic and read-only; there are no cloud accounts, credentials, customer records, network calls, clusters, brokers, or infrastructure mutations. The starter exposes every field and checking rule but leaves decisions, rationales, review dates, and stop conditions blank.

Guided pass:

1. Select AWR001-BASELINE-001.
2. Create exactly two records: KUBERNETES and KAFKA.
3. Copy the selected fixture’s evidence objects exactly; do not improve missing fields.
4. Decide each row independently using the weakest state supported by evidence.
5. Write your own rationale, review date, and stop or pivot trigger.
6. Run python run_checks.py evidence/decision_matrix.template.json.

Repair pass:

• Run the service-count mutation and explain why count alone does not activate the Kubernetes trigger.
• Run the replay mutation and recompute Kafka from retention, replay, ordering, and consumer evidence.
• Run the owner-missing mutation and preserve OWNER_MISSING for the affected positive path.
• Run the missing-evidence fixture and preserve typed incomplete states.

Independent pass:

Use transfer/transfer_fixture.json, copy it to the transfer submission template, remove any remembered baseline labels, and recompute both rows. A passing checker proves rule consistency and exact evidence binding; your rationale is still reviewed for problem fit, authority boundary, trade-offs, and transfer.

What makes the evidence package acceptable?

A reviewer must be able to reproduce both decisions without hidden model reasoning. The package therefore includes:

• the exact fixture identity and unmodified evidence object for each technology;
• explicit problem class, current symptom, measurement, and local threshold;
• availability plus event rate, retention, replay, ordering, and consumer fields where applicable;
• a measured simpler alternative, not merely its name;
• operational owner and skill readiness;
• failure surface, migration cost, rollback path, and review trigger;
• source references that distinguish official capability facts from MAKNOW design rules;
• a state no stronger than the evidence;
• one negative mutation and one changed-context transfer;
• deterministic checker output plus human-reviewable rationale.

AI may organize supplied evidence and run deterministic comparisons. It may not invent a replay requirement, scale pressure, owner, skill readiness, adoption approval, or migration success. A green checker is necessary but not sufficient: it detects structural inconsistency and unsupported state transitions, while a human remains accountable for the bounded architecture decision.

Assessment and transfer

AWR-001-F1: state the owned boundary and one adjacent non-goal. Your answer must separate workload orchestration from durable event streaming.

AWR-001-F2: diagnose the mutation that raises service count while leaving one team, low release frequency, and no availability pressure. Name the field that remains below the declared trigger and the evidence needed to reopen it.

AWR-001-F3: recompute after adding hard replay and several independent consumers. Do not copy the baseline Kafka state.

AWR-001-S1: explain the lesson boundary in your own words and identify one task this lesson does not perform.

AWR-001-S2: build Kubernetes/Kafka Use-Not-Now Decision Matrix v1 for the baseline synthetic fixture.

AWR-001-S3: diagnose the service-count shortcut. “Eight services need Kubernetes” is insufficient; relate workload, team, release, placement, availability, current alternative, ownership, and rollback evidence.

AWR-001-S4: a second system has dozens of independently deployed workloads, multiple teams, frequent releases, bursty traffic, several event consumers, and a regulatory replay/audit requirement. Recompute both rows independently. Complete evidence may support bounded pilots; it does not automatically authorize USE.

Transfer passes only when each new state traces to changed evidence and preserves the distinction between official capability facts and local adoption design.

Sources and freshness boundary

K8SOV — the official Kubernetes concepts overview supports the bounded facts that Kubernetes is a portable, extensible, open-source platform for managing containerized workloads and services using declarative configuration and automation, with workload-management capabilities described there. It does not prove that this product needs Kubernetes. [K8SOV]

KAFKAINTRO — the official Apache Kafka introduction supports the bounded facts that Kafka publishes, durably stores, and processes event streams, and that topics are partitioned with order guaranteed within a topic-partition. It does not prove that this product needs Kafka or that every event needs a streaming platform. [KAFKAINTRO]

The primary pages were rechecked on 2026-09-12; no material change required expanding the locked claim boundaries. The next scheduled review remains 2026-12-02, or earlier if the cited Kubernetes or Kafka semantics materially change. Local thresholds, state labels, owner requirements, migration gates, and adoption authority remain MAKNOW design—not claims attributed to either project.