Kubernetes and Kafka platform modernisation for critical infrastructure
Client context
An Australian energy utility (gas and electricity distribution) running a Confluent Kafka data-streaming platform that serves grid operations, including voltage management and distribution management workloads written as Java Kafka Streams applications.
The problem
The Confluent platform ran on legacy EC2 infrastructure. The utility needed to move it onto Kubernetes and, at the same time, stand up a reusable enterprise Kubernetes platform: standardised cluster provisioning, a governed add-on stack, single sign-on against the corporate identity provider, and security controls fit for a critical-infrastructure operator.
What was built
- A Terraform EKS blueprint built on upstream AWS modules plus GitOps Bridge, bootstrapping Argo CD into every new cluster. Later productised into a governed internal Terraform module registry entry that standardises and enforces common configuration across all clusters, supporting both EKS and Rancher RKE.
- A GitOps add-on layer: Argo CD ApplicationSets delivering cert-manager, external-dns, external-secrets, AWS Load Balancer Controller, Velero backup, Kyverno policy enforcement, OpenTelemetry, and endpoint security tooling to every cluster.
- Confluent Platform on Kubernetes via the Confluent for Kubernetes operator (Kafka, Schema Registry, ksqlDB, Connect), managed with Kustomize and ApplicationSets across sandbox, dev, QA, and prod, with Entra SSO and OAuth integrated across both Kubernetes and Confluent.
- Grid-domain stream processors: Java 17 and Spring Boot Kafka Streams services with Avro schemas, with modernised CI and deployment (GitLab OIDC, non-root containers, a reusable Argo CD deploy step).
- A DevSecOps supply-chain gate: Helm chart and container image scanning plus Kyverno admission control enforcing that only approved-registry images run.
- An airgapped API gateway for the operational technology DMZ, packaging Kong, k3s, and Argo CD into offline bundles.
Outcomes
- Provisioning a new cluster with the blueprint takes about 15 minutes. The same build previously took weeks of hand assembly.
- The pattern now underpins clusters for the integration, security, and platform teams across four environments (dev, SIT, staging, production) in two tenancy locations: multi-AZ AWS across three availability zones in Sydney, and a bare metal, airgapped on premises deployment.
- The legacy integration processor moved from a slow, error prone, ceremony heavy release process to a pipeline that is fully auditable through git. The manual release cutting step was eliminated and the failure rate dropped.