← Back to projects

Bethesda Kubernetes Platform Engineering

Database Migration & Platform Automation at AAA Scale

Platform & Database EngineerApr 2026 - PresentBethesda Softworks (via Jahnel Group)

Overview

Platform and database engineering on the Bethesda Kubernetes Platform (BKP) — the self-managed Kubernetes foundation behind Bethesda's account and services systems. The work spans three tracks: migrating a production MySQL database off AWS RDS onto a self-managed Percona XtraDB Cluster running inside Kubernetes, extending the platform's Terraform modules to support ProxySQL and multi-namespace operators, and optimizing business-intelligence query performance on the PostgreSQL side.

The Challenge: Self-Managed Databases on Kubernetes

Moving a live, high-traffic database off a managed service and onto Kubernetes trades operational convenience for control and cost — and every part of that trade has to be earned:

  • Zero-tolerance cutover: An account-platform database can't lose data or take extended downtime
  • Resilience during pod cycling: The cluster must survive node and pod churn that RDS abstracts away entirely
  • Reproducibility: Everything — compute, storage, replication, routing — has to be codified so the migration can be rehearsed and repeated
  • Cost discipline: The whole point is reducing hosting spend versus RDS while standardizing on the platform's self-managed PXC pattern
  • Heterogeneous hardware: Automation has to work across ARM64 and x86_64 instances without hand-tuning

Track 1 — MySQL Migration: RDS → Percona XtraDB Cluster on Kubernetes

Infrastructure as Code

Context: A production MySQL database running on AWS RDS, targeted for migration to a self-managed Percona XtraDB Cluster (PXC) inside the Bethesda Kubernetes Platform.

Approach:

  • Authored the Terraform for the migration seed environment: EC2, a 700 GB EBS data volume, S3 migration buckets, and IAM roles
  • Built an interactive prep-env Make target that auto-discovers the RDS source CIDR, validates the AWS profile, and bootstraps the Terraform state backend
  • Added a target to auto-update the PXC replication source host after EC2 replacement, so a re-provisioned seed node reconnects without manual edits
  • Handled Terraform backend configuration drift on re-init to keep state reproducible

Ansible Migration Automation

Technical Decisions:

  • Architecture detection: Playbooks auto-select Percona / mydumper / myloader packages for ARM64 vs x86_64 instead of hardcoding a platform
  • Storage: Dedicated MySQL tmpdir on an XFS volume, with an async volume resize to avoid SSM channel timeouts during long operations
  • InnoDB tuning: Derived from the actual EC2 instance resources — scaling innodb_redo_log_capacity up to 16 GB / 32 GB for the bulk seed and index-rebuild phases
  • Load throughput: myloader Make targets that thread-limit based on available CPU headroom to keep the seed fast without starving the box
  • Replaced brittle bash socket-detection conditionals with Jinja2-based logic for reliable rendering

Deployment & Routing

  • ProxySQL and HAProxy configurations for the primary and BI read paths
  • Pinned Percona images to specific 8.0.x patch levels for reproducible, drift-free deployments
  • Hardened the Kubernetes manifests: disabled service-mesh export labels, removed namespace auto-creation, and added External Secrets Operator annotations with refresh intervals
  • Wrote the deployment README, environment-configuration guide, and post-cutover backup-flow documentation so the platform team can operate it without me

Track 2 — ProxySQL & Percona Operator on the Platform

Extended the BKP Terraform platform itself so the MySQL work above is a repeatable platform capability, not a one-off.

  • New ProxySQL Terraform module: a reusable module for deploying ProxySQL alongside the existing Percona MySQL module
  • Multi-namespace operator support: added watchAllNamespaces to the Percona MySQL module so the operator can monitor all namespaces instead of a hardcoded one
  • Safer YAML rendering: refactored the watch-namespace configuration from unreliable string templating to yamlencode for correct multi-value output
  • Service-adoption hardening: guard logic to adopt orphaned Helm-managed Services before an operator upgrade, preventing resource conflicts

Track 3 — BI Query & PostgreSQL Performance Optimization

Context: BI teams reported slow queries against the PostgreSQL account database. Investigation surfaced missing indexes, an inefficient view definition, and no query-level observability.

  • Rewrote an age-group aggregation view to eliminate a full-table scan, and added the supporting index to back it
  • Enabled pg_stat_statements across integration and production via controlled change tickets, giving the team ongoing query-level visibility
  • Turned those statistics into concrete query rewrites and index recommendations for the BI team

Key Technical Decisions & Trade-offs

Self-Managed PXC vs Managed RDS

This is the inverse of the trade-off I usually recommend to clients — and it was the right call here. RDS gives you backups, patching, and HA for free; PXC on Kubernetes makes you build them. The justification was concrete: meaningful cost reduction at this scale, tighter resilience during pod cycling, and standardization on a self-managed pattern the platform already ran for other services. Managed services win when operational overhead outweighs cost; at a certain scale and with a platform team in place, that equation flips.

Rehearse Before You Cut Over

The migration was built to be rehearsed — a full production-scale run with no cutover — so tuning, timing, and failure modes are known before the real window. Codifying the seed environment in Terraform and Ansible is what makes that rehearsal cheap enough to actually do, rather than a plan on paper.

Impact

RDS → K8s
Production MySQL moved to self-managed Percona XtraDB Cluster on Kubernetes
700 GB
Data volume migrated via a fully codified, rehearsable seed pipeline
2 arch
ARM64 and x86_64 supported by the same automation, no hand-tuning
IaC
Compute, storage, replication, and routing all Terraform/Ansible-managed

Technical Stack

TerraformAnsiblePercona XtraDB ClusterProxySQLHAProxyMySQL 8.0PostgreSQLKubernetesHelmAWS (EC2, EBS, S3, RDS, SSM, IAM)External Secrets Operatormydumper / myloaderpg_stat_statementsMakePython

Lessons Learned

  • Derive tuning from the machine, not a template: InnoDB settings scaled from actual EC2 resources beat copied-over constants — the same playbook stays correct across instance sizes and phases.
  • Long operations need async-safe automation: resizing a volume synchronously over SSM will time the channel out. Fire-and-poll patterns keep long storage operations from breaking the run.
  • Render YAML, don't template strings: yamlencode for multi-value operator config removed a whole class of whitespace and quoting bugs that string templating kept reintroducing.
  • Observability before optimization: enabling pg_stat_statementsfirst turned "the BI queries feel slow" into a ranked, evidence-backed list of what to actually fix.
← Back to all projects