Intro

Containers have become the next big thing in infrastructure software. However, for you to take full advantage of containers you need to be conversant on how to turn them into production services. This is where Kubernetes shines — as an orchestrator of your containerized applications.

Kubernetes gives you the building block of a process that can be constantly up and be reliably migrated to another machine if one machine goes down. Or it can give you an invariant of the single leader out of several processes running at any time. Kubernetes is excellent at that.

The applications we deploy on Kubernetes need a persistent data store to go along with their stateless microservices.

Why Use PostgreSQL?

  1. It's the world's most advanced open source object-relational database developed by an excellent community.
  2. .It's a reliable database in that all data recorded by a committed transaction is stored in a nonvolatile area that is safe from operating system failure and hardware failure meaning that all committed transactions remain intact.
  3. It's scalable which simply means you can run it on bigger hardware and get better performance or scale out by running more replicas.
  4. .It's extensible because its operation is catalog-driven and you can define your own complex data types.

Why Use PostgreSQL?

  1. Elastically scale out memory, compute, and storage as you add nodes without downtime.
  2. Followers with an up-to-date read-only copy of the database that enables developers and data analysts  to run complex queries without placing any query load on the production database.
  3. PostgreSQL supports distributed transactions.

In this article, we are going to discuss some common challenges we’ve seen while running PostgreSQL on Kubernetes. We will also look at some tools that can help in managing stateful applications on Kubernetes.

The Setup

If you rely on PostgreSQL databases and you are looking for clustering solutions for HA, the basic solutions would be master-slave and master-master architectures. 



The servers work together to allow a secondary server to take over quickly if the primary server fails (high availability), or to allow several VMs to serve the same data (load balancing).

A PostgreSQL Master-Slave architecture enables us to  survive and recover if the node with a database master (primary) crashes. These standby databases will remain synchronized (or almost synchronized) with the master.

The hardest thing in running PostgreSQL or a similar database on Kubernetes is to realize that Kubernetes is not aware of the deployment details of PostgreSQL. A naive deployment could result in a complete data loss.

The Event

Data loss pattern seen with asynchronous replication

Here’s a typical scenario when that happens. You set up streaming replication and let’s say the first master is up. All writes go there, and they asynchronously replicate to the standby. Then all of a sudden the leader goes down but the asynchronous replication has a huge lag. If the naive failover leader election algorithm kicks in or an admin who isn’t aware of the state manually triggers failover, the secondary becomes the master. That becomes the source of truth. All the data committed during that period is lost because all the writes that were not replicated will disappear. At whatever time the admin recovers the first master, it’s no longer the master anymore and it will naturally sync the state from the second node which has now been elected as the master.

The Root Cause

This issue lies with the asynchronous replication sending operations to the followers. Those modifications could be changes to the state, writes, or creating new values. Whenever a chunk of this data is lost there should be a mechanism that tells the receiving node that its data is out of sync. PostgreSQL has a mechanism that tracks replication lag. However, there’s no authority that analyzes this data that is built into PostgreSQL yet. This feature will help whoever is doing leader election.

The Fix

The Citus Data team handles replication correctly in their integration of Kubernetes with PostgreSQL. They have developed the capability to track replication state and replication lag - they have it all built in. But, if you are DIY, building Kubernetes deployments with failover, you should keep this in mind.

Kalc approaches this problem by replicating Kubernetes' behavior in AI. We are training our Kubernetes AI with the most common failure scenarios. Through our revolutionary AI planning tool kubectl-val, we let developers test their entire Kubernetes clusters' configurations against these scenarios.
kubectl-val will alert you of this and other possible errors when running Kubernetes in production. kubectl-val comes as a simple kubectl plugin, so a working kubectl is a requirement if you want to access a real cluster. 

Conclusion

We’ve seen that Kubernetes improves application resiliency, scalability and infrastructure security—but despite that—managing a distributed database is hard, which is why a DIY approach is challenging.

With Kubernetes you have better building blocks. The complex part is to create a domain-specific system that takes those building blocks and properly configures them — knows when to elect a leader or not, when is it safe, when is it not. The building blocks are there but you can still misuse them and have data loss as a result. It’s probably even easier to do so because the entry level is now lower. Anyone can write a Kubernetes operator for PostgreSQL but by running the kubectl-val cluster validator against the implementations you can just spot how they’ll lose data.