SUPFile — Designing a Resilient Multi-Datacenter Architecture

Goal

An architecture project carried out by a team of four around a file storage platform spread over several datacenters. The work focused mainly on design, sizing, technology choices, resilience, backups, costs and the operations plan. A separate proof of concept then validated part of the technical feasibility.

This page therefore describes a target architecture first: it was not deployed as a whole.

My contribution

My role in the team was data, storage and backup architect. Beyond that scope, I took on a large share of the overall design and documentation: sizing and the planned hardware, the budget, technology choices and reliability. Architecture decisions and diagrams were worked through continuously with the project’s lead architect.

Designing a multi-datacenter architecture

The target architecture spreads the platform over three sites. Paris and New York are active and both serve users, with geographic traffic distribution (GSLB / GeoDNS). Toronto is not exposed: it is reserved for backup, retention and disaster recovery.

One structural decision: do not stretch a cluster across continents. Each active site has its own Kubernetes cluster and its own storage. Intercontinental latency would make a stretched cluster fragile, and independent sites limit the failure domain: losing one site does not take the other down. Data moves between sites through asynchronous replication, over encrypted and filtered links.

Data, storage and resilience

PostgreSQL is designed to be highly available within each site, using Patroni. So that both active sites can write without conflicts, data is partitioned: each user has a home site, the only one allowed to write that user’s data, which is then replicated to the other site.

Storage separates two needs. User files go to CephFS, a distributed file system local to each site. Critical block volumes — PostgreSQL’s in particular — sit on a dedicated iSCSI SAN. Each type of data gets storage suited to how it is used, and a failure in one does not affect the other.

Backup and disaster recovery

Replication is not a backup: a deletion or corruption replicates just as fast as healthy data. The backup strategy is therefore designed separately. Toronto receives daily backups of PostgreSQL, files and critical volumes. For PostgreSQL, continuous archiving of the write-ahead log (WAL) allows point-in-time recovery (PITR). Regular local snapshots speed up everyday restores, and long-term archiving on LTO-9 tapes completes retention. The recovery plan relies on a restore area in Toronto and on documented restore procedures.

Sizing and budget

The project was not just a diagram: each site was sized, the planned hardware detailed, and costs estimated as capital (CAPEX) and operating (OPEX) expenses, with their assumptions. The budget was one of the architecture’s constraints, just like resilience or security.

Validation with a limited proof of concept

To test some of the assumptions against a real implementation, the team built a limited proof of concept, led mainly by another team member. It reproduces the three sites with three Debian virtual machines, each running a single-node k3s cluster. PostgreSQL is managed there by the CloudNativePG operator: a primary instance in Paris and read-only replicas in New York and Toronto, with asynchronous replication. A demo application runs on both active sites, behind an Nginx load balancer.

What the proof of concept validates — and what it does not

PostgreSQL replication across sites was implemented and tested: a row inserted in Paris is read in New York and in Toronto. Load balancing between the two active sites was also demonstrated during the final presentation.

The proof of concept does not validate the rest of the target architecture: not CephFS, the iSCSI SAN, Patroni or partitioned writes across two sites, Kubernetes high availability, geographic traffic distribution, encryption between datacenters or the planned security controls, nor backup to Toronto, PITR, tapes or the recovery plan. Its replication even runs unencrypted over a lab network: it is no evidence of the target’s security.