SAS Viya 3.5 platform with an N-node database cluster and Kafka ingestion
A multi-machine SAS Viya 3.5 deployment with no single point of failure: one writable PostgreSQL primary with N hot standbys behind pgpool-II, an odd-N Consul quorum, and Kafka topics loaded into CAS through SAS Event Stream Processing, so any single tier can fail without losing records.
1 writer, N standbys- Role
- Infrastructure engineer
- Scale
- N hosts, N-node data cluster, N partitions per topic
- Stack
- SAS Viya 3.5, Ansible, Viya-ARK
- Targets
- No single point of failure, replayable ingestion
Context
SAS Viya 3.5 runs as a set of N hosts across several tiers: coordination, data, microservices, HTTP proxies, CAS compute, and an ingestion tier fed from Kafka. The goal is a layout where any single host or tier can fail and the platform recovers without data loss and without anyone choosing between two writers.
Constraints:
- One logical database shared by all microservices
- Consul is required before any service starts
- Records arrive continuously and must not be lost or applied twice
- On-prem only; no managed services
Architecture and failover
Users reach the platform through a load balancer and N HTTP proxies, which forward to N microservice hosts. Every microservice stores its state in one logical database through the data tier below.
The data tier is the platform's SAS Infrastructure Data Server. Node 0 is the single writable primary. Nodes 1 to N are hot standbys on streaming replication. N pgpool-II instances sit in front, with a watchdog virtual IP held by one of them. Microservices connect only to the virtual IP, never to a node.
Consul runs as an odd N servers on the microservice hosts, with a client agent on every other host. Vault runs beside them and must be registered before pgpool starts. RabbitMQ runs as an N-node cluster for service messaging.
Analytics compute is a CAS grid: one controller and N workers. Streaming data arrives from Kafka topics with N partitions each. SAS Event Stream Processing reads the topics and loads the CAS grid, so the stream does not touch the database path.
The failover scenarios below use illustrative timings. Counts are shown as N throughout.
- Primary / serving
- Recovering
- Degraded
- Down
- Replication / control
Pick a failure to see how the system detects and recovers from it.
Key decisions
One primary and N hot standbys behind pgpool-II
All platform microservices share one logical database, so the data tier needs automatic failover without ever allowing two writers.
Trade-off: Write capacity is limited to a single primary; standbys add availability and read capacity, not write scale.
Odd-N Consul servers on separate failure domains
Consul is a hard dependency of service start and of the config rendering for pgpool and PostgreSQL, so its quorum must survive host loss.
Trade-off: More hosts reserved for coordination, and server restarts must be rolled one at a time.
Kafka topics as the ingestion buffer
Source systems publish continuously, and the platform must absorb restarts, failovers and patch windows without dropping or double-counting records.
Trade-off: Retention has to cover the longest recovery window, and the load into CAS must tolerate a replayed batch.
Ordered, gated deployment from one controller
The platform has strict start dependencies, and one unhealthy member can block every later step.
Trade-off: A single controller host is a dependency for changes (not for runtime) and must be backed up.
Lifecycle
- Define the inventory groups for each tier and the per-host variables for the data nodes and pgpool instances.
- Run the playbooks from the single controller. Start Consul servers and Vault first, then Consul agents on every other host.
- Start the data tier in order: consul-template services, node 0 as primary, standbys gated on a standby configuration file, then pgpool.
- Start RabbitMQ, the HTTP proxies, CAS and the remaining microservices, then the Event Stream Processing project.
- Verify before declaring done: every Consul member alive, one leader, N backends up in pgpool, replication streaming at equal positions.
- Patch and reboot standbys one at a time, waiting for replication lag to reach zero before the next.
- Switch over the primary to a patched standby, then patch the old primary, which rejoins as a standby.
- Roll Consul servers one at a time, confirming the same leader and N peers before the next.
- Pause the Event Stream Processing project during the switchover window. Kafka keeps the records, so nothing is lost.
- Source systems publish to Kafka topics; the topics buffer the stream.
- The Event Stream Processing project reads the topics and loads the CAS grid.
- After any outage, the project resumes from its last committed position.
- For a bad load, rewind to an earlier offset inside the retention window and replay.
Failure modes
| Failure | Blast radius | Detection | Recovery |
|---|---|---|---|
| Primary data node lost | Writes pause briefly; service calls retry | pgpool health check | Automatic promotion of a standby; clients reconnect through the virtual IP |
| Standby lost | Reduced read capacity and redundancy | Replication and pgpool state | Rebuild or restart as standby |
| One Consul server lost | None while a quorum remains | Member and leader checks | Restart the server; quorum returns to N |
| Consul quorum lost | No service start, no config rendering | Leader check on servers | Restore a majority of servers |
| pgpool instance lost | Virtual IP moves | Watchdog | Automatic; no client reconfiguration |
| CAS worker lost | Reduced capacity; running actions may fail | Controller worker state | Redundant table copies rebalance; rerun failed actions, replace the worker |
| Kafka broker lost | Partition leadership moves | Broker and under-replicated partition alerts | Automatic leader election; replace the broker |
| Controller host lost | No changes possible, runtime unaffected | Backup and host monitoring | Restore the controller from backup |
Results
What I'd change
I would pair every service-wrapper status with a real health check: Consul membership queried on a server, pgpool backend state, and replication position. A wrapper that only inspects a process or PID file can report a healthy database as stopped.
I would put the Consul servers and the data nodes on separate failure domains from the start, and add the VIP address to the database certificate names so clients that connect by address verify cleanly.
I would also size Kafka retention against the longest recovery window and make every load into CAS safe to replay from day one.