Abstract
A poker table can appear simple until something goes wrong: a player’s turn expires while a server restarts, a command is delivered twice, two workers race to update the same hand, or a downstream service records the end of a hand before its beginning. In real-money gameplay, these are not merely infrastructure inconveniences. They can affect game progression, chip movement, player trust and operator support.
PokerInfra Enterprise is designed on the assumption that processes stop, messages repeat, and work may be handled by different servers at different times. Every table is stored as shared state rather than owned by a particular game server.Durable timers remain available after restarts; atomic leases coordinate timer claims; AMQP competing consumers distribute commands across interchangeable workers; compare-and-set writes reject stale state; command identifiers make redelivery safe; and per-table sequence numbers help downstream consumers apply events in order
This paper explains the operational failure each mechanism is intended to prevent, why simpler approaches can fail, and how the implementation works. It includes code excerpts, guarantees, limitations, test scenarios and the configuration values operators need to understand. The focus is the game-processing architecture and its reliability mechanisms, rather than every account, wallet, administration or client component of a complete poker business. The diagrams illustrate the timer recovery lifecycle and the client-to-server command path.
The design uses Redis for live game state and coordination, and RabbitMQ for messaging: MQTT faces player clients, while AMQP carries internal work. The system uses at-least-once delivery and makes handlers safe to repeat; it does not claim universal exactly-once execution.
1. The operational problem: keeping an authoritative game correct
A poker table is a state machine—and mistakes have consequences
A poker table is a small state machine shared by several people over unreliable networks. Building the rules for a hand is only part of the job. The platform must continue to apply those rules correctly when players disconnect, messages arrive late or twice, a process restarts in the middle of a command, or several servers are available to process work.
In a server-authoritative game, the client displays the table and asks to act. The server shuffles and deals, determines whose turn it is, runs the clocks, validates bets, and moves chips. A client request is not trusted until it has been checked against the current table state. There is no client-side truth to fall back on if the server applies a bet to stale state, loses a turn timeout, or processes the same action twice.
Three failure conditions to design for
- Processes stop. Deployments, autoscaling, crashes and hardware failures can interrupt a server in the middle of a hand. In a multi-instance system, some process will always be starting or stopping
- Messages repeat or finish out of order. Brokers may redeliver a message after a consumer fails. With several consumers, two messages concerning the same table can finish in a different order from the order in which they were produced.
- Timers can be missed. A timer stored only in process memory disappears with that process. A one-off notification can be lost if no subscriber is listening when it is emitted.
Horizontal scaling makes these races more visible: two workers may try to fold the same player, a stale write may overwrite a newer table, or a consumer may record a hand’s end before its start.
The design stance
- State is the source of truth. Tables and timers are records that survive individual process lifetimes; messages request work or communicate results
- Every step must be safe to repeat. Delivery is at-least-once, so commands, timers and event handlers must tolerate duplicate execution.
- Every table write must prove it is current. A change is accepted only when it is based on the latest version of the table.
- No game server owns a table. Any eligible instance can process the next command for any table.
2. Architecture at a glance
PokerInfra Enterprise is built from stateless services around two shared systems: Redis for live game state and coordination, and RabbitMQ for messaging. Services can run as multiple instances; a game server does not keep authoritative table state between commands.
| Component | Responsibility | State retained between requests |
|---|---|---|
| Player apps | Display tables and send actions over MQTT. | No authoritative game state. |
| RabbitMQ | MQTT for players and AMQP for internal services on one broker. | Durable queues of pending work. |
| Lobby Service | Lobby service Lobby and room queries, session and maintenance checks, labels each command with a stable ID, forwards table commands to game servers, runs tournament publish/cancel, awards and refunds, and runs scheduled background jobs. | No game state. Any number of identical copies consume from shared queues |
| Game servers | Run the poker Hand Evaluator and consume commands from a queue per game format. | No table state; load, modify and save state for each command. |
| Redis | Live table state and version, timer queue, locks, dedupe keys and event counters. | Authoritative live game state and coordination records. |
| Heartbeat service | Player presence and disconnect grace periods, driven by durable timers. | No persistent in-process authority. |
| Extensions | Hand history, real-time results, chat and operator-added services. | Their own projections and queues. |
| API gateway, auth and database services | Accounts, wallets, ledgers and administration, backed by PostgreSQL. | Outside the scope of this paper. |
A player action arrives over MQTT, passes through the lobby service, enters a durable internal queue, and is processed by whichever eligible game server consumes it. That server reads the table from Redis, applies the Hand Evaluator, commits the new state, and publishes the result over MQTT. Domain events flow separately to extensions
Players speak MQTT; each command reaches exactly one game server
Two architectural properties drive the mechanisms described below. First, multiple instances share the same Redis and broker, so timers, locks and writes can race. Second, no game server owns a table, so a command can be processed by a different instance from the previous command. This enables scale-out and recovery without table reassignment.
3. Durable Timers - Preventing stuck tables due to missed timeouts
The operational problem
A poker timer is a game decision, not merely a UI countdown. When a player’s action clock runs out, the server may fold the hand. When a blind level ends, stakes change. When a disconnected player’s grace period expires, the seat may be released. If the timer is lost, the expected transition never happens: a table can stop progressing or a disconnected player can remain marked as online.
The server must fire timers even if the process that armed them has since crashed. The client cannot be trusted to enforce the clock
Why Redis expiry notifications are not enough
A common implementation writes a key with a TTL and subscribes to Redis keyspace notifications. It is simple, but the notification is a Redis Pub/Sub message and is therefore at-most-once. If a subscriber is disconnected during expiry because of a deployment, network interruption or long garbage-collection pause, the event is gone. There is no replay and no durable record that the notification was missed.
- The key and its payload may already have been deleted by the time the notification arrives, often leading to a second shadow key
- Every subscriber receives the event. With multiple game servers, all can race to process the same expiry
- Expiry notifications are emitted when Redis actually deletes the key, which can be later than the nominal TTL deadline.
- A process restarting at the wrong moment can leave a table hung or a disconnected player appearing online indefinitely.
The implementation: timers are records, not notifications
Each timer is represented in two Redis structures:
- A due queue: a sorted set whose member is the timer ID and whose score is the due time in milliseconds.
- A payload hash: the data the handler needs, such as table, player and turn.
-- ARM: payload first, then the due entry. fireAt comes from Redis TIME.
local t = redis.call('TIME')
local nowMs = tonumber(t[1]) * 1000 + math.floor(tonumber(t[2]) / 1000)
local fireAt = nowMs + tonumber(ARGV[2])
redis.call('HSET', KEYS[2], ARGV[1], ARGV[3])
redis.call('ZADD', KEYS[1], fireAt, ARGV[1])Game servers poll the due queue every 100 ms and request timers whose score is at or before Redis’s current time. Nothing is pushed as a one-off event. A timer remains in the queue until its handler completes and acknowledges it, or the game cancels it.
What this guarantees—and what it costs
- A restart does not erase the timer. If a timer becomes due while all servers are down, it is overdue and can be found on the first poll after startup, assuming the Redis data survives.
- Application-server clock skew cannot make a timer fire early, because due-time comparisons use Redis time.
- Long-range timers, such as a tournament registration deadline a week away, use the same queue.
- A maximum lifetime bounds repeated failures. On game servers, a timer that continues failing is removed with a warning after five minutes rather than retried forever.
- Polling costs a sorted-set range query per tick. With a 100 ms interval, a timer may be processed up to approximately one polling interval late, plus scheduling and handler delay.
4. When several servers see the same expired clock: atomic timer claims
The operational problem
A durable timer queue prevents a timer from disappearing, but every game server polls the same queue. When a player’s clock becomes due, multiple instances may see it in the same 100 ms window. Only one should act on that timer. Two folds or two blind-level transitions can corrupt game progression.
Why deleting first or using a simple lock is insufficient
- Delete the timer before running the handler: if the process crashes after deletion but before the fold, the timer is lost.
- Use a lock per timer: this adds coordination and still requires recovery when the lock holder dies.
- Assign a timer to one server: this pins work to a machine and creates an ownership and rebalancing problem when that server stops.
The implementation: lease the timer, then acknowledge with a fence
Claiming a timer does not delete it. An atomic Lua script re-checks that it is still due and moves its score into the future by a lease—30 seconds on game servers. Other pollers then see it as not yet due.
-- CLAIM: re-check it is due, then push it one lease into the future.
local s = redis.call('ZSCORE', KEYS[1], ARGV[1])
if s and tonumber(s) <= nowMs then
local leasedUntil = nowMs + tonumber(ARGV[2])
redis.call('ZADD', KEYS[1], leasedUntil, ARGV[1])
local p = redis.call('HGET', KEYS[2], ARGV[1])
end
return {0}The script provides one claimer per lease window because Redis executes it atomically. The payload is returned in the same call, avoiding a gap between claim and read. If the winning process dies mid-handler, the timer is still present; once the lease expires, another server can claim it
After a successful handler, the poller acknowledges the timer with a compare-and-delete. It removes the timer only if the current sorted-set score still matches the leased until value written by that claim.
-- ACK: delete only if the score is still the one this claim leased.
local s = redis.call('ZSCORE', KEYS[1], ARGV[1])
if s and tonumber(s) == tonumber(ARGV[2]) then
redis.call('ZREM', KEYS[1], ARGV[1])
redis.call('HDEL', KEYS[2], ARGV[1])
return 1
end
return 0The score check prevents a late acknowledgement from deleting a timer that has been re-armed or claimed by another worker. This matters when a handler schedules the next blind level under the same timer ID, or a time-bank extension re-arms a player’s clock. An unconditional delete previously removed the newly armed timer and could stop tournament levels advancing. With a fenced acknowledgement, the score changes and the old acknowledgement becomes a no-op.
A Claimed timer is deleted only by the claim that leased it
Route before claiming
Cash and tournament servers share the timer queue. A poller must check that it can handle a timer before claiming it. A tournament-only server once claimed cash timers it could not process, pushing them 30 seconds out of reach of cash servers; one table remained frozen for 13 minutes. A regression test now covers this case
Pollers also claim no more timers than their worker pool can run, so they do not hold leases on work they cannot start. Idle polling includes jitter to avoid every server hitting Redis in lockstep.
The precise guarantee
The guarantee is one claimer per lease window and at-least-once execution—not exactly-once execution. A handler can run more than once if a server crashes after applying the fold but before acknowledging the timer, or if a handler outlives its lease. The 30-second lease is intended to exceed normal handler time, and every timer handler must be idempotent. The compare-and-set state write described in Section 7 makes duplicate timer execution harmless when the command has already committed.
5. AMQP Competing Consumers - Implementing a fleet based architecture
The operational problem
If tables are assigned to specific game servers, one overloaded server can struggle while another has idle capacity. Scaling out then requires table reassignment, and a server failure leaves its tables waiting for recovery or reassignment. That is unnecessary when the game server does not need to retain table state between commands
- Prefetch is matched to the worker-pool size, limiting unacknowledged messages to the work a server can process.
- There are two levels of parallelism: RabbitMQ distributes messages across server instances, and each instance uses a worker pool sized to its CPU share.
- New instances begin consuming as soon as they connect. When an instance stops, its unacknowledged messages are returned to the queue for other consumers.
Follow-up commands must survive a crash too
A player action can trigger a chain of work: apply the action, process the resulting state change, then schedule the next hand. If that chain runs only inside one process, a crash halfway through can lose the next step. Instead, a handler returns a follow-up command and the processor publishes it back onto the queue. Any eligible server can pick it up.
if (result?.type === "chained-command") {
const next = {
...result,
payload: { ...result.payload, commandId: `${payload.commandId}:${result.topic}` },
};
await this.mqttClient.publish("/ops/cash/server/command",
Buffer.from(JSON.stringify(next)), { qos: 1, retain: false });
}The derived command ID lets a redelivered follow-up be recognized as the same logical step. Publishing is awaited until the broker confirms it at MQTT QoS 1 before the original message is acknowledged. A crash between publishing and acknowledging therefore leaves the follow-up safely published, or leaves the original unacknowledged for redelivery
Failure handling and acknowledgements
Messages are manually acknowledged only after the handler finishes. A shared consumer classifies failures:
The implementation: any eligible server can process any table
PokerInfra uses durable queues per game format: cash.server.inbound and tour.server.inbound. Every game server for that format consumes from the matching queue. For each command, the server loads the table from Redis, applies the poker Hand Evaluator, writes the result and publishes events. The next command for that table may be handled by another instance.
| Failure type | Handling |
|---|---|
| Permanent (invalid input or rule violation) | Move to a dead-letter queue with the error recorded, then acknowledge |
| Retryable (busy lock or transient Redis issue) | Requeue after a short delay so any instance can retry. |
| Unknown | Retry up to a fixed attempt limit tracked in a message header, then dead-letter. |
Error types are recognized by name so the classification survives a worker-thread-to-main-thread hop. Extension queues use the same policy and have their own dead-letter queues.
AMQP delivery tags belong to the channel that delivered a message. If a connection drops and reconnects while a message is in flight, the consumer refuses to acknowledge the message on the new channel. The broker has already requeued it; acknowledging on the wrong channel could acknowledge a different message
6. When client broadcasts and internal commands need different delivery: MQTT outside, AMQP inside
The operational problem
Player clients need a persistent, lightweight connection that can receive table updates and private responses. Internal commands have a different requirement: a command should be processed by one worker, not broadcast to every server. Using one delivery model for both creates unnecessary complexity or incorrect semantics.
MQTT faces the player
- One persistent connection carries all tables a player has open in both directions
- Retained messages provide the current table snapshot when a client subscribes or reconnects
- Hole cards are never published on the public table channel. Each player’s cards use that player’s private topic, are not retained, and are removed from public snapshots before publishing.
- A Last Will message lets the broker report an unexpected client disconnect without waiting for a timeout.
- Message expiry prevents momentary replies, such as an action result, from being treated as current after they have become stale.
AMQP handles internal work
MQTT subscriptions are fan-out: every subscriber receives matching messages. That is appropriate for a table broadcast but not for a command that should be handled by one worker. RabbitMQ’s MQTT plugin does not support MQTT shared subscriptions, so MQTT alone does not provide the command load-balancing model used by this fleet.
AMQP provides durable queues, competing consumers, manual acknowledgement, redelivery and dead-letter queues. RabbitMQ routes MQTT publishes through its built-in amq.topic exchange. A service declares a durable queue and binds it to the exchange using routing keys for the topics it handles. The mapping converts MQTT’s slash separator to AMQP’s dot separator:
/ "/ops/cash/server/command" -> ".ops.cash.server.command"
export const mqttTopicToRoutingKey = (topic: string): string =>
topic.replace(/[/.]/g, (c) => (c === "/" ? "." : "/"));A player action enters over MQTT, passes session and maintenance checks in the lobby service, receives a command ID, and is routed to a durable AMQP queue. One game server processes it. Results return over MQTT. Clients never need to know about AMQP, and internal services do not need a connection per player.
| Attribute | MQTT (client-facing) | AMQP (internal) |
|---|---|---|
| Who connects | Player apps and browsers | Game servers, lobby service, extensions and heartbeat service |
| Delivery shape | Fan-out to subscribers | One consumer handles each delivery; redelivery is possible |
| Used for | Table broadcasts, private cards, replies and presence | Commands, follow-up steps and domain events |
| Reliability tools | QoS 1, retained snapshots, Last Will and expiry | Durable queues, manual acknowledgements, retries and dead-letter queues |
One broker connection per process, not per worker thread
Inside a game server, handlers run on worker threads. Those workers do not open separate broker connections; they pass outbound messages to the main thread, which owns the process’s MQTT connection.
export class MqttClientPublisher {
public publish(topic: string, message: Buffer, options?: MQTTPublishOptions): void {
parentPort?.postMessage({ type: "mqtt-publish", topic, message, options });
}
}This keeps broker connection count proportional to process count rather than thread count, and leaves game logic on worker threads free of direct network I/O.
7. When a stale worker can overwrite a newer hand: locks, compare-and-set and deduplication
The operational problem
A distributed lock can reduce contention, but it cannot by itself guarantee that a worker still has permission to write. Imagine Worker A reads table version 41 and applies a raise. It then stalls long enough for its two-second lock lease to expire. Worker B acquires the lock, reads version 41, applies another valid change and commits version 42. If Worker A wakes and writes its old result without checking the version, it can overwrite B’s work.
The lock is a coordination aid, not the final correctness check
Each table has a Redis lock. A handler acquires it, reads the table, runs the game Hand Evaluator and writes the result. The lock is a two-second lease that renews while the handler runs, has a hard maximum hold time, and supplies an abort signal if the lease is lost. Handlers check that signal before externally visible writes
Even with renewal, a long garbage-collection pause, starved CPU or Redis interruption can prevent renewal. The lease expires, another worker takes the lock, and the first worker may wake believing it still holds it. A lock alone cannot prevent that stale worker from writing.
Compare-and-set rejects stale state
Every table state carries a version. A write says: save this new state only if the current version is still the version I read. The version check and write happen in one Lua script, so another write cannot slip between them.
-- Has this exact command already been applied?
if redis.call('EXISTS', dedupeKey) == 1 then return {'DUPLICATE'} end
local current = cjson.decode(redis.call('GET', stateKey))
if tonumber(current.state.version) ~= expectedVersion then
return {'VERSION_CONFLICT', tostring(current.state.version)}
end
redis.call('SET', stateKey, newStateJson) -- version = expected + 1
redis.call('SET', dedupeKey, '1', 'EX', dedupeTtl)
return {'COMMITTED'}Returning to the example: Worker A wakes expecting version 41, but the table is already at version 42. Its write is rejected. The stale result cannot overwrite the newer table state. The version conflict is classified as retryable because the command may still be valid after a fresh read.
| Attribute | Lock | Compare-and-set (CAS) |
|---|---|---|
| Question answered | Who may work on the table right now? | Is this write still based on the latest state? |
| Protects against | Concurrent handlers working at the same time | A stale handler committing an outdated result |
| Failure mode | Lease may expire without the holder knowing | A version mismatch rejects the write |
| Role in this design | Reduces contention and keeps work orderly | Provides the correctness check for state writes |
The lock runs on a single Redis node and is treated as a best-effort guard. Correctness does not depend on the lock never failing; the version check prevents a stale write from being committed.
Command IDs make at-least-once delivery safe
A broker can redeliver a command after a crash, and a timer can run again after a lost acknowledgement. Each command therefore carries an ID, stamped by the lobby service. Follow-up commands derive their ID from the parent, using the form parent:topic, so redelivery preserves the same logical identity.
- The command ID is checked in the same atomic Lua operation as the table version
- A successfully committed command is remembered for 300 seconds. A later delivery with the same ID returns DUPLICATE and is dropped
- VERSION_CONFLICT is retryable. DUPLICATE and a missing table are permanent failures because retrying them will not make the same command valid
If a process crashes during a command, redelivery has two safe outcomes: if the first attempt committed, the dedupe key causes the redelivery to be dropped; if it did not commit, the command can run against current state. This is at-most-once application within the dedupe window, built on at-least-once delivery—not a claim of exactly-once message processing
Only a COMMITTED result allows the handler to continue. The next turn timer, table broadcast and follow-up step are issued after the state write, so a rejected write produces none of those effects.
8. When downstream systems receive events out of order: per-table sequencing
The operational problem
Domain events such as hand_started, player_acted and hand_ended feed services including hand history and real-time results. With competing consumers, consecutive events for the same table can be handled by different instances and finish in either order. A hand-history service could record a hand’s end before its start, or record the same action twice after redelivery
Routing each table to one fixed consumer is a familiar solution, but it reintroduces the ownership problem that competing consumers were intended to remove. A consumer failure would stall the table’s event stream until reassignment.
Table writes already happen under the table’s lock. PokerInfra uses that ordering point to number events per table. Handlers collect events in an outbox while they work. Just before the lock is released, the outbox reserves the next sequence numbers using the table’s eventSeq counter, saves the counter, and sends events in order.
Producer side: assign sequence numbers under the table lock
const first = (record.eventSeq ?? 0) + 1;
record.eventSeq = first + events.length - 1;
await redis.set(recordKey, record);
events.forEach((event, i) =>
this.options.send({ ...event, streamKey: this.options.streamKeyOf(event), seq: first + i })
);The hook lives in the lock itself, so the roughly 47 event-producing call sites do not each need to implement sequencing. The lock’s scope follows the asynchronous call chain, allowing events created deep inside a helper to be numbered against the correct table. A test with 200 concurrent lock holders on one table produces sequence numbers 1 through 200 without gaps or repeats
Consumer side: apply each table stream in order
Each consumer runs incoming events through a sequence guard. The guard stores the last applied sequence number for each table in Redis and serializes processing for that table with a short lock, across all consumer instances.
| Incoming event | Decision |
|---|---|
| Sequence is the next expected number | Apply it, then record it as last applied. |
| Sequence is at or below the last applied | Treat it as a duplicate and skip it. |
| Sequence is ahead of the next expected number | Wait for the missing event, backing off from 50 ms to 1 second. |
| The event is still missing after 5 seconds | Apply the later event, report a gap, and allow that table’s stream to proceed. |
const last = await this.store.getLastSeq(streamKey);
if (last !== null && seq <= last) return 'duplicate';
const isNext = last === null ? seq === 1 : seq === last + 1;
if (!isNext && this.clock.now() < deadline) return 'early';
await apply();
await this.store.setLastSeq(streamKey, seq);
return isNext ? 'applied' : 'gap';The counter advances only after the handler succeeds; a failed application is retried rather than incorrectly treated as a duplicate. A first-seen event above sequence 1 waits because sequence 1 may still be in flight on another instance. If an event never arrives, the five-second timeout prevents one table from being blocked indefinitely; the gap is reported instead of hidden. Consumers pass every event in a stream through the guard, including events they do not otherwise handle, so gaps remain detectable.
Applying an event and recording its sequence number are separate operations. If a consumer crashes between them, redelivery can apply the event again. Consumer handlers must therefore be idempotent.
9. When an extension is slow or unavailable: isolate it from gameplay
The operational problem
Hand history, real-time results, chat, fraud analysis and streaming are useful platform capabilities, but they should not determine whether a poker action can complete. If the game server waits for a reporting service or chat handler, a slow extension can add latency to the game path. If an extension goes down, its work should wait without blocking core gameplay
Separate durable queues for each extension
Game servers publish domain events and continue; they do not wait for an extension to reply. Each extension owns durable queues bound to the same RabbitMQ topic exchange. If an extension is slow or offline, its queue accumulates work and the extension processes the backlog when it returns
| Extension | Domain-event queue | Command queue | Dead-letter queue |
|---|---|---|---|
| Hand history | ext.hand-history.domain | ext.hand-history.commands | ext.hand-history.dlq |
| Real-time results | ext.realtime-results.domain | None | ext.realtime-results.dlq |
| Chat | ext.chat.domain | ext.chat.commands | ext.chat.dlq |
await this.consumer.start([
{
queue: "ext.hand-history.domain",
exchange: "amq.topic"
bindingKeys: [`${mqttTopicToRoutingKey(EVENTS.DOMAIN_EVENT_PREFIX)}.#`],
deadLetterQueue: "ext.hand-history.dlq",
handler: (msg) => this.processDomainEvent(msg),
},
]);Each extension receives its own copy of the event stream, so adding one does not slow another, and a poison message in one queue does not block unrelated extensions. Extensions scale independently by starting more instances. They compete on their own queues, while the sequence guard preserves per-table event order across instances. Within each process, the main thread handles network I/O and handlers run in worker threads; prefetch matches the worker-pool size.
Each extension maps errors to the shared retry policy. Busy locks, Redis outages and upstream 5xx responses are retryable. Validation failures and 4xx responses are permanent and go to the dead-letter queue.
Routing commands to extensions
Some extensions accept player commands, such as requesting hand history or sending chat. An extension advertises the commands it serves by publishing a registration at startup. The lobby service records those entries in a routing table and forwards matching commands to the extension queue.
- Registration is confirmed, not assumed. After registering, the extension checks its routing table and republishes missing entries every two seconds, up to ten times, covering the race where the lobby service is not ready yet.
- Registrations are not withdrawn on shutdown. The routing table describes where a command belongs, not which process is alive. Commands remain in the durable queue while the extension restarts.
Patterns shared by extensions
- Store raw data and decide visibility at read time. Hand history stores every card; a viewer sees their own cards, never folded cards, and other players’ cards only if they reached showdown
- Reconcile instead of trusting one event. Chat periodically removes rosters for tables that have ended, because a table-ended event can occasionally be missed
- Keep providers swappable. Chat delivery sits behind an interface so its transport can change by configuration without changing the rules around it.
Adding an extension
- Declare its domain, command and dead-letter queues, and bind them to the topics it needs
- Pass every event through the sequence guard
- Map errors to retryable and permanent categories.
- Register the player commands it serves, if any
10. How we test the failure modes
Test the infrastructure behaviour, not just the handler logic
Timer, lock, outbox and sequence tests run against a real Redis server because Redis server time, Lua atomicity and lease expiry are part of the behaviour being tested. A mock cannot faithfully reproduce all of those properties. The timer suite has around 30 cases, including:
- Two pollers racing for the same queue, with each timer claimed once.
- A handler failing and the timer becoming available again after its lease
- Timers becoming due while no poller is running, then firing after restart
- A handler re-arming its own timer and the new timer surviving the old acknowledgement.
- A poller skipping timers it cannot handle without hiding them from the correct server type
- Every branch of the maximum-lifetime sweep.
A separate suite checks the in-memory game-server test double against the real Lua scripts, reducing the risk that faster handler tests drift from production behaviour.
Crash and race tests
Fleet tests fork separate operating-system processes, connect them to real RabbitMQ and Redis, and kill processes with SIGKILL while they are handling messages.
| Scenario | Observed result |
|---|---|
| Three consumer instances process 600 events across 20 tables while instances are killed mid-run. | Every event is applied once and in order per table in the tested scenario. |
| Fifty events are published while every consumer is down. | All 50 are applied in order after consumers restart. |
| Two hundred concurrent lock holders number events for one table. | Sequence numbers 1–200, with no gaps or repeats. |
| A lock lease is lost mid-section under fault injection. | The abort checkpoint stops the stale write before it lands. |
These are test results for the described scenarios, not a blanket claim that every possible failure is impossible. When an edge case is not yet fixed, a test records the known behaviour and labels it as a gap. For example, deleting and recreating a queue while a consumer is attached causes the broker to cancel that consumer; a test records how the consumer behaves next. This keeps edge cases visible and makes changes deliberate
11. What operators should configure and monitor
Settings that affect behaviour
| Setting | What it controls | Value |
|---|---|---|
| Game-server instances per format | Total game capacity; scale by adding instances. | Deployment choice |
| CASH_GAME_CPU_ALLOCATION_PERCENT / TOUR_GAME_CPU_ALLOCATION_PERCENT | CPU share allocated to each format’s worker threads. | Deployment choice |
| NUMBER_WORKERS | Messages a server takes at once and timers it claims at once; keep aligned with worker-thread count | Deployment choice |
| Timer poll interval | Polling contribution to timer lateness. | 100 ms |
| Timer lease | How long a claimed timer is hidden from other servers; must exceed the slowest handler | 30 seconds |
| Timer maximum lifetime | How long a failing timer is retried before removal | 5 minutes |
| Command dedupe window | How long a processed command ID is remembered. | 300 seconds |
| Sequence gap timeout | How long a consumer waits for a missing event before moving on. | 5 seconds |
| Extension instances | Capacity of each extension, scaled independently of the game | Deployment choice |
Signals worth watching
- Dead-letter queues: messages there failed permanently or exhausted retries; inspect the recorded reason and headers
- Sequence-gap warnings: monitor the domain_event_gap log emitted when a consumer advances past a missing event
- Unclaimed timers: a due timer that no process of its kind can fire can indicate that a format’s server fleet is down or misconfigured.
- Queue depth: growing command or extension queues indicate consumers are falling behind and may require additional instances or investigation.
Rules for adding game features
- Nothing player-visible happens before commit. Acquire the table lock, read the state, run the Hand Evaluator, write using compare-and-set, and only then broadcast, arm timers or issue follow-ups.
- Every handler must tolerate running twice. At-least-once delivery is intentional
- Acknowledge timers using the lease fence. Never delete a timer unconditionally if its handler may re-arm it
- Publish events inside the table’s lock so each event receives the correct per-table sequence.
To add a game command, implement a handler class and register it with the command factory. To add a timer, arm it through the timer client with an ID shaped like {format}:{type}:{action}:{id}. To add an extension, follow the four steps in Section 9
12. Evaluating the platform
The fastest way to evaluate a codebase is to inspect the implementation with the people who built it. PokerInfra offers technical walkthroughs for teams evaluating PokerInfra Enterprise, focused on the mechanisms that matter most to their architecture and operational requirements.
- Book a technical walkthrough with the Hand Evaluatorering team.
- Explore the PokerInfra Enterprise source-code licence, including what is included, self-hosting and ownership terms
This paper is intended to make that evaluation more concrete: it describes the failure modes the design addresses, the implementation patterns used, the guarantees they provide and the trade-offs they retain.
Appendix A. Guarantees at a glance
| Mechanism | Guarantee | Depends on | Section |
|---|---|---|---|
| Durable timer queue | A timer is not lost merely because of a restart, deployment or missed notification | Redis persistence and pollers running. | 3 |
| Lease-bump timer claim | One server claims a due timer per lease window; a crashed claim becomes available again. | Lease longer than handler duration | 4 |
| Fenced timer acknowledgement | An old acknowledgement cannot delete a re-armed timer or another claim. | The leased score acts as a fencing value. | 4 |
| Competing consumers | Any eligible server can process any table; capacity grows by adding instances. | Stateless handlers and durable queues. | 5 |
| Manual ack after confirmed follow-up publish | A crash does not lose both the original command and its follow-up | Broker confirmation at MQTT QoS 1 and correct ack ordering. | 5 |
| Compare-and-set state write | A write computed from stale state is rejected. | A version number on every table state. | 7 |
| Command deduplication | A redelivered command is not applied again within the dedupe window. | A command ID and 300-second dedupe record | 7 |
| Per-table event numbering | Events produced for one table receive an ordered sequence. | Events are published inside the table lock. | 8 |
| Sequence guard | Each consumer applies a table’s events in order where available, skips duplicates and reports gaps after timeout | Idempotent consumers and the gap-timeout policy. | 8 |
| Extension isolation | A slow or failed extension does not synchronously block the game path. | A durable queue per extension. | 9 |
Appendix B. Glossary
| Term | Meaning |
|---|---|
| At-least-once | Every message or timer is processed one or more times; duplicates are possible and must be harmless. |
| At-most-once | A message is processed once or not at all; losses are possible |
| Competing consumers | Several instances consume from one queue, with each delivery assigned to one consumer at a time; redelivery can occur after failure. |
| Compare-and-set (CAS) | Write a new value only if the current value is still the version that was read. |
| Dead-letter queue | A queue for messages that fail permanently or exhaust their retry limit |
| Fencing token | A value identifying a lease or claim so a stale holder’s action can be rejected. |
| Idempotent | Safe to apply more than once without changing the final outcome beyond the first successful application. |
| Keyspace notification | A Redis Pub/Sub event emitted when a key changes or expires. |
| Lease | A lock or claim that expires unless it is renewed. |
| Prefetch | The number of unacknowledged messages a broker may deliver to a consumer. |
| Retained message | An MQTT message the broker stores and sends to new subscribers of that topic |