The claim
The safety mission proves that OpenSearch's cluster coordination never forks, whatever the network does: messages lost, duplicated, delayed or reordered, nodes restarting. Progress cannot be proved in that generality. In that model a message may never be delivered, and nothing stops candidates from starting elections in ever higher terms, each start-join discarding the votes and the leadership of the term before. The obstacle is fundamental: Fischer, Lynch and Paterson showed that no deterministic consensus protocol can guarantee termination in an asynchronous network in which a single node may crash. Protocols of the Paxos and Raft family, OpenSearch's included, are designed to be safe always and to make progress when the system behaves well for long enough (partial synchrony, in the terms of Dwork, Lynch and Stockmeyer). OpenSearch makes such periods likely with pre-voting, which keeps nodes that cannot win from starting elections, and with randomized election timers, which make it likely that one candidate runs alone; both are the subject of other missions.
This mission states progress for the executions of a stable period: a window in which one node runs for election, or publishes a cluster state, and the steps excluded below do not occur, while every other step of the cluster model remains possible. Under the hypotheses listed below, it proves that in every such execution the candidate, once in the election term, accepts and counts every voter's join for that term; that every acknowledging node already in the cluster-manager's term has acknowledged the publication or can still accept it, and the cluster-manager accepts and counts every acknowledgement of it sent by an acknowledging node; that once all of a quorum's votes (acknowledgements) are counted, the election is won (the publication is committed); and that such executions exist. No theorem states that the start-joins an election or a publication needs stay enabled: the periods allow them, and handleStartJoin checks only that the requested term is higher than the node's current term. The model, the steps and the safety invariant are those of the safety mission, over the transcribed CoordinationState of OpenSearch revision b2a54ba5c0defba3d1d20423e911d4bf920bab19.
The stable periods
A stable step is a step of the safety model with, for some kinds of step, an extra condition; so every stable execution from a reachable cluster stays reachable, and the safety invariant holds along it.
A stable election for a candidate c in term t with voters Q, writing (kt, kv) for the term and version of c's accepted state when the period starts, excludes:
- a start-join request in a term at least
t, at any node, other than c's own request for t: no competing candidate for t or a higher term;
- any new publication;
- a bootstrap of
c or of a voter, and a restart of c or of a voter that is cluster-manager-eligible (the election theorems assume they all are; restartAny, the restart of a node that is not eligible, is not restricted);
- the acceptance, by
c or by a voter, of a publication fresher than (kt, kv): a stale publication of a former cluster-manager arriving during the election, which would make the voter's join fresher than c's state, and hence rejected.
Everything else remains possible, in any order and any number of times: the delivery of joins, publish responses and commits to any node; the acceptance of publications no fresher than (kt, kv) by the voters, and of any publication by the other nodes; start-join requests of lower terms at any node, voters included; commits at c and at the voters; bootstraps and restarts of the other nodes.
A stable publication for the cluster-manager c of term t with acknowledging nodes Q excludes:
- a start-join request in a term above
t at c or at a node of Q;
- a further publication by
c;
- a restart of
c if it is cluster-manager-eligible, as every publisher in a reachable cluster is (restartAny, the restart of a node that is not eligible, is not restricted).
Everything else remains possible: start-join requests of any term at the other nodes and of terms up to t at the nodes of Q (which lets them reach term t), publications by other nodes (a stale cluster-manager of an older term, or a node that has won a later term), the acceptance of any publication by any node, joins, commits (at c too, while its publication is in flight), bootstraps, and restarts of every node other than c, the nodes of Q included.
What is proved
Every theorem starts from a reachable cluster and assumes, as the safety theorems do, compatible bootstrap configurations and OpenSearch's default election strategy.
- Elections. Let
c and the voters Q be cluster-manager-eligible and in terms below t.
election_vote_accepted: if every voter's accepted state is no fresher than c's (compared as handleJoin compares them) and c's last accepted configuration is nonempty, then in every stable election execution, once c is in term t, c accepts every voter's join for t whenever it arrives, and counts the vote.
election_won: if Q is a quorum of both of c's configurations, c has won as soon as it is in term t and has counted the votes of all of Q, whatever else happened in the period.
election_possible (the goal): if Q is a quorum of both of c's configurations and every voter's accepted state is no fresher than c's, there is a stable election execution at the end of which c is in term t and has won. The theorem asserts only that such an execution exists; in the proof's witness, c handles its own start-join request for t, each voter not yet in term t handles it, and c handles the voters' joins.
- Publications. Let
c be elected in term t, with s its latest publication, and Q cluster-manager-eligible.
publication_acks_accepted: in every stable publication execution, every node of Q in term t has already acknowledged s or accepts it now, and c accepts and counts every acknowledgement of s from a node of Q, whenever it arrives.
publication_committed: if Q is a quorum of c's two configurations and of s's two configurations, and c has counted no acknowledgement yet (as right after publishing), then in every stable publication execution the publication is committed (a commit for (t, s.version) has been sent) as soon as c has counted the acknowledgements of all of Q. This theorem does not cover a period that starts after c has counted some acknowledgement.
publication_possible: if Q is a quorum of the same four configurations and the nodes of Q are in terms at most t, whatever acknowledgements c has already counted, there is a stable publication execution at the end of which the publication is committed. The theorem asserts only that such an execution exists; in the proof's witness, each node of Q moves to term t if needed and accepts and acknowledges s unless it already has, and c counts the acknowledgements.
The witness executions show that the premises of election_won and publication_committed are reached: the stable periods allow every step the witnesses take, so these theorems are not vacuous.
Why four configurations. A cluster-manager commits when its publish votes are a quorum of its current last committed configuration and of the configuration being published. Java's handleCommit lets it apply the commit of its accepted state while its next publication is in flight (the behaviour, called D1, by which the safety mission's model departs from Elastic's TLA+ model); its last committed configuration then becomes its last accepted one, and when it later accepts its own publication s it becomes s's last committed configuration again: it moves backwards. For example, let s carry the last committed configuration {1,2,3} and the last accepted configuration {1,2,4}, a reconfiguration in flight, and let the cluster-manager 1 have applied the commit of its accepted state, so that its own last committed configuration is {1,2,4}. With Q = {2,4}, a quorum of {1,2,4} but not of {1,2,3}: if node 1 accepts s before counting the acknowledgements of nodes 2 and 4, its last committed configuration is {1,2,3} again, and all of Q is counted without a commit. publication_committed therefore asks Q to be a quorum of every configuration the cluster-manager may be using when the last acknowledgement arrives: its own two and the published state's two.
The proofs rest on the safety mission's invariant, which supplies, for instance, that a node that has not won holds no quorum of both its configurations, and that every publication of the cluster-manager's term is its own, with a version at most its last published version. Since a node has always acknowledged the publication it holds, a node of Q that rejects s has therefore already acknowledged it.
What this rests on and what it does not say
The safety mission's model and inductive invariant (its definitions are imported, and its theorems used as published), the per-node transcription of CoordinationState, and the argument of the safety mission that the Coordinator's calls are steps of the model.
- A stable period is a hypothesis. Nothing says that a real network reaches one, how long it lasts or how long its steps take: the model has no time. The theorems are about finite executions; that progress eventually happens follows only under the informal assumption that the needed deliveries take place while the period lasts.
- The links to the other missions are informal. A pre-voting round that has started an election establishes the election hypotheses (a quorum of voters no fresher than the candidate, in terms below the new term) if its senders have not changed since answering, and randomized timers make it likely that a candidate fires well before the others; neither link is a composed theorem.
- Failure detection is not modelled. The
LeaderChecker and FollowersChecker, through which nodes notice a failed cluster-manager or follower and become candidates, and the publication timeouts are outside the model.
- Inherited assumptions. Compatible bootstrap configurations, the default election strategy, and the safety mission's assumptions (fixed roles, no disk rollback, no offline tools, exact remote publication) and per-node abstractions.
- Remote-store writes are taken to succeed. On a failed remote cluster-state write, Java throws after the local effects where the transcription returns the result (
handlePublishRequest, CS:453-456; handlePreCommit after the vote is counted in handlePublishResponse, CS:503-510). For safety this is message loss; for progress it is not: in OpenSearch a node can accept s locally without acknowledging it and then reject every re-delivery of s, and the acknowledgement that completes a quorum can be counted without a commit being sent. The theorems describe OpenSearch only when remote writes succeed or no remote cluster-state store is configured.
- The remote commit write is taken to succeed. With a remote cluster-state store configured, OpenSearch's
handlePublishResponse, once the counted votes form a publish quorum, marks the published state as committed in the store before returning the commit (handlePreCommit, CoordinationState.java:503-510, 593-599); if that write fails, the method throws after counting the vote, and no commit is sent. The transcription has no remote store and returns the commit. For safety this is a lost message; for progress it is not: in OpenSearch the acknowledgement that completes a quorum can be counted without a commit being sent. Read for OpenSearch, the publication theorems therefore presuppose that this write succeeds or that no remote cluster-state store is configured.