We also had at some point a similar issue, where minimum_master_nodes did not prevent the cluster from having two different views of the nodes at the same time.

As our indices were created automatically, some of the indices were created twice, once in every half of the cluster with the two masters broadcasting different states, and after a full cluster restart some shards were unable to be allocated, as the state has been mixed up. This was on 0.17. so I am not sure, if data would still be lost, as the state is now saved with the shards. But the other question is what happens when an index exists twice in the cluster (as it has been created on every master).

I think we should have a method to recover from such a situation. As I don't know how the zen discovery works exactly, I can not say how to solve it, but IMHO a node should only be in one cluster, in your second image node 1 should either be with 2, preventing 3 from becoming master, or with node 3, preventing 2 from staying master.

see Issue #2117 as well, I'm not sure if the Unicast discovery is making it worse for you, but I think we captured the underlying problem over on that issue, but would like your thoughts too.

From #2117:

The split brain occurs if the nodeId(UUID) of the disconnected node is such that the disconnected node picks itself as the next logical master while pinging the other nodes(NodeFaultDetection).

Ditto.

The split brain only occurs on the second time that the node is disconnected/isolated.

I see a split on the first partial isolation. To me, these bug reports look like two different problems.

I believe I ran into this issue yesterday in a 3 node cluster- a node elects itself master when the current master is disconnected from it. The remaining partipant node toggles between having the other nodes as its master before settling on one. Is this what you saw @saj?

Yes, @trollybaz.

I ended up working around the problem (in testing) by using elasticsearch-zookeeper in place of Zen discovery. We already had reliable Zookeeper infrastructure up for other applications, so this approach made a whole lot of sense to me. I was unable to reproduce the problem with the Zookeeper discovery module.

I'm pretty sure we're suffering from this in certain situations, and I don't think that it's limited to unicast discovery.

We've had some bad networking, some Virtual Machine stalls (result of SAN issues, or VMWare doing weird stuff), or even heavy GC activity can cause enough pauses for aspects of the split brain to occur.

We were originally running pre-0.19.5 which contained an important fix for an edge case I thought we were suffering from, but since moving to 0.19.10 we've had at least one split brain (VMware->SAN related) that caused 1 of the 3 ES nodes to lose touch with the master, and declare itself master, while still then maintaing links back to other nodes.

I'm going to be tweaking our ES logging config to output DEBUG level discovery to a separate file so that I can properly trace these cases, but there have just been too many of these not to consider ES not handling these adversarial environment cases.

I believe #2117 is still an issue and is an interesting edge case, but I think this issue here best represents the majority of the issues people are having. My gut/intuition seems to indicate that the probability of this issue occurring does drop with a larger cluster, so the 3-node, minimum_master_node=2 is the most prevalent case.

It seems like when the 'split brain' new master connects to it's known child nodes, any node that already has an upstream connection to an existing master probably should be flagging it as a problem, and telling the newly connected master node "hey, I don't think you fully understand the cluster situation".

I believe there are two issues at hand. One being the possible culprits for a node being disconnected from the cluster: network issues, large GC, discover bug, etc... The other issue, and the more important one IMHO, is the failure in the master election process to detect that a node belongs to two separate clusters (with different masters). Clusters should embrace node failures for whatever reason, but master election needs to be rock solid. Tough problem in systems without an authoritative process such as ZooKeeper.

To add more data to the issue: I have seen the issue on two different 0.20RC1 clusters. One having eight nodes, the other with four.

I'm not sure the former is really something ES should be actively dealing with, the latter I agree, and is the main point here, in how ES detects and recovers from cases where 2 masters have been elected.

There was supposed to have been some code in, I think, 0.19.5 that 'recovers' from this state by choosing the side that has the most recent ClusterStatus object (see Issue #2042) , but it doesn't appear in practice to be working as expected, because we get these child nodes accepting connections from multiple masters.

I think gathering the discovery-level DEBUG logging from the multiple nodes and presenting it here is the only way to get further traction on this case.

It's possible going through the steps in Issue #2117 may uncover edge cases related to this one (even though the source conditions are different); at least it might be a reproducible case to explore.

@s1monw nudge - have you had a chance to look into #2117 at all... ? :)

Paul, I agree that the former is not something to focus on. Should have stated that. :) The beauty of many of the new big data systems is that they embrace failure. Nodes will come and go, either due to errors or just simple maintenance. #2117 might have a different source condition, but the recovery process after the fact should be identical.

I have enabled DEBUG logging at the discovery level and I can pinpoint when a node has left/joined a cluster, but I still have no insights on the election process.

suffered from this the other day when an accidental provisioning error had a 4GB ES Heap instance running on a 4GB O/S memory, which was always going to end up in trouble. The node swapped, process hung, and the intersection issue described here happened.

Yes, the provisioning error could have been avoided, yes, probably use of mlockall may have prevented the destined-to-die-a-horrible-swap-death, but there's other scenarios that could cause a hung process (bad I/O causing stalls for example) where the way ES handles the cluster state is poor, and leads to this problem.

we hope very much someone is looking hard into ways to make ES a bit more resilient when facing these situations to improve data integrity... (goes on bended knees while pleading)

Btw. why not adopt ZK, which I believe would make this situation impossible(?)? I don't love the extra process/management that the use of ZK would imply..... though maybe it could be embedded, like in SolrCloud, to work around that?

From my understanding, the single embedded Zookeeper model is not ideal for production and that a full Zookeeper cluster is preferred. Never tried myself, so I cannot personally comment.

FYI - there is a zookeeper plugin for ES

Oh, I didn't mean to imply a single embedded ZK. I meant N of them in different ES processes. Right Simon, there is the plugin, but I suspect people are afraid of using it because it's not clear if it's 100% maintained, if it works with the latest ES and such. So my Q is really about adopting something like that and supporting it officially. Is that a possibility?

@otisg: The problem with the ZK plugin is that with clients being part of the cluster, they need to know about ZK in order to be able to discover the servers in the cluster. Some client libraries (such as the one used by the application that started this bug report -- I'm a colleague of Saj's) doesn't support ZK discovery. In order for ZK to be a useful alternative in general, there either needs to be universal support of ZK in client libraries, or a backwards-compatible way for non-ZK-aware client libraries to discover the servers (perhaps a ZK-to-Zen translator or something... I don't know, I've got bugger-all knowledge of how ES actually works under the hood).

this issue (partial network splits causing split brain) is fixed now, yes.

I'm not sure why this issue was closed--people keep citing it and saying the problem is solved, but the Jepsen test from earlier in this thread still fails. Partial network partitions (and, for that matter, clean network partitions, and single-node partitions, and single-node pauses) continue to result in split-brain and lost data, for both compare-and-set and document-creation tests. I don't think the changes from #7493 were sufficient to solve the problem, though they may have improved the odds of successfully retaining data.

For instance, here's a test in which we induce randomized 120-second long intersecting partitions, for 600 seconds, with 10 seconds of complete connectivity in between each failure. This pattern resulted in 22/897 acknowledged documents being lost due to concurrent, conflicting primary nodes. You can reproduce this in Jepsen 7d0a718 by going to the elasticsearch directory and running lein test :only elasticsearch.core-test/create-bridge--may take a couple runs to actually trigger the race though.

I'm not sure why this issue was closed

This issue, as it is stated, relates to have two master nodes elected during partial network split, despite of min_master_nodes. This issue should be solved now. The thinking is that we will open issues for different scenarios as they are discovered. An example is #7572 as well as your recent tickets (#10407 & #10426). Once we figure out the root cause of those failure (and the one mentioned in your previous comment) and if it turns out to be similar to this issue, it will of course be re-opened.

Not directly on topic to this issue, but why is it so difficult to avoid/prevent this split brain issue? If there are two master nodes on a network (ie, a split brain configuration), why can't there be some protocol for the two masters to figure out which one should become a slave?

I imagine some mechanism would need to detect that the system is in a split-brain state, and then a heuristic would be applied to choose the real master (e.g., oldest running server, most number of docs, random choice, etc.). This probably takes work to do, but it does not seem too difficult.

Michael: Split brain occurs precisely because the two masters can't
communicate. If they could they would resolve it.

On Thu, Feb 25, 2016 at 6:54 PM Michael Sander notifications@github.com
wrote:

Not directly on topic to this issue, but why is it so difficult to
avoid/prevent this split brain issue? If there are two master nodes on a
network (ie, a split brain configuration), why can't there be some protocol
for the two masters to figure out which one should become a slave?

I imagine some mechanism would need to detect that the system is in a
split-brain state, and then a heuristic would be applied to choose the real
master (e.g., oldest running server, most number of docs, random choice,
etc.). This probably takes work to do, but it does not seem too difficult.

—
Reply to this email directly or view it on GitHub
#2488 (comment)
.

Got it. Earlier this week two nodes in my cluster appeared to be fighting for who was the master of the cluster. They were both on the same network and I believe were in communication with each other, but they went back and forth over which was the master. I shut down one of the nodes, gave it five minutes, restarted that node, and everything was fine. I thought that this was a split brain issue, but I guess it may be something else.

Earlier this week two nodes in my cluster appeared to be fighting for who was the master of the cluster.

@speedplane Do you have exactly two master-eligible nodes? Do you have minimum master nodes set to two (if you're going to run with exactly two master-eligible nodes you should, although this means that your cluster becomes semi-unavailable if one of the masters faults; ideally if you have multiple master-eligible nodes you'll have at least three and have minimum master nodes set to a quorum of them)?

I thought that this was a split brain issue, but I guess it may be something else.

Split brain is when two nodes in a cluster are simultaneously acting as masters for that cluster.

@jasontedor Yes, I had exactly two nodes, and minimum master nodes was set to one. I did this intentionally for the exact reason you described. It appeared that the two nodes were simultaneously acting as a master, but they were both in communication with each other, so shouldn't they be able to resolve it, as @hamiltop suggests?

Yes, I had exactly two nodes, and minimum master nodes was set to one.

@speedplane This is bad because it does subject you to split brain.

I did this intentionally for the exact reason you described.

That's not what I recommend. Either drop to one (and lose high-availability), or increase to three (and set minimum master nodes to two).

It appeared that the two nodes were simultaneously acting as a master, but they were both in communication with each other, so shouldn't they be able to resolve it, as @hamiltop suggests?

What evidence do you have that they were simultaneously acting as master? How do you know that they were in communication with each other? What version of Elasticsearch?

What evidence do you have that they were simultaneously acting as master?

In the Big Desk plugin, the little star next to node name kept on bouncing back and forth between my two nodes (see screenshot).

How do you know that they were in communication with each other?

I don't think I explicitly tested whether one could contact the other, but I was able to ssh into both, they were on the same network, and there did not appear to be any network issues.

What version of Elasticsearch?

1.7.3

In the Big Desk plugin, the little star next to node name kept on bouncing back and forth between my two nodes (see screenshot).

@speedplane I'm not familiar with the Big Desk plugin, sorry. Let's just assume that it's correct and as you say. Have you checked the logs or any other monitoring for repeated long-running garbage collections pauses on both of these nodes?

I don't think I explicitly tested whether one could contact the other, but I was able to ssh into both, they were on the same network, and there did not appear to be any network issues.

Networks are fickle things but I do suspect something else here.

1.7.3

Thanks.

@speedplane "2 node situation" is inherently hard to deal with because there is no one metric iy could be decided which one should be shot down.

"Most written to" or "last written to" doesnt really mean much and in most cases alerting that something is wrong is preferable to "just throw away whatever other node had".

That is why a lot of distributed software recommends at least 3 nodes, because with 3 there is always majority, so you can set it up to only allow requests if at least n/2+1 nodes are up