I think you should register "global" services to all of master nodes to have HA. But as I understand services should register to client node, and it should register using localhost:8500. It prevets returnig client IP address when that client node is down

@spawluk Thanks for your answer,I tried to register the service to the client and use localhost to register, but the result is still not working. When the client is down, the service registered on it is still unavailable.

Firstly I would have some question.
What is the meaning of the node is broken? Is that removed from the cluster? Was it the elected leader?
If these nodes were in cluster have you seen your registered service on all node? Are they populate the information between each other?
After the node broke, was that restarted automatically and joined back to the cluster?

hi @PumpkinSeed
A broken node means that the simulated node is down, kill it with the kill command, and the node is not a leader.
When the cluster is running normally, I can see that the services I registered are normal on other nodes, but after one of the nodes is broken, I will see that the node-check of the service registered by the node is not passed.
After the node is disconnected, the services registered to the node will not be discovered until the node restarts and joins the cluster.

The problem description is the same as this, but the solution is not solved.

https://stackoverflow.com/questions/52535678/achieve-fault-tolerance-with-consul-cluster/58501596#58501596

I should replicate the situation to have a clear answer to you. What happens if you try to register the service via a consul agent? It should solve the issue where the services not connecting directly to the servers but I'm not totally sure about.

I'm not sure if I understood correctly what you doing. I'll try to describe what I understood:
You:

1. Created 3 node cluster of master eliglabe nodes
2. Added 1 Consul client node to cluster that connected to maser nodes successfuly
3. Registered service into Consul Client node
4. Killed Consul Client node
5. Cannot find service registered to Consul Client node

Do I understand correctly?

If so, that is correct consul behavior. Any consul node that you register service into becomes main source of truth of service existence. This node f.e. executes health-checks of registered service and passes information about it to master nodes. If you kill main source of truth, service is down.

The way we work with this is many servers with same service, each server has its own consul client instance connected to masters servers. This is what I understand correct architecture of consul infrastructure

@spawluk you understand, thanks, and I still have some questions to ask.

This kind of architecture requires us to start multiple instances of the same service to register with different agents to ensure that the service is finally available.
If this is the case, deploying multiple instances on k8s can become cumbersome. Is it solved by randomly registering the agent with the nginx agent?

This architecture must require that the service is also clustered. If the service is single-node, after the consul-agent is down, the single-node service registered on it will not be discovered.

Is it understandable that the high availability of a consul cluster is dependent on the service cluster being highly available.

First I have to say that I didn't use k8s ever so I may be wrong here.

I think consul authors ment that every kubernetes container with service should have consul client installed along with service. Then when you scale service, you add consul client for each copy of your service. In that approach it does not matter that consul is down in one of your contaiers. It disables also that one single service, but there are other copies of it in cluster, hence HA

if the service is a single node,The registered consul-agent is down. Does it mean that this service will not be available?
In fact, this is caused by the consul machine, not the service itself.Is it understandable that the high availability of a consul cluster is dependent on the service cluster being highly available.

I think you misunderstood the service discovery. You should somehow reach the consensus between the Consul cluster's nodes. This means if you register the service you should see it on all nodes of the Consul cluster. The best if you have 3 server node whose are independent from each other, but sharing the state. You can follow the installation here. On the other hand you should have Consul clients on each pod where you want to have service discovery. These client's should connect to the servers. If they don't connect they won't do anything with the requests, because the clients directly send all of the requests to the one of the server nodes.
You can join the client to the server's as the follow: consul join -http-addr=<ip of this machine>:8500 <ip of the consul server agent>.
If everything wrapped up, they can be fault-tolarent on the way if one of the server get destroy the client will still know about the state changes.

Howdy all,
We are going to close this issue due to it being a question, but feel free to continue talking on this thread. If this is a bug or feature request, feel free to create a new issue and link it to this one! 👍

Issues on GitHub for Consul are intended to be related to bugs or feature requests, so we recommend using our other community resources instead of asking here.

• FAQ (Frequently asked questions)
• Consul Guides
• Any other questions can be sent to the Consul section of the [HashiCorp forum]: (https://discuss.hashicorp.com/c/consul)

If you feel this is a bug, please open a new issue with the appropriate information.