Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Consumer rebalance problem when using docker container #667

Closed
vancefantasy opened this issue Jan 7, 2019 · 10 comments
Closed

Consumer rebalance problem when using docker container #667

vancefantasy opened this issue Jan 7, 2019 · 10 comments

Comments

@vancefantasy
Copy link

vancefantasy commented Jan 7, 2019

BUG REPORT

  1. Please describe the issue you observed:

docker环境下,2个不同的consumer实例,生成的clientId是重复的,进而导致rebalance的结果混乱。(经过debug发现获取到的ip是一样的,怀疑跟rancher网络有关)

  1. Please tell us about your environment:

rocketmq client "4.3.0"
java version "1.8.0_181"
Docker version "18.09.0"

  1. Other information (e.g. detailed explanation, logs, related issues, suggestions how to fix, etc):

建议:改进ClientConfig类中instanceName的生成规则

@vongosling vongosling changed the title 在docker环境下,consumer端rebalance发生混乱 Consumer rebalance problem when using docker container Jan 9, 2019
@huanwei
Copy link

huanwei commented Jan 9, 2019

What's the current clientId generation rule in ClientConfig class? And how to reproduce it?

@vancefantasy
Copy link
Author

vancefantasy commented Jan 12, 2019

@huanwei

clientId:

 public String buildMQClientId() {
    StringBuilder sb = new StringBuilder();
    sb.append(this.getClientIP());
    sb.append("@");
    sb.append(this.getInstanceName());
    if (!UtilAll.isBlank(this.unitName)) {
        sb.append("@");
        sb.append(this.unitName);
    }
    return sb.toString();
}

instanceName:

private String instanceName = System.getProperty("rocketmq.client.name", "DEFAULT");

public void changeInstanceNameToPID() {
    if (this.instanceName.equals("DEFAULT")) {
        this.instanceName = String.valueOf(UtilAll.getPid());
    }
}

在docker环境下(使用host模式),获取到的clientIp和pid都是一样的。当然clientIp一样的确切原因还没完全确认,可能和引入Rancher有关

wx20190112-180557 2x

#668

@caigy
Copy link
Contributor

caigy commented May 21, 2019

I encountered the same problem. Under HOST networking mode, all dockers managed by Rancher are with the same docker0 IP, which is 172.17.0.1. Thus, RemotingUtil.getLocalAddress() always returns "172.17.0.1" for those dockers, which leads to collision between consumers with the same pid, if using default "IP@pid" clientId.

  • RocketMQ client Version: 4.3.2
  • Rancher Version: 1.16.10
  • Docker Version: 18.09.5

@vongosling
Copy link
Member

@huanwei This is a known problem in the docker container if we deploy mutl-docker containers in the one machine.

@duhenglucky
Copy link
Contributor

@caigy @huanwei if used Rancher to manage container, all docker will always return "172.17.0.1" when getLocalAddress, not only in bridge model, but also in host model, so I think it may be an issue caused by Rancher, so you can temporarily solve this problem by setting the instanceName.

@huanwei
Copy link

huanwei commented May 22, 2019

That's known issue caused by Rancher container network. Setting different instanceName should solve this problem, just as @duhenglucky suggested.

@haycco
Copy link

haycco commented Jul 1, 2019

Which version have been solved, I also meeting the same problem.

@maixiaohai
Copy link
Contributor

Has this be fixed?

@Aaron-TangCode
Copy link
Contributor

除了通过手动自定义instanceName外,RocketMQ会做什么来兼容吗?还是不考虑兼容了?

@Git-Yang
Copy link
Member

Git-Yang commented Jan 8, 2022

I would like a more general way to completely solve the problem of instance conflicts. #3680

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

10 participants