Skip to content

[FLINK-35332][yarn] Removed setting rest.address to the actual IP add… #24775

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 1 commit into from
Nov 6, 2024

Conversation

paul8263
Copy link
Contributor

…ress of the Yarn NodeManager in order to support dual network environment

What is the purpose of the change

Removed setting rest.address to the actual IP address of the Yarn NodeManager in order to support dual network environment.

Given the Hadoop Yarn cluster with dual networks:

  • 192.168.x.x: For data transfer. Speed: 10Gbps.
  • 10.x.x.x: For management only. Speed: 1Gbps.

A client outside the Hadoop Yarn cluster is configured, with management network only(10.x.x.x) and data transfer high speed network not accessible. To reproduce, we sumbit a Flink job from this client(Batch word count for example), the job can be successfully submitted but the result cannot be retrieved, with the exception: Connection refused: {jobmanager_hostname}:{jm_port}. The root cause is the job manager rest address is bind to its actual address (192.168.x.x) rather than 0.0.0.0. Manually setting rest.bind-address does not work.

One of the changes in Flink-24474 in YarnEntrypointUtils overwrites RestOptions.BIND_ADDRESS to the node's actual address. This change should be reverted.

Brief change log

  • flink-yarn/src/main/java/org/apache/flink/yarn/entrypoint/YarnEntrypointUtils.java

Verifying this change

Please make sure both new and modified tests in this PR follow the conventions for tests defined in our code quality guide.

This change added tests and can be verified as follows:

Compiled and passed manually test.

Does this pull request potentially affect one of the following parts:

  • Dependencies (does it add or upgrade a dependency): no
  • The public API, i.e., is any changed class annotated with @Public(Evolving): no
  • The serializers: no
  • The runtime per-record code paths (performance sensitive): no
  • Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn, ZooKeeper: (no
  • The S3 file system connector: no

Documentation

  • Does this pull request introduce a new feature? no
  • If yes, how is the feature documented? (not applicable / docs / JavaDocs / not documented)

@flinkbot
Copy link
Collaborator

flinkbot commented May 13, 2024

CI report:

Bot commands The @flinkbot bot supports the following commands:
  • @flinkbot run azure re-run the last Azure build

@ferenc-csaky
Copy link
Contributor

@paul8263 Logic LGTM, but pls. add a unit test to YarnEntrypointUtilsTest that validates we keep the preconfigured value.

@paul8263
Copy link
Contributor Author

paul8263 commented Nov 5, 2024

Hi @ferenc-csaky ,
Thank you for reviewing the code. Sorry for the late response. YarnEntrypointUtilsTest has been added.

Copy link
Contributor

@ferenc-csaky ferenc-csaky left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM! Will merge by tomorrow EOD (CET) if CI passes and no objections until then.

@ferenc-csaky
Copy link
Contributor

@paul8263 pls. rebase your PR against the latest master and force-push again. The error that failed the CI is already addressed.

…ress of the Yarn NodeManager in order to support dual network environment
@paul8263
Copy link
Contributor Author

paul8263 commented Nov 6, 2024

@ferenc-csaky
Thanks. I rebased the PR and I am waiting for the CI to finish.

@ferenc-csaky
Copy link
Contributor

@flinkbot run azure

@ferenc-csaky ferenc-csaky merged commit b138d47 into apache:master Nov 6, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

3 participants