[SPARK-23679][YARN] Setting RM_HA_URLS for AmIpFilter to avoid redirect failure in YARN mode #22164
Conversation
|
Test build #94991 has finished for PR 22164 at commit
|
|
@vanzin @tgravescs would you please help to review, thanks! |
|
Gently ping again @vanzin @tgravescs . Thanks! |
|
Is this something that changed in Hadoop 3 (or some post-2.6 version)? I'm pretty sure the existing |
|
Also, usual nit: PR title should explain the fix, not the problem. |
jerryshao
changed the title
|
I think it should be related to this JIRA (https://issues.apache.org/jira/browse/YARN-7269). Seems like a Hadoop 2.9/3.0+ issue. |
| @@ -126,4 +136,21 @@ private[spark] class YarnRMClient extends Logging { | |||
| } | |||
| } | |||
|
|
|||
| private def getUrlByRmId(conf: Configuration, rmId: String): String = { | |||
There was a problem hiding this comment.
The code looks ok, but it also looks similar to this:
I'm wondering if we could just call that class instead, somehow? It seems available in 2.6 which is the oldest version we support.
There was a problem hiding this comment.
For the Spark usage, I think it may not be so useful to use AmFilterInitializer, because we need to pass the filter parameters to driver either from RPC (client mode) or from configuration (cluster mode), in either way we should know how to set each parameter, so from my understanding using AmFilterInitializer seems not so useful.
There was a problem hiding this comment.
Ah, right, it would be hard to use that class in the client case.
|
Merging to master. |
|
Thanks @vanzin . |
What changes were proposed in this pull request?
YARN
AmIpFilteradds a new parameter "RM_HA_URLS" to support RM HA, but Spark on YARN doesn't provide a such parameter, so it will be failed to redirect when running on RM HA. The detailed exception can be checked from JIRA. So here fixing this issue by adding "RM_HA_URLS" parameter.How was this patch tested?
Local verification.