Skip to content

[SUPPORT] Avoid direct dependency on Flink Table Planner Jar #10937

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Open
ysmintor opened this issue Mar 29, 2024 · 2 comments
Open

[SUPPORT] Avoid direct dependency on Flink Table Planner Jar #10937

ysmintor opened this issue Mar 29, 2024 · 2 comments
Labels
feature-enquiry issue contains feature enquiries/requests or great improvement ideas flink Issues related to flink

Comments

@ysmintor
Copy link

ysmintor commented Mar 29, 2024 •
edited
Loading

Tips before filing an issue

  • Have you gone through our FAQs?

  • Join the mailing list to engage in conversations and get faster support at dev-subscribe@hudi.apache.org.

  • If you have triaged this as a bug, then file an issue directly.

Describe the problem you faced
Apache Kyuubi with Flink need flink-table-planner-loader(which is deault in Flink /lib), However, Hudi api depends flink-table-planner_2.12.
A clear and concise description of the problem.
There is another guy who suffer the two jar conflict problem. Is there any chance to move on flink-table-planner-loader.jar ?

To Reproduce
https://nightlies.apache.org/flink/flink-docs-master/docs/dev/configuration/advanced/#table-planner-and-table-planner-loader

In the upcoming Flink versions, we will stop shipping the flink-table-planner_2.12 artifact in the Flink distribution. We strongly suggest migrating your jobs and your custom connectors/formats to work with the API modules, without relying on planner internals. If you need some functionality from the planner, which is currently not exposed through the API modules, please open a ticket in order to discuss it with the community.

Expected behavior

A clear and concise description of what you expected to happen.

Environment Description

  • Hudi version : 1.14.1

  • Flink version : 1.17.2

  • Hive version : 3.1.3

  • Hadoop version :

  • Storage (HDFS/S3/GCS..) :

  • Running on Docker? (yes/no) : no

Additional context

Add any other context about the problem here.

Stacktrace

Add the stacktrace of the error.
If I use flink-table-planner_2.12_1.17.2.jar, Apache kyuubi will have problem below.

Error: Error operating ExecuteStatement: org.apache.flink.table.api.TableException: Error obtaining Flink planner module ClassLoader. Make sure a flink-table-planner-loader.jar is on the classpath
        at org.apache.kyuubi.engine.flink.operation.ExecuteStatement.getPlannerModuleClassLoader(ExecuteStatement.scala:117)
        at org.apache.kyuubi.engine.flink.operation.ExecuteStatement.parseExtendedStatement(ExecuteStatement.scala:90)
        at org.apache.kyuubi.engine.flink.operation.ExecuteStatement.executeStatement(ExecuteStatement.scala:69)
        at org.apache.kyuubi.engine.flink.operation.ExecuteStatement.runInternal(ExecuteStatement.scala:62)
        at org.apache.kyuubi.operation.AbstractOperation.run(AbstractOperation.scala:173)
        at org.apache.kyuubi.session.AbstractSession.runOperation(AbstractSession.scala:100)
        at org.apache.kyuubi.session.AbstractSession.$anonfun$executeStatement$1(AbstractSession.scala:130)
        at org.apache.kyuubi.session.AbstractSession.withAcquireRelease(AbstractSession.scala:81)
        at org.apache.kyuubi.session.AbstractSession.executeStatement(AbstractSession.scala:127)
        at org.apache.kyuubi.service.AbstractBackendService.executeStatement(AbstractBackendService.scala:66)
        at org.apache.kyuubi.service.TFrontendService.ExecuteStatement(TFrontendService.scala:253)
        at org.apache.kyuubi.shaded.hive.service.rpc.thrift.TCLIService$Processor$ExecuteStatement.getResult(TCLIService.java:1557)
        at org.apache.kyuubi.shaded.hive.service.rpc.thrift.TCLIService$Processor$ExecuteStatement.getResult(TCLIService.java:1542)
        at org.apache.kyuubi.shaded.thrift.ProcessFunction.process(ProcessFunction.java:38)
        at org.apache.kyuubi.shaded.thrift.TBaseProcessor.process(TBaseProcessor.java:38)
        at org.apache.kyuubi.service.authentication.TSetIpAddressProcessor.process(TSetIpAddressProcessor.scala:35)
        at org.apache.kyuubi.shaded.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:250)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:750)
Caused by: java.lang.NoSuchMethodException: Cannot find method: getInstance
        at org.apache.kyuubi.util.reflect.DynMethods$Builder.buildChecked(DynMethods.java:473)
        at org.apache.kyuubi.util.reflect.DynMethods$Builder.buildStaticChecked(DynMethods.java:500)
        at org.apache.kyuubi.engine.flink.operation.ExecuteStatement.getPlannerModuleClassLoader(ExecuteStatement.scala:105)
        ... 19 more (state=,code=0)
@danny0405
Copy link
Contributor

Does the datastream api support sort now? Maybe we can support that in the future.

@ysmintor
Copy link
Author

Does the datastream api support sort now? Maybe we can support that in the future.

Yeah. Hope someone can check it, and later hudi can support flink-table-planner_2.12.

@github-project-automation github-project-automation bot moved this to ⏳ Awaiting Triage in Hudi Issue Support Apr 1, 2024
@codope codope added feature-enquiry issue contains feature enquiries/requests or great improvement ideas flink Issues related to flink labels Apr 1, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
feature-enquiry issue contains feature enquiries/requests or great improvement ideas flink Issues related to flink
Projects
Hudi Issue Support
Status: ⏳ Awaiting Triage
Milestone
No milestone
Development

No branches or pull requests
3 participants
@ysmintor @danny0405 @codope