[FLINK-33531] Nightly Python fails with NPE at metadataHandlerProvider on AZP (StreamDependencyTests.test_add_python_archive)

• Type: Technical Debt
• Status: Resolved
• Priority: Blocker
• Resolution: Fixed
• Affects Version/s: 1.18.0, 1.19.0
• Fix Version/s: 1.19.0, 1.18.1
• Component/s: API / Python
• Labels: pull-request-available test-stability
• Assignee: Xingbo Huang
• Reporter: Sergey Nuyanzin
• Created: 13/Nov/23 00:54
• Updated: 20/Dec/23 17:54
• Resolved: 20/Nov/23 11:23

Description

It seems starting 02.11.2023 every master nightly fails with this (that's why it is a blocker)

for instance
https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=54512&view=logs&j=9cada3cb-c1d3-5621-16da-0f718fb86602&t=c67e71ed-6451-5d26-8920-5a8cf9651901

2023-11-12T02:10:24.5082784Z Nov 12 02:10:24         if is_error(answer)[0]:
2023-11-12T02:10:24.5083620Z Nov 12 02:10:24             if len(answer) > 1:
2023-11-12T02:10:24.5084326Z Nov 12 02:10:24                 type = answer[1]
2023-11-12T02:10:24.5085164Z Nov 12 02:10:24                 value = OUTPUT_CONVERTER[type](answer[2:], gateway_client)
2023-11-12T02:10:24.5086061Z Nov 12 02:10:24                 if answer[1] == REFERENCE_TYPE:
2023-11-12T02:10:24.5086850Z Nov 12 02:10:24 >                   raise Py4JJavaError(
2023-11-12T02:10:24.5087677Z Nov 12 02:10:24                         "An error occurred while calling {0}{1}{2}.\n".
2023-11-12T02:10:24.5088538Z Nov 12 02:10:24                         format(target_id, ".", name), value)
2023-11-12T02:10:24.5089551Z Nov 12 02:10:24 E                   py4j.protocol.Py4JJavaError: An error occurred while calling o3371.executeInsert.
2023-11-12T02:10:24.5090832Z Nov 12 02:10:24 E                   : java.lang.NullPointerException: metadataHandlerProvider
2023-11-12T02:10:24.5091832Z Nov 12 02:10:24 E                   	at java.util.Objects.requireNonNull(Objects.java:228)
2023-11-12T02:10:24.5093399Z Nov 12 02:10:24 E                   	at org.apache.calcite.rel.metadata.RelMetadataQueryBase.getMetadataHandlerProvider(RelMetadataQueryBase.java:122)
2023-11-12T02:10:24.5094480Z Nov 12 02:10:24 E                   	at org.apache.calcite.rel.metadata.RelMetadataQueryBase.revise(RelMetadataQueryBase.java:118)
2023-11-12T02:10:24.5095365Z Nov 12 02:10:24 E                   	at org.apache.calcite.rel.metadata.RelMetadataQuery.getPulledUpPredicates(RelMetadataQuery.java:844)
2023-11-12T02:10:24.5096306Z Nov 12 02:10:24 E                   	at org.apache.calcite.rel.rules.ReduceExpressionsRule$ProjectReduceExpressionsRule.onMatch(ReduceExpressionsRule.java:307)
2023-11-12T02:10:24.5097238Z Nov 12 02:10:24 E                   	at org.apache.calcite.plan.AbstractRelOptPlanner.fireRule(AbstractRelOptPlanner.java:337)
2023-11-12T02:10:24.5098014Z Nov 12 02:10:24 E                   	at org.apache.calcite.plan.hep.HepPlanner.applyRule(HepPlanner.java:556)
2023-11-12T02:10:24.5098753Z Nov 12 02:10:24 E                   	at org.apache.calcite.plan.hep.HepPlanner.applyRules(HepPlanner.java:420)
2023-11-12T02:10:24.5099517Z Nov 12 02:10:24 E                   	at org.apache.calcite.plan.hep.HepPlanner.executeRuleInstance(HepPlanner.java:243)
2023-11-12T02:10:24.5100373Z Nov 12 02:10:24 E                   	at org.apache.calcite.plan.hep.HepInstruction$RuleInstance$State.execute(HepInstruction.java:178)
2023-11-12T02:10:24.5101313Z Nov 12 02:10:24 E                   	at org.apache.calcite.plan.hep.HepPlanner.lambda$executeProgram$0(HepPlanner.java:211)
2023-11-12T02:10:24.5102410Z Nov 12 02:10:24 E                   	at org.apache.flink.calcite.shaded.com.google.common.collect.ImmutableList.forEach(ImmutableList.java:422)
2023-11-12T02:10:24.5103343Z Nov 12 02:10:24 E                   	at org.apache.calcite.plan.hep.HepPlanner.executeProgram(HepPlanner.java:210)
2023-11-12T02:10:24.5104105Z Nov 12 02:10:24 E                   	at org.apache.calcite.plan.hep.HepProgram$State.execute(HepProgram.java:118)
2023-11-12T02:10:24.5104868Z Nov 12 02:10:24 E                   	at org.apache.calcite.plan.hep.HepPlanner.executeProgram(HepPlanner.java:205)
2023-11-12T02:10:24.5105616Z Nov 12 02:10:24 E                   	at org.apache.calcite.plan.hep.HepPlanner.findBestExp(HepPlanner.java:191)
2023-11-12T02:10:24.5106421Z Nov 12 02:10:24 E                   	at org.apache.flink.table.planner.plan.optimize.program.FlinkHepProgram.optimize(FlinkHepProgram.scala:64)
2023-11-12T02:10:24.5107359Z Nov 12 02:10:24 E                   	at org.apache.flink.table.planner.plan.optimize.program.FlinkHepRuleSetProgram.optimize(FlinkHepRuleSetProgram.scala:78)
2023-11-12T02:10:24.5108346Z Nov 12 02:10:24 E                   	at org.apache.flink.table.planner.plan.optimize.program.FlinkChainedProgram.$anonfun$optimize$1(FlinkChainedProgram.scala:59)
2023-11-12T02:10:24.5109407Z Nov 12 02:10:24 E                   	at scala.collection.TraversableOnce.$anonfun$foldLeft$1(TraversableOnce.scala:156)
2023-11-12T02:10:24.5110241Z Nov 12 02:10:24 E                   	at scala.collection.TraversableOnce.$anonfun$foldLeft$1$adapted(TraversableOnce.scala:156)
2023-11-12T02:10:24.5111078Z Nov 12 02:10:24 E                   	at scala.collection.Iterator.foreach(Iterator.scala:937)
2023-11-12T02:10:24.5111734Z Nov 12 02:10:24 E                   	at scala.collection.Iterator.foreach$(Iterator.scala:937)
2023-11-12T02:10:24.5112410Z Nov 12 02:10:24 E                   	at scala.collection.AbstractIterator.foreach(Iterator.scala:1425)
2023-11-12T02:10:24.5113145Z Nov 12 02:10:24 E                   	at scala.collection.IterableLike.foreach(IterableLike.scala:70)
2023-11-12T02:10:24.5113840Z Nov 12 02:10:24 E                   	at scala.collection.IterableLike.foreach$(IterableLike.scala:69)
2023-11-12T02:10:24.5114528Z Nov 12 02:10:24 E                   	at scala.collection.AbstractIterable.foreach(Iterable.scala:54)
2023-11-12T02:10:24.5115232Z Nov 12 02:10:24 E                   	at scala.collection.TraversableOnce.foldLeft(TraversableOnce.scala:156)
2023-11-12T02:10:24.5116054Z Nov 12 02:10:24 E                   	at scala.collection.TraversableOnce.foldLeft$(TraversableOnce.scala:154)
2023-11-12T02:10:24.5116775Z Nov 12 02:10:24 E                   	at scala.collection.AbstractTraversable.foldLeft(Traversable.scala:104)
2023-11-12T02:10:24.5117583Z Nov 12 02:10:24 E                   	at org.apache.flink.table.planner.plan.optimize.program.FlinkChainedProgram.optimize(FlinkChainedProgram.scala:55)
2023-11-12T02:10:24.5118581Z Nov 12 02:10:24 E                   	at org.apache.flink.table.planner.plan.optimize.StreamCommonSubGraphBasedOptimizer.optimizeTree(StreamCommonSubGraphBasedOptimizer.scala:176)
2023-11-12T02:10:24.5119603Z Nov 12 02:10:24 E                   	at org.apache.flink.table.planner.plan.optimize.StreamCommonSubGraphBasedOptimizer.doOptimize(StreamCommonSubGraphBasedOptimizer.scala:83)
2023-11-12T02:10:24.5120689Z Nov 12 02:10:24 E                   	at org.apache.flink.table.planner.plan.optimize.CommonSubGraphBasedOptimizer.optimize(CommonSubGraphBasedOptimizer.scala:87)
2023-11-12T02:10:24.5121602Z Nov 12 02:10:24 E                   	at org.apache.flink.table.planner.delegation.PlannerBase.optimize(PlannerBase.scala:324)
2023-11-12T02:10:24.5122403Z Nov 12 02:10:24 E                   	at org.apache.flink.table.planner.delegation.PlannerBase.translate(PlannerBase.scala:182)
2023-11-12T02:10:24.5123301Z Nov 12 02:10:24 E                   	at org.apache.flink.table.api.internal.TableEnvironmentImpl.translate(TableEnvironmentImpl.java:1277)
2023-11-12T02:10:24.5124195Z Nov 12 02:10:24 E                   	at org.apache.flink.table.api.internal.TableEnvironmentImpl.executeInternal(TableEnvironmentImpl.java:862)
2023-11-12T02:10:24.5125097Z Nov 12 02:10:24 E                   	at org.apache.flink.table.api.internal.TableEnvironmentImpl.executeInternal(TableEnvironmentImpl.java:1097)
2023-11-12T02:10:24.5125945Z Nov 12 02:10:24 E                   	at org.apache.flink.table.api.internal.TablePipelineImpl.execute(TablePipelineImpl.java:59)
2023-11-12T02:10:24.5126720Z Nov 12 02:10:24 E                   	at org.apache.flink.table.api.Table.executeInsert(Table.java:1074)

UPD:
It looks like it is a very similar to what has already been reported at FLINK-32424 and FLINK-24241
However something made it happening every nightly

Issue Links

• links to: GitHub Pull Request #23755