[BEAM-3964] Java Github HEAD doesn't run on Dataflow

• Type: Bug
• Status: Resolved
• Priority: P2
• Resolution: Fixed
• Fix Version/s: Not applicable
• Component/s: runner-dataflow
• Assignee: Thomas Groh
• Reporter: Thomas Groh
• Created: 28/Mar/18 18:01
• Updated: 16/May/20 13:55
• Resolved: 02/Apr/18 23:16

Description

Rather: it runs, but cannot make progress, seemingly due to an Error with metric names:

 

exception: "java.lang.NoSuchMethodError: org.apache.beam.sdk.metrics.MetricName.name()Ljava/lang/String; at com.google.cloud.dataflow.worker.MetricsToCounterUpdateConverter.structuredNameAndMetadata(MetricsToCounterUpdateConverter.java:99) at com.google.cloud.dataflow.worker.MetricsToCounterUpdateConverter.fromCounter(MetricsToCounterUpdateConverter.java:68) at com.google.cloud.dataflow.worker.BatchModeExecutionContext.lambda$null$1(BatchModeExecutionContext.java:463) at com.google.cloud.dataflow.worker.repackaged.com.google.common.collect.Iterators$7.transform(Iterators.java:750) at com.google.cloud.dataflow.worker.repackaged.com.google.common.collect.TransformedIterator.next(TransformedIterator.java:47) at com.google.cloud.dataflow.worker.repackaged.com.google.common.collect.MultitransformedIterator.next(MultitransformedIterator.java:66) at com.google.cloud.dataflow.worker.repackaged.com.google.common.collect.ImmutableCollection$Builder.addAll(ImmutableCollection.java:388) at com.google.cloud.dataflow.worker.repackaged.com.google.common.collect.ImmutableCollection$ArrayBasedBuilder.addAll(ImmutableCollection.java:472) at com.google.cloud.dataflow.worker.repackaged.com.google.common.collect.ImmutableList$Builder.addAll(ImmutableList.java:669) at com.google.cloud.dataflow.worker.WorkItemStatusClient.populateCounterUpdates(WorkItemStatusClient.java:256) at com.google.cloud.dataflow.worker.WorkItemStatusClient.createStatusUpdate(WorkItemStatusClient.java:240) at com.google.cloud.dataflow.worker.WorkItemStatusClient.reportError(WorkItemStatusClient.java:94) at com.google.cloud.dataflow.worker.BatchDataflowWorker.doWork(BatchDataflowWorker.java:358) at com.google.cloud.dataflow.worker.BatchDataflowWorker.getAndPerformWork(BatchDataflowWorker.java:284) at com.google.cloud.dataflow.worker.DataflowBatchWorkerHarness$WorkerThread.doWork(DataflowBatchWorkerHarness.java:134) at com.google.cloud.dataflow.worker.DataflowBatchWorkerHarness$WorkerThread.call(DataflowBatchWorkerHarness.java:114) at com.google.cloud.dataflow.worker.DataflowBatchWorkerHarness$WorkerThread.call(DataflowBatchWorkerHarness.java:101) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745)

 

Probably requires an updated container version.

Issue Links

• blocks: BEAM-3942 Update performance testing framework to use Gradle.
• is duplicated by: BEAM-3963 Dataflow PostCommits have not built since Mar 26, 2018
• links to: GitHub Pull Request #4971
• links to: GitHub Pull Request #4981

Sub-Tasks

• Rebuild the Dataflow worker to use new MetricQueryResults API: Resolved