Uploaded image for project: 'Flink'
  1. Flink
  2. FLINK-9258

ConcurrentModificationException in ComponentMetricGroup.getAllVariables

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 1.4.2
    • Fix Version/s: 1.4.3, 1.5.1, 1.6.0
    • Component/s: Runtime / Metrics
    • Labels:
      None

      Description

      Seeing this exception at the job startup time. Looks like there is a race condition when the metrics variables are constructed.

      The error is intermittent.

      Exception in thread "main" org.apache.flink.runtime.client.JobExecutionException: Job execution failed.

             at org.apache.flink.runtime.jobmanager.JobManager$$anonfun$handleMessage$1$$anonfun$applyOrElse$6.apply$mcV$sp(JobManager.scala:897)

             at org.apache.flink.runtime.jobmanager.JobManager$$anonfun$handleMessage$1$$anonfun$applyOrElse$6.apply(JobManager.scala:840)

             at org.apache.flink.runtime.jobmanager.JobManager$$anonfun$handleMessage$1$$anonfun$applyOrElse$6.apply(JobManager.scala:840)

             at scala.concurrent.impl.Future$PromiseCompletingRunnable.liftedTree1$1(Future.scala:24)

             at scala.concurrent.impl.Future$PromiseCompletingRunnable.run(Future.scala:24)

             at akka.dispatch.TaskInvocation.run(AbstractDispatcher.scala:39)

             at akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(AbstractDispatcher.scala:415)

             at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)

             at scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)

             at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)

             at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)

      Caused by: java.util.ConcurrentModificationException

             at java.util.HashMap$HashIterator.nextNode(HashMap.java:1437)

             at java.util.HashMap$EntryIterator.next(HashMap.java:1471)

             at java.util.HashMap$EntryIterator.next(HashMap.java:1469)

             at java.util.HashMap.putMapEntries(HashMap.java:511)

             at java.util.HashMap.putAll(HashMap.java:784)

             at org.apache.flink.runtime.metrics.groups.ComponentMetricGroup.getAllVariables(ComponentMetricGroup.java:63)

             at org.apache.flink.runtime.metrics.groups.ComponentMetricGroup.getAllVariables(ComponentMetricGroup.java:63)

             at com.netflix.spaas.metrics.MetricsReporterRegistry.getTags(MetricsReporterRegistry.java:147)

             at com.netflix.spaas.metrics.MetricsReporterRegistry.mergeWithSourceAndSinkTags(MetricsReporterRegistry.java:170)

             at com.netflix.spaas.metrics.MetricsReporterRegistry.addReporter(MetricsReporterRegistry.java:75)

             at com.netflix.spaas.nfflink.connector.kafka.source.Kafka010Consumer.createFetcher(Kafka010Consumer.java:69)

             at org.apache.flink.streaming.connectors.kafka.FlinkKafkaConsumerBase.run(FlinkKafkaConsumerBase.java:549)

             at org.apache.flink.streaming.api.operators.StreamSource.run(StreamSource.java:86)

             at org.apache.flink.streaming.api.operators.StreamSource.run(StreamSource.java:55)

             at org.apache.flink.streaming.runtime.tasks.SourceStreamTask.run(SourceStreamTask.java:94)

             at org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:264)

             at org.apache.flink.runtime.taskmanager.Task.run(Task.java:718)

             at java.lang.Thread.run(Thread.java:748)

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                chesnay Chesnay Schepler
                Reporter:
                narayaruna Narayanan Arunachalam
              • Votes:
                0 Vote for this issue
                Watchers:
                3 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: