Uploaded image for project: 'Hadoop Common'
  1. Hadoop Common
  2. HADOOP-10456

Bug in Configuration.java exposed by Spark (ConcurrentModificationException)

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 2.3.0
    • Fix Version/s: 2.4.1
    • Component/s: conf
    • Labels:
      None
    • Target Version/s:
    • Hadoop Flags:
      Reviewed

      Description

      The following exception occurs non-deterministically:
      java.util.ConcurrentModificationException
      at java.util.HashMap$HashIterator.nextEntry(HashMap.java:926)
      at java.util.HashMap$KeyIterator.next(HashMap.java:960)
      at java.util.AbstractCollection.addAll(AbstractCollection.java:341)
      at java.util.HashSet.<init>(HashSet.java:117)
      at org.apache.hadoop.conf.Configuration.<init>(Configuration.java:671)
      at org.apache.hadoop.mapred.JobConf.<init>(JobConf.java:439)
      at org.apache.spark.rdd.HadoopRDD.getJobConf(HadoopRDD.scala:110)
      at org.apache.spark.rdd.HadoopRDD$$anon$1.<init>(HadoopRDD.scala:154)
      at org.apache.spark.rdd.HadoopRDD.compute(HadoopRDD.scala:149)
      at org.apache.spark.rdd.HadoopRDD.compute(HadoopRDD.scala:64)
      at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:241)
      at org.apache.spark.rdd.RDD.iterator(RDD.scala:232)
      at org.apache.spark.rdd.MappedRDD.compute(MappedRDD.scala:31)
      at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:241)
      at org.apache.spark.rdd.RDD.iterator(RDD.scala:232)
      at org.apache.spark.rdd.MappedRDD.compute(MappedRDD.scala:31)
      at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:241)
      at org.apache.spark.rdd.RDD.iterator(RDD.scala:232)
      at org.apache.spark.rdd.MappedRDD.compute(MappedRDD.scala:31)
      at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:241)
      at org.apache.spark.rdd.RDD.iterator(RDD.scala:232)
      at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:34)
      at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:241)
      at org.apache.spark.rdd.RDD.iterator(RDD.scala:232)
      at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:161)
      at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:102)
      at org.apache.spark.scheduler.Task.run(Task.scala:53)
      at org.apache.spark.executor.Executor$TaskRunner$$anonfun$run$1.apply$mcV$sp(Executor.scala:213)
      at org.apache.spark.deploy.SparkHadoopUtil$$anon$1.run(SparkHadoopUtil.scala:42)
      at org.apache.spark.deploy.SparkHadoopUtil$$anon$1.run(SparkHadoopUtil.scala:41)
      at java.security.AccessController.doPrivileged(Native Method)
      at javax.security.auth.Subject.doAs(Subject.java:415)
      at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1548)
      at org.apache.spark.deploy.SparkHadoopUtil.runAsUser(SparkHadoopUtil.scala:41)
      at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:178)
      at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
      at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
      at java.lang.Thread.run(Thread.java:744)

        Attachments

        1. HADOOP-10456_nravi.patch
          0.8 kB
          Nishkam Ravi

          Issue Links

            Activity

              People

              • Assignee:
                nravi Nishkam Ravi
                Reporter:
                nravi Nishkam Ravi
              • Votes:
                0 Vote for this issue
                Watchers:
                6 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: