Uploaded image for project: 'Hive'
  1. Hive
  2. HIVE-16395

ConcurrentModificationException on config object in HoS

    XMLWordPrintableJSON

    Details

    • Type: Task
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 3.0.0
    • Component/s: Spark
    • Labels:
      None

      Description

      Looks like this is happening inside spark executors, looks to be some race condition when modifying Configuration objects.

      Stack-Trace:

      java.io.IOException: java.lang.reflect.InvocationTargetException
      	at org.apache.hadoop.hive.io.HiveIOExceptionHandlerChain.handleRecordReaderCreationException(HiveIOExceptionHandlerChain.java:97)
      	at org.apache.hadoop.hive.io.HiveIOExceptionHandlerUtil.handleRecordReaderCreationException(HiveIOExceptionHandlerUtil.java:57)
      	at org.apache.hadoop.hive.shims.HadoopShimsSecure$CombineFileRecordReader.initNextRecordReader(HadoopShimsSecure.java:267)
      	at org.apache.hadoop.hive.shims.HadoopShimsSecure$CombineFileRecordReader.<init>(HadoopShimsSecure.java:213)
      	at org.apache.hadoop.hive.shims.HadoopShimsSecure$CombineFileInputFormatShim.getRecordReader(HadoopShimsSecure.java:334)
      	at org.apache.hadoop.hive.ql.io.CombineHiveInputFormat.getRecordReader(CombineHiveInputFormat.java:682)
      	at org.apache.spark.rdd.HadoopRDD$$anon$1.<init>(HadoopRDD.scala:240)
      	at org.apache.spark.rdd.HadoopRDD.compute(HadoopRDD.scala:211)
      	at org.apache.spark.rdd.HadoopRDD.compute(HadoopRDD.scala:101)
      	at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:306)
      	at org.apache.spark.rdd.RDD.iterator(RDD.scala:270)
      	at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)
      	at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:306)
      	at org.apache.spark.rdd.RDD.iterator(RDD.scala:270)
      	at org.apache.spark.rdd.UnionRDD.compute(UnionRDD.scala:87)
      	at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:306)
      	at org.apache.spark.rdd.RDD.iterator(RDD.scala:270)
      	at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:73)
      	at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:41)
      	at org.apache.spark.scheduler.Task.run(Task.scala:89)
      	at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:242)
      	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
      	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
      	at java.lang.Thread.run(Thread.java:745)
      Caused by: java.lang.reflect.InvocationTargetException
      	at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
      	at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
      	at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
      	at java.lang.reflect.Constructor.newInstance(Constructor.java:526)
      	at org.apache.hadoop.hive.shims.HadoopShimsSecure$CombineFileRecordReader.initNextRecordReader(HadoopShimsSecure.java:253)
      	... 21 more
      Caused by: java.util.ConcurrentModificationException
      	at java.util.Hashtable$Enumerator.next(Hashtable.java:1167)
      	at org.apache.hadoop.conf.Configuration.iterator(Configuration.java:2455)
      	at org.apache.hadoop.fs.s3a.S3AUtils.propagateBucketOptions(S3AUtils.java:716)
      	at org.apache.hadoop.fs.s3a.S3AFileSystem.initialize(S3AFileSystem.java:181)
      	at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:2815)
      	at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:98)
      	at org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:2852)
      	at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:2834)
      	at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:387)
      	at org.apache.hadoop.fs.Path.getFileSystem(Path.java:296)
      	at org.apache.hadoop.mapred.LineRecordReader.<init>(LineRecordReader.java:108)
      	at org.apache.hadoop.mapred.TextInputFormat.getRecordReader(TextInputFormat.java:67)
      	at org.apache.hadoop.hive.ql.io.CombineHiveRecordReader.<init>(CombineHiveRecordReader.java:68)
      	... 26 more
      

        Attachments

        1. HIVE-16395.2.patch
          7 kB
          Andrew Sherman
        2. HIVE-16395.1.patch
          7 kB
          Andrew Sherman

          Activity

            People

            • Assignee:
              asherman Andrew Sherman
              Reporter:
              stakiar Sahil Takiar
            • Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: