Uploaded image for project: 'Hadoop Map/Reduce'
  1. Hadoop Map/Reduce
  2. MAPREDUCE-6766

Concurrent local job failures due to uniqueNumberGenerator = new AtomicLong(System.currentTimeMillis())

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Open
    • Priority: Major
    • Resolution: Unresolved
    • Affects Version/s: 2.3.0, 2.7.1
    • Fix Version/s: None
    • Component/s: None
    • Labels:
      None
    • Environment:

      Druid 0.8.3

      Description

      I am seeing the following exception when attempting to execute multiple Hadoop Local Jobs with Druid.

      java.io.IOException: Rename cannot overwrite non empty destination directory /tmp/hadoop-username/mapred/local/1472019105135
      

      From a quick look at the Hadoop code base, it seems that the uniqueNumberGenerator for the LocalDistributedCacheManager is based on the System time, and this appears to cause problems for concurrent jobs.

      // Generating unique numbers for FSDownload.
          AtomicLong uniqueNumberGenerator =
      new AtomicLong(System.currentTimeMillis());
      

      I am pretty sure the following line of code is responsible, and this seems to exist in latter versions of such as 2.7.1:

      Full Stack Trace

      java.lang.RuntimeException: java.lang.reflect.InvocationTargetException
      	at com.google.common.base.Throwables.propagate(Throwables.java:160) ~[guava-16.0.1.jar:?]
      	at io.druid.indexing.common.task.HadoopTask.invokeForeignLoader(HadoopTask.java:138) ~[druid-indexing-service-0.8.3.jar:0.8.3]
      	at io.druid.indexing.common.task.HadoopIndexTask.run(HadoopIndexTask.java:206) ~[druid-indexing-service-0.8.3.jar:0.8.3]
      	at io.druid.indexing.overlord.ThreadPoolTaskRunner$ThreadPoolTaskRunnerCallable.call(ThreadPoolTaskRunner.java:285) [druid-indexing-service-0.8.3.jar:0.8.3]
      	at io.druid.indexing.overlord.ThreadPoolTaskRunner$ThreadPoolTaskRunnerCallable.call(ThreadPoolTaskRunner.java:265) [druid-indexing-service-0.8.3.jar:0.8.3]
      	at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:1.8.0_71]
      	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [?:1.8.0_71]
      	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [?:1.8.0_71]
      	at java.lang.Thread.run(Thread.java:745) [?:1.8.0_71]
      Caused by: java.lang.reflect.InvocationTargetException
      	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.8.0_71]
      	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:1.8.0_71]
      	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_71]
      	at java.lang.reflect.Method.invoke(Method.java:497) ~[?:1.8.0_71]
      	at io.druid.indexing.common.task.HadoopTask.invokeForeignLoader(HadoopTask.java:135) ~[druid-indexing-service-0.8.3.jar:0.8.3]
      	... 7 more
      Caused by: java.lang.RuntimeException: java.io.IOException: java.util.concurrent.ExecutionException: java.io.IOException: Rename cannot overwrite non empty destination directory /tmp/hadoop-username/mapred/local/1472019105135
      	at io.druid.indexer.IndexGeneratorJob.run(IndexGeneratorJob.java:211) ~[druid-indexing-hadoop-0.8.3.jar:0.8.3]
      	at io.druid.indexer.JobHelper.runJobs(JobHelper.java:321) ~[druid-indexing-hadoop-0.8.3.jar:0.8.3]
      	at io.druid.indexer.HadoopDruidIndexerJob.run(HadoopDruidIndexerJob.java:96) ~[druid-indexing-hadoop-0.8.3.jar:0.8.3]
      	at io.druid.indexing.common.task.HadoopIndexTask$HadoopIndexGeneratorInnerProcessing.runTask(HadoopIndexTask.java:259) ~[druid-indexing-service-0.8.3.jar:0.8.3]
      	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.8.0_71]
      	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:1.8.0_71]
      	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_71]
      	at java.lang.reflect.Method.invoke(Method.java:497) ~[?:1.8.0_71]
      	at io.druid.indexing.common.task.HadoopTask.invokeForeignLoader(HadoopTask.java:135) ~[druid-indexing-service-0.8.3.jar:0.8.3]
      	... 7 more
      Caused by: java.io.IOException: java.util.concurrent.ExecutionException: java.io.IOException: Rename cannot overwrite non empty destination directory /tmp/hadoop-username/mapred/local/1472019105135
      	at org.apache.hadoop.mapred.LocalDistributedCacheManager.setup(LocalDistributedCacheManager.java:143) ~[?:?]
      	at org.apache.hadoop.mapred.LocalJobRunner$Job.<init>(LocalJobRunner.java:163) ~[?:?]
      	at org.apache.hadoop.mapred.LocalJobRunner.submitJob(LocalJobRunner.java:731) ~[?:?]
      	at org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:240) ~[?:?]
      	at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1290) ~[?:?]
      	at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1287) ~[?:?]
      	at java.security.AccessController.doPrivileged(Native Method) ~[?:1.8.0_71]
      	at javax.security.auth.Subject.doAs(Subject.java:422) ~[?:1.8.0_71]
      	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657) ~[?:?]
      	at org.apache.hadoop.mapreduce.Job.submit(Job.java:1287) ~[?:?]
      	at io.druid.indexer.IndexGeneratorJob.run(IndexGeneratorJob.java:199) ~[druid-indexing-hadoop-0.8.3.jar:0.8.3]
      	at io.druid.indexer.JobHelper.runJobs(JobHelper.java:321) ~[druid-indexing-hadoop-0.8.3.jar:0.8.3]
      	at io.druid.indexer.HadoopDruidIndexerJob.run(HadoopDruidIndexerJob.java:96) ~[druid-indexing-hadoop-0.8.3.jar:0.8.3]
      	at io.druid.indexing.common.task.HadoopIndexTask$HadoopIndexGeneratorInnerProcessing.runTask(HadoopIndexTask.java:259) ~[druid-indexing-service-0.8.3.jar:0.8.3]
      	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.8.0_71]
      	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:1.8.0_71]
      	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_71]
      	at java.lang.reflect.Method.invoke(Method.java:497) ~[?:1.8.0_71]
      	at io.druid.indexing.common.task.HadoopTask.invokeForeignLoader(HadoopTask.java:135) ~[druid-indexing-service-0.8.3.jar:0.8.3]
      	... 7 more
      Caused by: java.util.concurrent.ExecutionException: java.io.IOException: Rename cannot overwrite non empty destination directory /tmp/hadoop-username/mapred/local/1472019105135
      	at java.util.concurrent.FutureTask.report(FutureTask.java:122) ~[?:1.8.0_71]
      	at java.util.concurrent.FutureTask.get(FutureTask.java:192) ~[?:1.8.0_71]
      	at org.apache.hadoop.mapred.LocalDistributedCacheManager.setup(LocalDistributedCacheManager.java:139) ~[?:?]
      	at org.apache.hadoop.mapred.LocalJobRunner$Job.<init>(LocalJobRunner.java:163) ~[?:?]
      	at org.apache.hadoop.mapred.LocalJobRunner.submitJob(LocalJobRunner.java:731) ~[?:?]
      	at org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:240) ~[?:?]
      	at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1290) ~[?:?]
      	at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1287) ~[?:?]
      	at java.security.AccessController.doPrivileged(Native Method) ~[?:1.8.0_71]
      	at javax.security.auth.Subject.doAs(Subject.java:422) ~[?:1.8.0_71]
      	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657) ~[?:?]
      	at org.apache.hadoop.mapreduce.Job.submit(Job.java:1287) ~[?:?]
      	at io.druid.indexer.IndexGeneratorJob.run(IndexGeneratorJob.java:199) ~[druid-indexing-hadoop-0.8.3.jar:0.8.3]
      	at io.druid.indexer.JobHelper.runJobs(JobHelper.java:321) ~[druid-indexing-hadoop-0.8.3.jar:0.8.3]
      	at io.druid.indexer.HadoopDruidIndexerJob.run(HadoopDruidIndexerJob.java:96) ~[druid-indexing-hadoop-0.8.3.jar:0.8.3]
      	at io.druid.indexing.common.task.HadoopIndexTask$HadoopIndexGeneratorInnerProcessing.runTask(HadoopIndexTask.java:259) ~[druid-indexing-service-0.8.3.jar:0.8.3]
      	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.8.0_71]
      	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:1.8.0_71]
      	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_71]
      	at java.lang.reflect.Method.invoke(Method.java:497) ~[?:1.8.0_71]
      	at io.druid.indexing.common.task.HadoopTask.invokeForeignLoader(HadoopTask.java:135) ~[druid-indexing-service-0.8.3.jar:0.8.3]
      	... 7 more
      Caused by: java.io.IOException: Rename cannot overwrite non empty destination directory /tmp/hadoop-username/mapred/local/1472019105135
      	at org.apache.hadoop.fs.AbstractFileSystem.renameInternal(AbstractFileSystem.java:735) ~[?:?]
      	at org.apache.hadoop.fs.FilterFs.renameInternal(FilterFs.java:236) ~[?:?]
      	at org.apache.hadoop.fs.AbstractFileSystem.rename(AbstractFileSystem.java:678) ~[?:?]
      	at org.apache.hadoop.fs.FileContext.rename(FileContext.java:958) ~[?:?]
      	at org.apache.hadoop.yarn.util.FSDownload.call(FSDownload.java:366) ~[?:?]
      	at org.apache.hadoop.yarn.util.FSDownload.call(FSDownload.java:62) ~[?:?]
      	... 4 more
      

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                Unassigned
                Reporter:
                marks1900 Mark S
              • Votes:
                0 Vote for this issue
                Watchers:
                3 Start watching this issue

                Dates

                • Created:
                  Updated: