Uploaded image for project: 'Hadoop Map/Reduce'
  1. Hadoop Map/Reduce
  2. MAPREDUCE-6441

Improve temporary directory name generation in LocalDistributedCacheManager for concurrent processes

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 3.2.0, 3.1.4
    • Component/s: None
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      Kicking off many sqoop processes in different threads results in:

      2014-08-01 13:47:24 -0400:  INFO - 14/08/01 13:47:22 ERROR tool.ImportTool: Encountered IOException running import job: java.io.IOException: java.util.concurrent.ExecutionException: java.io.IOException: Rename cannot overwrite non empty destination directory /tmp/hadoop-hadoop/mapred/local/1406915233073
      
      2014-08-01 13:47:24 -0400:  INFO - 	at org.apache.hadoop.mapred.LocalDistributedCacheManager.setup(LocalDistributedCacheManager.java:149)
      
      2014-08-01 13:47:24 -0400:  INFO - 	at org.apache.hadoop.mapred.LocalJobRunner$Job.<init>(LocalJobRunner.java:163)
      
      2014-08-01 13:47:24 -0400:  INFO - 	at org.apache.hadoop.mapred.LocalJobRunner.submitJob(LocalJobRunner.java:731)
      
      2014-08-01 13:47:24 -0400:  INFO - 	at org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:432)
      
      2014-08-01 13:47:24 -0400:  INFO - 	at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1285)
      
      2014-08-01 13:47:24 -0400:  INFO - 	at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1282)
      
      2014-08-01 13:47:24 -0400:  INFO - 	at java.security.AccessController.doPrivileged(Native Method)
      
      2014-08-01 13:47:24 -0400:  INFO - 	at javax.security.auth.Subject.doAs(Subject.java:415)
      
      2014-08-01 13:47:24 -0400:  INFO - 	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1548)
      
      2014-08-01 13:47:24 -0400:  INFO - 	at org.apache.hadoop.mapreduce.Job.submit(Job.java:1282)
      
      2014-08-01 13:47:24 -0400:  INFO - 	at org.apache.hadoop.mapreduce.Job.waitForCompletion(Job.java:1303)
      
      2014-08-01 13:47:24 -0400:  INFO - 	at org.apache.sqoop.mapreduce.ImportJobBase.doSubmitJob(ImportJobBase.java:186)
      
      2014-08-01 13:47:24 -0400:  INFO - 	at org.apache.sqoop.mapreduce.ImportJobBase.runJob(ImportJobBase.java:159)
      
      2014-08-01 13:47:24 -0400:  INFO - 	at org.apache.sqoop.mapreduce.ImportJobBase.runImport(ImportJobBase.java:239)
      
      2014-08-01 13:47:24 -0400:  INFO - 	at org.apache.sqoop.manager.SqlManager.importQuery(SqlManager.java:645)
      
      2014-08-01 13:47:24 -0400:  INFO - 	at org.apache.sqoop.tool.ImportTool.importTable(ImportTool.java:415)
      
      2014-08-01 13:47:24 -0400:  INFO - 	at org.apache.sqoop.tool.ImportTool.run(ImportTool.java:502)
      
      2014-08-01 13:47:24 -0400:  INFO - 	at org.apache.sqoop.Sqoop.run(Sqoop.java:145)
      
      2014-08-01 13:47:24 -0400:  INFO - 	at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
      
      2014-08-01 13:47:24 -0400:  INFO - 	at org.apache.sqoop.Sqoop.runSqoop(Sqoop.java:181)
      
      2014-08-01 13:47:24 -0400:  INFO - 	at org.apache.sqoop.Sqoop.runTool(Sqoop.java:220)
      
      2014-08-01 13:47:24 -0400:  INFO - 	at org.apache.sqoop.Sqoop.runTool(Sqoop.java:229)
      
      2014-08-01 13:47:24 -0400:  INFO - 	at org.apache.sqoop.Sqoop.main(Sqoop.java:238)
      

      If two are kicked off in the same second. The issue is the following lines of code in the org.apache.hadoop.mapred.LocalDistributedCacheManager class:

          // Generating unique numbers for FSDownload.
          AtomicLong uniqueNumberGenerator =
             new AtomicLong(System.currentTimeMillis());
      

      and

      Long.toString(uniqueNumberGenerator.incrementAndGet())),
      

        Attachments

        1. MAPREDUCE-6441-branch-3.1.001.patch
          12 kB
          Peter Bacsko
        2. MAPREDUCE-6441.011.patch
          12 kB
          Haibo Chen
        3. MAPREDUCE-6441.010.patch
          12 kB
          Haibo Chen
        4. MAPREDUCE-6441.009.patch
          16 kB
          Ray Chiang
        5. MAPREDUCE-6441.008.patch
          16 kB
          Ray Chiang
        6. MAPREDUCE-6441.006.patch
          15 kB
          Ray Chiang
        7. MAPREDUCE-6441.005.patch
          15 kB
          Ray Chiang
        8. MAPREDUCE-6441.004.patch
          7 kB
          Ray Chiang
        9. HADOOP-10924.03.jobid-plus-uuid.patch
          9 kB
          William Watson
        10. HADOOP-10924.02.patch
          2 kB
          William Watson

          Issue Links

            Activity

              People

              • Assignee:
                haibochen Haibo Chen
                Reporter:
                wattsinabox William Watson
              • Votes:
                1 Vote for this issue
                Watchers:
                14 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: