Uploaded image for project: 'Hadoop Map/Reduce'
  1. Hadoop Map/Reduce
  2. MAPREDUCE-7325

Intermediate data encryption is broken in LocalJobRunner

    XMLWordPrintableJSON

Details

    Description

      With enabling intermediate encryption, running a job using LocalJobRunner with multiple reducers fails with the following error:

       

      2021-02-23 18:18:05,145 WARN  [Thread-2381] mapred.LocalJobRunner (LocalJobRunner.java:run(590)) - job_local1344328673_0004
      java.lang.Exception: org.apache.hadoop.mapreduce.task.reduce.Shuffle$ShuffleError: error in shuffle in localfetcher#5
      	at org.apache.hadoop.mapred.LocalJobRunner$Job.runTasks(LocalJobRunner.java:492)
      	at org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:559)
      Caused by: org.apache.hadoop.mapreduce.task.reduce.Shuffle$ShuffleError: error in shuffle in localfetcher#5
      	at org.apache.hadoop.mapreduce.task.reduce.Shuffle.run(Shuffle.java:136)
      	at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:377)
      	at org.apache.hadoop.mapred.LocalJobRunner$Job$ReduceTaskRunnable.run(LocalJobRunner.java:347)
      	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
      	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
      	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
      	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
      	at java.lang.Thread.run(Thread.java:748)
      Caused by: org.apache.hadoop.fs.ChecksumException: Checksum Error
      	at org.apache.hadoop.mapred.IFileInputStream.doRead(IFileInputStream.java:229)
      	at org.apache.hadoop.mapred.IFileInputStream.read(IFileInputStream.java:153)
      	at org.apache.hadoop.io.IOUtils.readFully(IOUtils.java:210)
      	at org.apache.hadoop.mapreduce.task.reduce.InMemoryMapOutput.doShuffle(InMemoryMapOutput.java:91)
      	at org.apache.hadoop.mapreduce.task.reduce.IFileWrappedMapOutput.shuffle(IFileWrappedMapOutput.java:63)
      	at org.apache.hadoop.mapreduce.task.reduce.LocalFetcher.copyMapOutput(LocalFetcher.java:156)
      	at org.apache.hadoop.mapreduce.task.reduce.LocalFetcher.doCopy(LocalFetcher.java:103)
      	at org.apache.hadoop.mapreduce.task.reduce.LocalFetcher.run(LocalFetcher.java:86)
      

      The bug can be reproduced with any test unit like TestLocalJobSubmission.

      Another way to reproduce the bug is to run LargeSorter with multiple reducers.

      Configuration config = new Configuration();
      // set all the necessary configurations
      config.setInt(LargeSorter.NUM_REDUCE_TASKS,2);
      String[] args = new String[] {"output-dir"};
      int res = ToolRunner.run(config, new LargeSorter(), args);
      

      Attachments

        1. MAPREDUCE-7325.001.patch
          17 kB
          Ahmed Hussein
        2. MAPREDUCE-7325.002.patch
          17 kB
          Ahmed Hussein

        Issue Links

          Activity

            People

              ahussein Ahmed Hussein
              ahussein Ahmed Hussein
              Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: