Uploaded image for project: 'Hadoop YARN'
  1. Hadoop YARN
  2. YARN-1219

FSDownload changes file suffix making FileUtil.unTar() throw exception

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 2.1.1-beta, 2.2.0, 3.0.0-alpha1
    • Fix Version/s: 2.2.0
    • Component/s: nodemanager
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      While running a Hive join operation on Yarn, I saw exception as described below. This is caused by FSDownload copy the files into a temp file and change the suffix into ".tmp" before unpacking it. In unpack(), it uses FileUtil.unTar() which will determine if the file is "gzipped" by looking at the file suffix:

      boolean gzipped = inFile.toString().endsWith("gz");
      

      To fix this problem, we can remove the ".tmp" in the temp file name.

      Here is the detailed exception:

      org.apache.commons.compress.archivers.tar.TarArchiveInputStream.getNextTarEntry(TarArchiveInputStream.java:240)
      at org.apache.hadoop.fs.FileUtil.unTarUsingJava(FileUtil.java:676)
      at org.apache.hadoop.fs.FileUtil.unTar(FileUtil.java:625)
      at org.apache.hadoop.yarn.util.FSDownload.unpack(FSDownload.java:203)
      at org.apache.hadoop.yarn.util.FSDownload.call(FSDownload.java:287)
      at org.apache.hadoop.yarn.util.FSDownload.call(FSDownload.java:50)
      at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
      at java.util.concurrent.FutureTask.run(FutureTask.java:166)
      at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
      at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
      at java.util.concurrent.FutureTask.run(FutureTask.java:166)
      at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
      at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)

      at java.lang.Thread.run(Thread.java:722)

        Attachments

        1. YARN-1219.patch
          13 kB
          shanyu zhao

          Activity

            People

            • Assignee:
              shanyu shanyu zhao
              Reporter:
              shanyu shanyu zhao
            • Votes:
              0 Vote for this issue
              Watchers:
              7 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: