Uploaded image for project: 'Hadoop Common'
  1. Hadoop Common
  2. HADOOP-5175

Option to prohibit jars unpacking

VotersWatch issueWatchersCreate sub-taskLinkCloneUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

    Details

    • Type: New Feature
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 0.19.0
    • Fix Version/s: 0.21.0
    • Component/s: None
    • Labels:
      None
    • Environment:

      Hadoop cluster of 5 servers, each with:
      HDD: two disks WDC WD1000FYPS-01ZKB0
      OS: Linux 2.6.26-1-686 #1 SMP
      FS: XFS

    • Hadoop Flags:
      Reviewed
    • Release Note:
      Jars passed to the -libjars option of hadoop jars are no longer unpacked inside mapred.local.dir.

      Description

      I've noticed that task tracker moves all unpacked jars into
      ${hadoop.tmp.dir}/mapred/local/taskTracker.

      We are using a lot of external libraries, that are deployed via "-libjars"
      option. The total number of files after unpacking is about 20 thousands.

      After running a number of jobs, tasks start to be killed with timeout reason
      ("Task attempt_200901281518_0011_m_000173_2 failed to report status for 601
      seconds. Killing!"). All killed tasks are in "initializing" state. I've
      watched the tasktracker logs and found such messages:

      Thread 20926 (Thread-10368):
      State: BLOCKED
      Blocked count: 3611
      Waited count: 24
      Blocked on java.lang.ref.Reference$Lock@e48ed6
      Blocked by 20882 (Thread-10341)
      Stack:
      java.lang.StringCoding$StringEncoder.encode(StringCoding.java:232)
      java.lang.StringCoding.encode(StringCoding.java:272)
      java.lang.String.getBytes(String.java:947)
      java.io.UnixFileSystem.getBooleanAttributes0(Native Method)
      java.io.UnixFileSystem.getBooleanAttributes(UnixFileSystem.java:228)
      java.io.File.isDirectory(File.java:754)
      org.apache.hadoop.fs.FileUtil.getDU(FileUtil.java:427)
      org.apache.hadoop.fs.FileUtil.getDU(FileUtil.java:433)
      org.apache.hadoop.fs.FileUtil.getDU(FileUtil.java:433)
      org.apache.hadoop.fs.FileUtil.getDU(FileUtil.java:433)
      org.apache.hadoop.fs.FileUtil.getDU(FileUtil.java:433)
      org.apache.hadoop.fs.FileUtil.getDU(FileUtil.java:433)
      org.apache.hadoop.fs.FileUtil.getDU(FileUtil.java:433)
      org.apache.hadoop.fs.FileUtil.getDU(FileUtil.java:433)
      org.apache.hadoop.fs.FileUtil.getDU(FileUtil.java:433)
      org.apache.hadoop.fs.FileUtil.getDU(FileUtil.java:433)
      org.apache.hadoop.fs.FileUtil.getDU(FileUtil.java:433)
      org.apache.hadoop.fs.FileUtil.getDU(FileUtil.java:433)
      org.apache.hadoop.fs.FileUtil.getDU(FileUtil.java:433)
      org.apache.hadoop.fs.FileUtil.getDU(FileUtil.java:433)

      HADOOP-4780 patch brings the code which stores map of directories along
      with their DU's, thus reducing the number of calls to DU. However, the delete operation takes too long. I've manually deleted archive after 10 jobs had run and it took over 30 minutes on XFS.

      I suppose that an option to prohibit jars unpacking would be helpfull in my situation.

        Attachments

        1. hadoop-5175.txt
          0.6 kB
          Todd Lipcon

          Activity

            People

            • Assignee:
              tlipcon Todd Lipcon
              Reporter:
              gudok Andrei Gudkov

              Dates

              • Created:
                Updated:
                Resolved:

                Issue deployment