Uploaded image for project: 'Hadoop Common'
  1. Hadoop Common
  2. HADOOP-2899

[HOD] hdfs:///mapredsystem directory not cleaned up after deallocation

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 0.16.0
    • Fix Version/s: 0.17.0
    • Component/s: contrib/hod
    • Labels:
      None
    • Hadoop Flags:
      Reviewed
    • Release Note:
      The mapred system directory generated by HOD is cleaned up at cluster deallocation time.

      Description

      Each submitted job creates a hdfs:///mapredsystem directory, created by (I guess) the hodring process. Problem is that it's not cleaned up at the end of the process; a use case would be:

      • user A allocates a cluster, the hodring is svrX, so a /mapredsystem/srvX directory is created
      • user A deallocates the cluster, but that directory is not cleaned up
      • user B allocates a cluster, and the first node chosen as hodring is svrX, so hodring tries to write hdfs:///mapredsystem but it fails
      • allocation succeeds, but there's no hodring running; looking at
        0-jobtracker/logdir/hadoop.log under the temporary directory I can read:

      2008-02-26 17:28:42,567 WARN org.apache.hadoop.mapred.JobTracker: Error starting tracker: org.apache.hadoop.ipc.RemoteException: org.apache.hadoop.fs.permission.AccessControlException: Permission denied: user=B, access=WRITE, inode="mapredsystem":hadoop:supergroup:rwxr-xr-x

      I guess a possible solution would be to clean up those directories during the deallocation process.

        Attachments

        1. 2899.1.patch
          16 kB
          Hemanth Yamijala

          Activity

            People

            • Assignee:
              yhemanth Hemanth Yamijala
              Reporter:
              tuka Luca Telloli
            • Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: