Hadoop Common
  1. Hadoop Common
  2. HADOOP-2899

[HOD] hdfs:///mapredsystem directory not cleaned up after deallocation

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 0.16.0
    • Fix Version/s: 0.17.0
    • Component/s: contrib/hod
    • Labels:
      None
    • Hadoop Flags:
      Reviewed
    • Release Note:
      The mapred system directory generated by HOD is cleaned up at cluster deallocation time.

      Description

      Each submitted job creates a hdfs:///mapredsystem directory, created by (I guess) the hodring process. Problem is that it's not cleaned up at the end of the process; a use case would be:

      • user A allocates a cluster, the hodring is svrX, so a /mapredsystem/srvX directory is created
      • user A deallocates the cluster, but that directory is not cleaned up
      • user B allocates a cluster, and the first node chosen as hodring is svrX, so hodring tries to write hdfs:///mapredsystem but it fails
      • allocation succeeds, but there's no hodring running; looking at
        0-jobtracker/logdir/hadoop.log under the temporary directory I can read:

      2008-02-26 17:28:42,567 WARN org.apache.hadoop.mapred.JobTracker: Error starting tracker: org.apache.hadoop.ipc.RemoteException: org.apache.hadoop.fs.permission.AccessControlException: Permission denied: user=B, access=WRITE, inode="mapredsystem":hadoop:supergroup:rwxr-xr-x

      I guess a possible solution would be to clean up those directories during the deallocation process.

      1. 2899.1.patch
        16 kB
        Hemanth Yamijala

        Activity

        No work has yet been logged on this issue.

          People

          • Assignee:
            Hemanth Yamijala
            Reporter:
            Luca Telloli
          • Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development