Hadoop Common
  1. Hadoop Common
  2. HADOOP-2899

[HOD] hdfs:///mapredsystem directory not cleaned up after deallocation

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 0.16.0
    • Fix Version/s: 0.17.0
    • Component/s: contrib/hod
    • Labels:
      None
    • Hadoop Flags:
      Reviewed
    • Release Note:
      The mapred system directory generated by HOD is cleaned up at cluster deallocation time.

      Description

      Each submitted job creates a hdfs:///mapredsystem directory, created by (I guess) the hodring process. Problem is that it's not cleaned up at the end of the process; a use case would be:

      • user A allocates a cluster, the hodring is svrX, so a /mapredsystem/srvX directory is created
      • user A deallocates the cluster, but that directory is not cleaned up
      • user B allocates a cluster, and the first node chosen as hodring is svrX, so hodring tries to write hdfs:///mapredsystem but it fails
      • allocation succeeds, but there's no hodring running; looking at
        0-jobtracker/logdir/hadoop.log under the temporary directory I can read:

      2008-02-26 17:28:42,567 WARN org.apache.hadoop.mapred.JobTracker: Error starting tracker: org.apache.hadoop.ipc.RemoteException: org.apache.hadoop.fs.permission.AccessControlException: Permission denied: user=B, access=WRITE, inode="mapredsystem":hadoop:supergroup:rwxr-xr-x

      I guess a possible solution would be to clean up those directories during the deallocation process.

      1. 2899.1.patch
        16 kB
        Hemanth Yamijala

        Activity

        Luca Telloli created issue -
        Hemanth Yamijala made changes -
        Field Original Value New Value
        Assignee Hemanth Yamijala [ yhemanth ]
        Fix Version/s 0.16.1 [ 12312927 ]
        Hemanth Yamijala made changes -
        Component/s contrib/hod [ 12312090 ]
        Component/s mapred [ 12310690 ]
        Assignee Hemanth Yamijala [ yhemanth ] Devaraj Das [ devaraj ]
        Devaraj Das made changes -
        Link This issue is blocked by HADOOP-2815 [ HADOOP-2815 ]
        Devaraj Das made changes -
        Fix Version/s 0.16.1 [ 12312927 ]
        Sameer Paranjpye made changes -
        Link This issue is blocked by HADOOP-2815 [ HADOOP-2815 ]
        Sameer Paranjpye made changes -
        Fix Version/s 0.16.1 [ 12312927 ]
        Hemanth Yamijala made changes -
        Fix Version/s 0.16.1 [ 12312927 ]
        Component/s mapred [ 12310690 ]
        Component/s contrib/hod [ 12312090 ]
        Assignee Devaraj Das [ devaraj ] Hemanth Yamijala [ yhemanth ]
        Fix Version/s 0.17.0 [ 12312913 ]
        Summary hdfs:///mapredsystem directory not cleaned up after deallocation [HOD] hdfs:///mapredsystem directory not cleaned up after deallocation
        Hemanth Yamijala made changes -
        Attachment 2899.1.patch [ 12378256 ]
        Hemanth Yamijala made changes -
        Status Open [ 1 ] Patch Available [ 10002 ]
        Devaraj Das made changes -
        Resolution Fixed [ 1 ]
        Status Patch Available [ 10002 ] Resolved [ 5 ]
        Hemanth Yamijala made changes -
        Hadoop Flags [Reviewed]
        Release Note The mapred system directory generated by HOD is cleaned up at cluster deallocation time.
        Nigel Daley made changes -
        Status Resolved [ 5 ] Closed [ 6 ]

          People

          • Assignee:
            Hemanth Yamijala
            Reporter:
            Luca Telloli
          • Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development