Details

    • Type: New Feature New Feature
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 0.20.0
    • Fix Version/s: 0.21.0
    • Component/s: None
    • Labels:
      None
    • Hadoop Flags:
      Reviewed
    • Release Note:
      Hide
      New mradmin command -refreshNodes updates the job tracker's node lists.
      Show
      New mradmin command -refreshNodes updates the job tracker's node lists.

      Description

      Its not always possible to shutdown the tasktracker to stop scheduling tasks on the node. (eg you can't login to the node but the TT is up).

      This can be via

      • mapred.exclude and should be refreshed with out restarting the tasktracker
      • hadoop job -fail-tracker <tracker id>
      1. Fixed+5643-0.20-final
        57 kB
        Amar Kamat
      2. Fixed+5643-0.20-part2
        2 kB
        Amar Kamat
      3. Fixed 5643-0.20
        55 kB
        Robert Chansler
      4. HADOOP-5643-v5.12-testcase.patch
        13 kB
        Amar Kamat
      5. HADOOP-5643-v5.12.patch
        50 kB
        Amar Kamat
      6. HADOOP-5643-v5.9.patch
        60 kB
        Amar Kamat
      7. HADOOP-5643-v5.5.patch
        59 kB
        Amar Kamat
      8. HADOOP-5643-v5.1.patch
        54 kB
        Amar Kamat
      9. HADOOP-5643-v4.6.patch
        82 kB
        Amar Kamat
      10. HADOOP-5643-v4.0.patch
        81 kB
        Amar Kamat
      11. HADOOP-5643-v3.4.patch
        60 kB
        Amar Kamat

        Issue Links

          Activity

          Rajiv Chittajallu created issue -
          Koji Noguchi made changes -
          Field Original Value New Value
          Link This issue is related to HADOOP-4733 [ HADOOP-4733 ]
          Hide
          Koji Noguchi added a comment -

          Just like we use dfs.exclude for blocking AND decommissioning datanodes,
          can we use this feature for blocking(blacklisting) and decommisioning TaskTrackers?

          Show
          Koji Noguchi added a comment - Just like we use dfs.exclude for blocking AND decommissioning datanodes, can we use this feature for blocking(blacklisting) and decommisioning TaskTrackers?
          Hide
          Owen O'Malley added a comment -

          I think we should also be able to do this via the web ui, which is very convenient.

          There should be a way to make it not black listed any more.

          It should be persistent across job tracker restarts.

          It probably should be decommissioning instead of black listing.

          It should probably start rerunning all of the running and stored tasks. (including the map outputs that are stored there)

          Show
          Owen O'Malley added a comment - I think we should also be able to do this via the web ui, which is very convenient. There should be a way to make it not black listed any more. It should be persistent across job tracker restarts. It probably should be decommissioning instead of black listing. It should probably start rerunning all of the running and stored tasks. (including the map outputs that are stored there)
          Amar Kamat made changes -
          Assignee Amar Kamat [ amar_kamat ]
          Hide
          Amar Kamat added a comment -

          I think calling this as blacklisting will lead to more confusion. As Owen suggested we can call it as decommissioning/recommissioning of trackers which would essentially mean that irrespective of what state the tracker is, the jobtracker is asked to decommission(rerun+ignore)/recommission(add back) it. So the command would be

          bin/hadoop jobtracker -decommission tracker1,tracker2.... and bin/hadoop jobtracker -recommission tracker1,tracker2.....

          All the running tasks (also completed maps) that were launched on that machine will be killed and rerun. We can reuse the lost-tracker code for doing this. Maybe a thread should be started on demand (similar to cleanup queue thread) for a decommissioning request. Also these tracker will be added to the ignore list (i.e issue a 'shutdown' upon contact). So a decommission request is equivalent to lost-tracker + add-to-ignore-list.

          Upon a recommission, the trackers will be removed from the ignore list. This can be done inline.

          From the webui, a simple checkbox against all the trackers can be provided and an action named 'Decommission' can be provided (similar to actions for jobs on jobtracker.jsp). On the trackers page, we can provide another section for decommissioned trackers and there we can provide a checkbox for recommissioning it.

          Note :
          1) Acls check should be done before decommissioning and recommissioning.
          2) This info needs to be persisted. Upon every decommission/recommission, persist this info to system.dir/jobtracker.info
          3) Upon restart, the ignore list will also be recovered and loaded (i.e invoke jobtracker.decommission(recovered-list) from recovery-manager)
          4) These new apis can be added to the TaskTrackerManager interface as there really are tasktracker level actions.


          Thoughts?

          Show
          Amar Kamat added a comment - I think calling this as blacklisting will lead to more confusion. As Owen suggested we can call it as decommissioning/recommissioning of trackers which would essentially mean that irrespective of what state the tracker is, the jobtracker is asked to decommission(rerun+ignore)/recommission(add back) it. So the command would be bin/hadoop jobtracker -decommission tracker1,tracker2.... and bin/hadoop jobtracker -recommission tracker1,tracker2.... . All the running tasks (also completed maps) that were launched on that machine will be killed and rerun. We can reuse the lost-tracker code for doing this. Maybe a thread should be started on demand (similar to cleanup queue thread) for a decommissioning request. Also these tracker will be added to the ignore list (i.e issue a 'shutdown' upon contact). So a decommission request is equivalent to lost-tracker + add-to-ignore-list. Upon a recommission, the trackers will be removed from the ignore list. This can be done inline. From the webui, a simple checkbox against all the trackers can be provided and an action named 'Decommission' can be provided (similar to actions for jobs on jobtracker.jsp). On the trackers page, we can provide another section for decommissioned trackers and there we can provide a checkbox for recommissioning it. Note : 1) Acls check should be done before decommissioning and recommissioning. 2) This info needs to be persisted. Upon every decommission/recommission, persist this info to system.dir/jobtracker.info 3) Upon restart, the ignore list will also be recovered and loaded (i.e invoke jobtracker.decommission(recovered-list) from recovery-manager) 4) These new apis can be added to the TaskTrackerManager interface as there really are tasktracker level actions. Thoughts?
          Hide
          Amar Kamat added a comment -

          One more thing i forgot to add is that the jobtracker already reads the hosts file and the exclude file but just once. There is no refresh facility to it. I think we can add that to MR too. So here is the sequence of things :

          1. files to include can be specified via mapred.hosts and will be read by the jobtracker upon init
          2. files to exclude can be specified via mapred.hosts.exclude and will also be read by the jobtracker upon init
          3. Admins can change these files and invoke a refresh from the command line (maybe from the webui too). These files will be loaded back.
          4. If the hosts are added via command line or webui, it will be appended to the include/exclude files. So that upon next restart, the previously included/excluded host info is re-used.
          Show
          Amar Kamat added a comment - One more thing i forgot to add is that the jobtracker already reads the hosts file and the exclude file but just once. There is no refresh facility to it. I think we can add that to MR too. So here is the sequence of things : files to include can be specified via mapred.hosts and will be read by the jobtracker upon init files to exclude can be specified via mapred.hosts.exclude and will also be read by the jobtracker upon init Admins can change these files and invoke a refresh from the command line (maybe from the webui too). These files will be loaded back. If the hosts are added via command line or webui, it will be appended to the include/exclude files. So that upon next restart, the previously included/excluded host info is re-used.
          Hide
          eric baldeschwieler added a comment -

          Do we have a good security story for actions taken through the web UI? Absent that, I'd suggest we don't enable this there.

          Being able to modify the excludes file and hup the server is probably good enough for an operator.

          Show
          eric baldeschwieler added a comment - Do we have a good security story for actions taken through the web UI? Absent that, I'd suggest we don't enable this there. Being able to modify the excludes file and hup the server is probably good enough for an operator.
          Hide
          Amar Kamat added a comment -

          Had an offline discussion with Devaraj and we think it makes sense to provide a default location for mapred.hosts.exclude. The purpose of doing this is to provide persistence. The default file would be something like $

          {hadoop.log.dir}

          /history/hosts.exclude. By default the jobtracker persists the decommission/recommission host info in this file.

          @Eric
          I think we should do what we do for job killing i.e private actions thingy. There is always the option to do it either via refresh or -decommission/-recommission command line option. All the admin operations at the jobtracker will be checked for owner access. For now I think only the user who runs the jobtracker should be allowed to fire these commands. Thoughts?

          Show
          Amar Kamat added a comment - Had an offline discussion with Devaraj and we think it makes sense to provide a default location for mapred.hosts.exclude. The purpose of doing this is to provide persistence. The default file would be something like $ {hadoop.log.dir} /history/hosts.exclude. By default the jobtracker persists the decommission/recommission host info in this file. @Eric I think we should do what we do for job killing i.e private actions thingy. There is always the option to do it either via refresh or -decommission/-recommission command line option. All the admin operations at the jobtracker will be checked for owner access. For now I think only the user who runs the jobtracker should be allowed to fire these commands. Thoughts?
          Hide
          Amar Kamat added a comment -

          Attaching a patch implementing the above discussed approach. Result of test patch

          [exec] -1 overall.  
               [exec] 
               [exec]     +1 @author.  The patch does not contain any @author tags.
               [exec] 
               [exec]     +1 tests included.  The patch appears to include 9 new or modified tests.
               [exec] 
               [exec]     +1 javadoc.  The javadoc tool did not generate any warning messages.
               [exec] 
               [exec]     +1 javac.  The applied patch does not increase the total number of javac compiler warnings.
               [exec] 
               [exec]     +1 findbugs.  The patch does not introduce any new Findbugs warnings.
               [exec] 
               [exec]     +1 Eclipse classpath. The patch retains Eclipse classpath integrity.
               [exec] 
               [exec]     -1 release audit.  The applied patch generated 472 release audit warnings (more than the trunk's current 469 warnings).
          

          Not clear why release audit warnings are there. This patch is tested on local box and testing is in progress. Will upload a new patch with fixed warnings and testcases.

          Show
          Amar Kamat added a comment - Attaching a patch implementing the above discussed approach. Result of test patch [exec] -1 overall. [exec] [exec] +1 @author. The patch does not contain any @author tags. [exec] [exec] +1 tests included. The patch appears to include 9 new or modified tests. [exec] [exec] +1 javadoc. The javadoc tool did not generate any warning messages. [exec] [exec] +1 javac. The applied patch does not increase the total number of javac compiler warnings. [exec] [exec] +1 findbugs. The patch does not introduce any new Findbugs warnings. [exec] [exec] +1 Eclipse classpath. The patch retains Eclipse classpath integrity. [exec] [exec] -1 release audit. The applied patch generated 472 release audit warnings (more than the trunk's current 469 warnings). Not clear why release audit warnings are there. This patch is tested on local box and testing is in progress. Will upload a new patch with fixed warnings and testcases.
          Amar Kamat made changes -
          Attachment HADOOP-5643-v3.4.patch [ 12406689 ]
          Hide
          Amar Kamat added a comment -

          Also added a new parameter mapred.permissions.supergroup to allow admins specify supergroups. Either the user running the jobtracker or user in the supergroup can issue admin commands.

          Show
          Amar Kamat added a comment - Also added a new parameter mapred.permissions.supergroup to allow admins specify supergroups. Either the user running the jobtracker or user in the supergroup can issue admin commands.
          Hide
          Amar Kamat added a comment -

          Attaching a patch with the test case. Result of test-patch

          [exec] +1 overall.  
               [exec] 
               [exec]     +1 @author.  The patch does not contain any @author tags.
               [exec] 
               [exec]     +1 tests included.  The patch appears to include 15 new or modified tests.
               [exec] 
               [exec]     +1 javadoc.  The javadoc tool did not generate any warning messages.
               [exec] 
               [exec]     +1 javac.  The applied patch does not increase the total number of javac compiler warnings.
               [exec] 
               [exec]     +1 findbugs.  The patch does not introduce any new Findbugs warnings.
               [exec] 
               [exec]     +1 Eclipse classpath. The patch retains Eclipse classpath integrity.
               [exec] 
               [exec]     +1 release audit.  The applied patch does not increase the total number of release audit warnings.
          

          Testing in progress.

          Show
          Amar Kamat added a comment - Attaching a patch with the test case. Result of test-patch [exec] +1 overall. [exec] [exec] +1 @author. The patch does not contain any @author tags. [exec] [exec] +1 tests included. The patch appears to include 15 new or modified tests. [exec] [exec] +1 javadoc. The javadoc tool did not generate any warning messages. [exec] [exec] +1 javac. The applied patch does not increase the total number of javac compiler warnings. [exec] [exec] +1 findbugs. The patch does not introduce any new Findbugs warnings. [exec] [exec] +1 Eclipse classpath. The patch retains Eclipse classpath integrity. [exec] [exec] +1 release audit. The applied patch does not increase the total number of release audit warnings. Testing in progress.
          Amar Kamat made changes -
          Attachment HADOOP-5643-v4.0.patch [ 12406776 ]
          Hide
          Amar Kamat added a comment -

          While testing the patch, we found that manually changing the excludes file (maintained by the jobtracker) results in checksum error. So, for now we think keeping it as it is (i.e using java to write/read) makes more sense. Also here are some of the test bugs :

          1. documentation for command line is not sufficient
          2. passing a wrong file to hosts reader causes the earlier data to be lost. So the correct way would be to first load the new file and on success replace the internal structures.
          3. web ui doesnt work as expected.

          Will upload a new patch soon.

          Show
          Amar Kamat added a comment - While testing the patch, we found that manually changing the excludes file (maintained by the jobtracker) results in checksum error. So, for now we think keeping it as it is (i.e using java to write/read) makes more sense. Also here are some of the test bugs : documentation for command line is not sufficient passing a wrong file to hosts reader causes the earlier data to be lost. So the correct way would be to first load the new file and on success replace the internal structures. web ui doesnt work as expected. Will upload a new patch soon.
          Hide
          Rajiv Chittajallu added a comment -

          >While testing the patch, we found that manually changing the excludes file (maintained by the jobtracker) results in checksum error.

          For the name NameNode, this is allowed. JT should do the same.

          Show
          Rajiv Chittajallu added a comment - >While testing the patch, we found that manually changing the excludes file (maintained by the jobtracker) results in checksum error. For the name NameNode, this is allowed. JT should do the same.
          Hide
          eric baldeschwieler added a comment -

          I think we are going through an expensive process of reinventing the wheel here. We should think about solving this sort of issue once by maintain such lists in a plugable source of configuration and supporting the ability to "hup" the service.

          We should then implement config in LDAP / SQL / or some other service via plugins and then we can modify these configurations in an environment with lots of tools to support this stuff. Adding ad hock commands and odd side files that will be lost if we need to swap hardware is awkward.

          Show
          eric baldeschwieler added a comment - I think we are going through an expensive process of reinventing the wheel here. We should think about solving this sort of issue once by maintain such lists in a plugable source of configuration and supporting the ability to "hup" the service. We should then implement config in LDAP / SQL / or some other service via plugins and then we can modify these configurations in an environment with lots of tools to support this stuff. Adding ad hock commands and odd side files that will be lost if we need to swap hardware is awkward.
          Hide
          Amar Kamat added a comment -

          Attaching a patch fixing some bugs. Result of test-patch

           [exec] +1 overall.  
               [exec] 
               [exec]     +1 @author.  The patch does not contain any @author tags.
               [exec] 
               [exec]     +1 tests included.  The patch appears to include 15 new or modified tests.
               [exec] 
               [exec]     +1 javadoc.  The javadoc tool did not generate any warning messages.
               [exec] 
               [exec]     +1 javac.  The applied patch does not increase the total number of javac compiler warnings.
               [exec] 
               [exec]     +1 findbugs.  The patch does not introduce any new Findbugs warnings.
               [exec] 
               [exec]     +1 Eclipse classpath. The patch retains Eclipse classpath integrity.
               [exec] 
               [exec]     +1 release audit.  The applied patch does not increase the total number of release audit warnings
          

          Running ant test now. Will post the results of ant test and cluster run.

          Show
          Amar Kamat added a comment - Attaching a patch fixing some bugs. Result of test-patch [exec] +1 overall. [exec] [exec] +1 @author. The patch does not contain any @author tags. [exec] [exec] +1 tests included. The patch appears to include 15 new or modified tests. [exec] [exec] +1 javadoc. The javadoc tool did not generate any warning messages. [exec] [exec] +1 javac. The applied patch does not increase the total number of javac compiler warnings. [exec] [exec] +1 findbugs. The patch does not introduce any new Findbugs warnings. [exec] [exec] +1 Eclipse classpath. The patch retains Eclipse classpath integrity. [exec] [exec] +1 release audit. The applied patch does not increase the total number of release audit warnings Running ant test now. Will post the results of ant test and cluster run.
          Amar Kamat made changes -
          Attachment HADOOP-5643-v4.6.patch [ 12407150 ]
          Hide
          Amar Kamat added a comment -

          Attaching a patch that does what HDFS does. Testing the patch.

          Show
          Amar Kamat added a comment - Attaching a patch that does what HDFS does. Testing the patch.
          Amar Kamat made changes -
          Attachment HADOOP-5643-v5.1.patch [ 12407250 ]
          Hide
          Amar Kamat added a comment -

          Result of test patch.

          [exec] +1 overall.  
               [exec] 
               [exec]     +1 @author.  The patch does not contain any @author tags.
               [exec] 
               [exec]     +1 tests included.  The patch appears to include 15 new or modified tests.
               [exec] 
               [exec]     +1 javadoc.  The javadoc tool did not generate any warning messages.
               [exec] 
               [exec]     +1 javac.  The applied patch does not increase the total number of javac compiler warnings.
               [exec] 
               [exec]     +1 findbugs.  The patch does not introduce any new Findbugs warnings.
               [exec] 
               [exec]     +1 Eclipse classpath. The patch retains Eclipse classpath integrity.
               [exec] 
               [exec]     +1 release audit.  The applied patch does not increase the total number of release audit warnings.
          

          Running ant test

          Show
          Amar Kamat added a comment - Result of test patch. [exec] +1 overall. [exec] [exec] +1 @author. The patch does not contain any @author tags. [exec] [exec] +1 tests included. The patch appears to include 15 new or modified tests. [exec] [exec] +1 javadoc. The javadoc tool did not generate any warning messages. [exec] [exec] +1 javac. The applied patch does not increase the total number of javac compiler warnings. [exec] [exec] +1 findbugs. The patch does not introduce any new Findbugs warnings. [exec] [exec] +1 Eclipse classpath. The patch retains Eclipse classpath integrity. [exec] [exec] +1 release audit. The applied patch does not increase the total number of release audit warnings. Running ant test
          Hide
          Arun C Murthy added a comment -

          I think we are going through an expensive process of reinventing the wheel here. We should think about solving this sort of issue once by maintain such lists in a plugable source of configuration and supporting the ability to "hup" the service.

          +1. I've opened HADOOP-5772 for the same.

          Show
          Arun C Murthy added a comment - I think we are going through an expensive process of reinventing the wheel here. We should think about solving this sort of issue once by maintain such lists in a plugable source of configuration and supporting the ability to "hup" the service. +1. I've opened HADOOP-5772 for the same.
          Hide
          Amar Kamat added a comment -

          Ant tests passed on my box.

          Show
          Amar Kamat added a comment - Ant tests passed on my box.
          Hide
          Amar Kamat added a comment -

          Attaching a patch the tried to provide the refresh facility similar to HDFS. Result of test-patch

          [exec] +1 overall.  
               [exec] 
               [exec]     +1 @author.  The patch does not contain any @author tags.
               [exec] 
               [exec]     +1 tests included.  The patch appears to include 21 new or modified tests.
               [exec] 
               [exec]     +1 javadoc.  The javadoc tool did not generate any warning messages.
               [exec] 
               [exec]     +1 javac.  The applied patch does not increase the total number of javac compiler warnings.
               [exec] 
               [exec]     +1 findbugs.  The patch does not introduce any new Findbugs warnings.
               [exec] 
               [exec]     +1 Eclipse classpath. The patch retains Eclipse classpath integrity.
               [exec] 
               [exec]     +1 release audit.  The applied patch does not increase the total number of release audit warnings.
          

          Ant test passed on my box.

          Show
          Amar Kamat added a comment - Attaching a patch the tried to provide the refresh facility similar to HDFS. Result of test-patch [exec] +1 overall. [exec] [exec] +1 @author. The patch does not contain any @author tags. [exec] [exec] +1 tests included. The patch appears to include 21 new or modified tests. [exec] [exec] +1 javadoc. The javadoc tool did not generate any warning messages. [exec] [exec] +1 javac. The applied patch does not increase the total number of javac compiler warnings. [exec] [exec] +1 findbugs. The patch does not introduce any new Findbugs warnings. [exec] [exec] +1 Eclipse classpath. The patch retains Eclipse classpath integrity. [exec] [exec] +1 release audit. The applied patch does not increase the total number of release audit warnings. Ant test passed on my box.
          Amar Kamat made changes -
          Attachment HADOOP-5643-v5.5.patch [ 12407439 ]
          Hide
          Amar Kamat added a comment -

          Attaching a patch incorporating Devaraj's offline comment. Result of test-patch

           [exec] +1 overall.  
               [exec] 
               [exec]     +1 @author.  The patch does not contain any @author tags.
               [exec] 
               [exec]     +1 tests included.  The patch appears to include 15 new or modified tests.
               [exec] 
               [exec]     +1 javadoc.  The javadoc tool did not generate any warning messages.
               [exec] 
               [exec]     +1 javac.  The applied patch does not increase the total number of javac compiler warnings.
               [exec] 
               [exec]     +1 findbugs.  The patch does not introduce any new Findbugs warnings.
               [exec] 
               [exec]     +1 Eclipse classpath. The patch retains Eclipse classpath integrity.
               [exec] 
               [exec]     +1 release audit.  The applied patch does not increase the total number of release audit warnings.
          

          Ant tests passed on my box.

          Show
          Amar Kamat added a comment - Attaching a patch incorporating Devaraj's offline comment. Result of test-patch [exec] +1 overall. [exec] [exec] +1 @author. The patch does not contain any @author tags. [exec] [exec] +1 tests included. The patch appears to include 15 new or modified tests. [exec] [exec] +1 javadoc. The javadoc tool did not generate any warning messages. [exec] [exec] +1 javac. The applied patch does not increase the total number of javac compiler warnings. [exec] [exec] +1 findbugs. The patch does not introduce any new Findbugs warnings. [exec] [exec] +1 Eclipse classpath. The patch retains Eclipse classpath integrity. [exec] [exec] +1 release audit. The applied patch does not increase the total number of release audit warnings. Ant tests passed on my box.
          Amar Kamat made changes -
          Attachment HADOOP-5643-v5.9.patch [ 12407605 ]
          Hide
          Amar Kamat added a comment -

          Attaching a patch incorporating Devaraj's offline comments.

          1. Permission checks are now factored out into a common class
          2. Replaced decommissioned in some cases with excluded.

          Result of test-patch

          [exec] +1 overall.  
               [exec] 
               [exec]     +1 @author.  The patch does not contain any @author tags.
               [exec] 
               [exec]     +1 tests included.  The patch appears to include 9 new or modified tests.
               [exec] 
               [exec]     +1 javadoc.  The javadoc tool did not generate any warning messages.
               [exec] 
               [exec]     +1 javac.  The applied patch does not increase the total number of javac compiler warnings.
               [exec] 
               [exec]     +1 findbugs.  The patch does not introduce any new Findbugs warnings.
               [exec] 
               [exec]     +1 Eclipse classpath. The patch retains Eclipse classpath integrity.
               [exec] 
               [exec]     +1 release audit.  The applied patch does not increase the total number of release audit warnings.
          

          Running ant test now.

          Show
          Amar Kamat added a comment - Attaching a patch incorporating Devaraj's offline comments. Permission checks are now factored out into a common class Replaced decommissioned in some cases with excluded. Result of test-patch [exec] +1 overall. [exec] [exec] +1 @author. The patch does not contain any @author tags. [exec] [exec] +1 tests included. The patch appears to include 9 new or modified tests. [exec] [exec] +1 javadoc. The javadoc tool did not generate any warning messages. [exec] [exec] +1 javac. The applied patch does not increase the total number of javac compiler warnings. [exec] [exec] +1 findbugs. The patch does not introduce any new Findbugs warnings. [exec] [exec] +1 Eclipse classpath. The patch retains Eclipse classpath integrity. [exec] [exec] +1 release audit. The applied patch does not increase the total number of release audit warnings. Running ant test now.
          Amar Kamat made changes -
          Attachment HADOOP-5643-v5.12.patch [ 12407619 ]
          Hide
          Amar Kamat added a comment -

          Ant tests passed on my box.

          Show
          Amar Kamat added a comment - Ant tests passed on my box.
          Hide
          Amar Kamat added a comment -

          Missed the testcase.

          Show
          Amar Kamat added a comment - Missed the testcase.
          Amar Kamat made changes -
          Attachment HADOOP-5643-v5.12-testcase.patch [ 12407771 ]
          Hide
          Devaraj Das added a comment -

          I just committed this. Thanks, Amar! (Please add a release note describing the way to run the command for decommissioning TTs)

          Show
          Devaraj Das added a comment - I just committed this. Thanks, Amar! (Please add a release note describing the way to run the command for decommissioning TTs)
          Devaraj Das made changes -
          Status Open [ 1 ] Resolved [ 5 ]
          Hadoop Flags [Reviewed]
          Resolution Fixed [ 1 ]
          Amar Kamat made changes -
          Release Note Added the functionality to refresh jobtrackers node list via command line (bin/hadoop mradmin -refreshNodes). The command should be run as the jobtracker owner (jobtracker process owner) or from a super group (mapred.permissions.supergroup).
          Fix Version/s 0.21.0 [ 12313563 ]
          Hide
          Hudson added a comment -

          Integrated in Hadoop-trunk #833 (See http://hudson.zones.apache.org/hudson/job/Hadoop-trunk/833/)
          . Adding the file src/test/mapred/org/apache/hadoop/mapred/TestNodeRefresh.java that got missed earlier.
          . Adds a way to decommission TaskTrackers while the JobTracker is running. Contributed by Amar Kamat.

          Show
          Hudson added a comment - Integrated in Hadoop-trunk #833 (See http://hudson.zones.apache.org/hudson/job/Hadoop-trunk/833/ ) . Adding the file src/test/mapred/org/apache/hadoop/mapred/TestNodeRefresh.java that got missed earlier. . Adds a way to decommission TaskTrackers while the JobTracker is running. Contributed by Amar Kamat.
          Hide
          Tsz Wo Nicholas Sze added a comment -
          • src/hdfs/org/apache/hadoop/hdfs/server/namenode/PermissionChecker.java becomes an empty file in trunk.
          • Why renaming the FSNamesystem.checkSuperuserPrivilege() method to checkAccess() even though it is still for checking superuser privilege?
          Show
          Tsz Wo Nicholas Sze added a comment - src/hdfs/org/apache/hadoop/hdfs/server/namenode/PermissionChecker.java becomes an empty file in trunk. Why renaming the FSNamesystem.checkSuperuserPrivilege() method to checkAccess() even though it is still for checking superuser privilege?
          Hide
          Amar Kamat added a comment -

          src/hdfs/org/apache/hadoop/hdfs/server/namenode/PermissionChecker.java becomes an empty file in trunk.

          We should do a svn delete of this file.

          Why renaming the FSNamesystem.checkSuperuserPrivilege() method to checkAccess() even though it is still for checking superuser privilege?

          I feel checkSuperuserPrivilege() should be used for simply checking superuser privilege (without permission switch which is just in HDFS for now) and checkAccess() for making a guarded call to checkSuperuserPrivilege(). The reason for doing this was to keep both the MR and HDFS consistent wrt superuser checks.

          Show
          Amar Kamat added a comment - src/hdfs/org/apache/hadoop/hdfs/server/namenode/PermissionChecker.java becomes an empty file in trunk. We should do a svn delete of this file. Why renaming the FSNamesystem.checkSuperuserPrivilege() method to checkAccess() even though it is still for checking superuser privilege? I feel checkSuperuserPrivilege() should be used for simply checking superuser privilege (without permission switch which is just in HDFS for now) and checkAccess() for making a guarded call to checkSuperuserPrivilege(). The reason for doing this was to keep both the MR and HDFS consistent wrt superuser checks.
          Hide
          Tsz Wo Nicholas Sze added a comment -

          I feel checkSuperuserPrivilege() should be used for simply checking superuser privilege (without permission switch which is just in HDFS for now) and checkAccess() for making a guarded call to checkSuperuserPrivilege(). The reason for doing this was to keep both the MR and HDFS consistent wrt superuser checks.

          Then, why not using the name "checkSuperuserPrivilege" for superuser checks in both HDFS and MR? "checkAccess" does not seem to mean "check superuser".

          Also, "checkAccess" seems to be confusing in HDFS. In FSNamesystem, there are other methods checkPathAccess(..), checkParentAccess(..) and checkAncestorAccess(..) which are nothing to do with superuser.

          Show
          Tsz Wo Nicholas Sze added a comment - I feel checkSuperuserPrivilege() should be used for simply checking superuser privilege (without permission switch which is just in HDFS for now) and checkAccess() for making a guarded call to checkSuperuserPrivilege(). The reason for doing this was to keep both the MR and HDFS consistent wrt superuser checks. Then, why not using the name "checkSuperuserPrivilege" for superuser checks in both HDFS and MR? "checkAccess" does not seem to mean "check superuser". Also, "checkAccess" seems to be confusing in HDFS. In FSNamesystem, there are other methods checkPathAccess(..), checkParentAccess(..) and checkAncestorAccess(..) which are nothing to do with superuser.
          Hide
          Hudson added a comment -

          Integrated in Hadoop-trunk #834 (See http://hudson.zones.apache.org/hudson/job/Hadoop-trunk/834/)
          . Removing the empty file src/hdfs/org/apache/hadoop/hdfs/server/namenode/PermissionChecker.java.

          Show
          Hudson added a comment - Integrated in Hadoop-trunk #834 (See http://hudson.zones.apache.org/hudson/job/Hadoop-trunk/834/ ) . Removing the empty file src/hdfs/org/apache/hadoop/hdfs/server/namenode/PermissionChecker.java.
          Hide
          Amar Kamat added a comment -

          Nicholas,
          If checkAccess() adds to confusion then we better revert the renaming.

          Show
          Amar Kamat added a comment - Nicholas, If checkAccess() adds to confusion then we better revert the renaming.
          Tsz Wo Nicholas Sze made changes -
          Link This issue relates to HADOOP-5818 [ HADOOP-5818 ]
          Hide
          Tsz Wo Nicholas Sze added a comment -

          I filed HADOOP-5818: Revert the renaming from checkSuperuserPrivilege to checkAccess by HADOOP-5643.

          Show
          Tsz Wo Nicholas Sze added a comment - I filed HADOOP-5818 : Revert the renaming from checkSuperuserPrivilege to checkAccess by HADOOP-5643 .
          Hide
          Robert Chansler added a comment -

          Example patch for 0.20 not to be committed.

          Show
          Robert Chansler added a comment - Example patch for 0.20 not to be committed.
          Robert Chansler made changes -
          Attachment Fixed 5643-0.20 [ 12409833 ]
          Hide
          Amar Kamat added a comment -

          Example patch not to be committed.

          Show
          Amar Kamat added a comment - Example patch not to be committed.
          Amar Kamat made changes -
          Attachment Fixed+5643-0.20-part2 [ 12409937 ]
          Hide
          Amar Kamat added a comment -

          Attaching a new patch for branch 0.20 merging the 2 patches. Note that this is an example patch for 20 and not to be committed.

          Show
          Amar Kamat added a comment - Attaching a new patch for branch 0.20 merging the 2 patches. Note that this is an example patch for 20 and not to be committed.
          Amar Kamat made changes -
          Attachment Fixed+5643-0.20-final [ 12410619 ]
          Robert Chansler made changes -
          Release Note Added the functionality to refresh jobtrackers node list via command line (bin/hadoop mradmin -refreshNodes). The command should be run as the jobtracker owner (jobtracker process owner) or from a super group (mapred.permissions.supergroup). New mradmin command -refreshNodes updates the job tracker's node lists.
          Tom White made changes -
          Status Resolved [ 5 ] Closed [ 6 ]

            People

            • Assignee:
              Amar Kamat
              Reporter:
              Rajiv Chittajallu
            • Votes:
              0 Vote for this issue
              Watchers:
              10 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development