Hadoop Map/Reduce
  1. Hadoop Map/Reduce
  2. MAPREDUCE-2093

Herriot JT and TT clients should vend statistics

    Details

    • Type: Improvement Improvement
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 0.22.0
    • Fix Version/s: 0.22.0
    • Component/s: test
    • Labels:
      None
    • Tags:
      herriot

      Description

      Mapreduce counterpart of HDFS-1408

      1. MAPREDUCE-2093.patch
        3 kB
        Konstantin Boudnik
      2. MAPREDUCE-2093.patch
        3 kB
        Konstantin Boudnik
      3. MAPREDUCE-2093.patch
        2 kB
        Konstantin Boudnik

        Issue Links

          Activity

          Konstantin Boudnik created issue -
          Konstantin Boudnik made changes -
          Field Original Value New Value
          Link This issue is a clone of HDFS-1408 [ HDFS-1408 ]
          Konstantin Boudnik made changes -
          Project Hadoop HDFS [ 12310942 ] Hadoop Map/Reduce [ 12310941 ]
          Key HDFS-1424 MAPREDUCE-2093
          Affects Version/s 0.22.0 [ 12314184 ]
          Affects Version/s 0.22.0 [ 12314241 ]
          Component/s test [ 12312904 ]
          Component/s test [ 12312916 ]
          Konstantin Boudnik made changes -
          Summary CLONE -Herriot NN and DN clients should vend statistics Herriot JT and TT clients should vend statistics
          Description The HDFS web user interface serves useful information through dfshealth.jsp and dfsnodelist.jsp.

          The Herriot interface to the namenode and datanode (as implemented in NNClient and DNClient, respectively) would benefit from the addition of some way to channel this information. In the case of DNClient this can be an injected method that returns a DatanodeDescriptor relevant to the underlying datanode.

          There seems to be no analagous NamenodeDescriptor. It may be useful to add this as a facade to a visitor that aggregates values across the filesystem datanodes. These values are (from dfshealth JSP):

          Configured Capacity
          DFS Used
          Non DFS Used
          DFS Remaining
          DFS Used%
          DFS Remaining%
          Live Nodes
          Dead Nodes
          Decommissioning Nodes
          Number of Under-Replicated Blocks

          Attributes reflecting the web user interface header may also be useful such as When-Started, Version, When-Compiled, and Upgrade-Status.

          A NamenodeDescriptor would essentially "push down" the code in dfshealth web UI behind a more general abstraction. If it is objectionable to make this class available in HDFS, perhaps this could be packaged in a Herriot specific way.
          Mapreduce counterpart of HDFS-1408
          Hide
          Konstantin Boudnik added a comment -

          MR part of the fix

          Show
          Konstantin Boudnik added a comment - MR part of the fix
          Konstantin Boudnik made changes -
          Attachment MAPREDUCE-2093.patch [ 12455788 ]
          Konstantin Boudnik made changes -
          Link This issue is blocked by HADOOP-6977 [ HADOOP-6977 ]
          Hide
          Konstantin Boudnik added a comment -

          Corresponding part of MR fix.

          Show
          Konstantin Boudnik added a comment - Corresponding part of MR fix.
          Konstantin Boudnik made changes -
          Attachment MAPREDUCE-2093.patch [ 12457237 ]
          Hide
          Konstantin Boudnik added a comment -

          Re-fitting to correspond HADOOP-6977 latest patch.

          Show
          Konstantin Boudnik added a comment - Re-fitting to correspond HADOOP-6977 latest patch.
          Konstantin Boudnik made changes -
          Attachment MAPREDUCE-2093.patch [ 12458301 ]
          Konstantin Boudnik made changes -
          Attachment MAPREDUCE-2093.patch [ 12458301 ]
          Konstantin Boudnik made changes -
          Attachment MAPREDUCE-2093.patch [ 12458303 ]
          Hide
          Konstantin Boudnik added a comment -

          The blocker is fixed - let's check.

          Show
          Konstantin Boudnik added a comment - The blocker is fixed - let's check.
          Konstantin Boudnik made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          Hide
          Konstantin Boudnik added a comment -

          I have ran test-patch manually. FindBugs warning is unrelated because FindBugs doesn't run against Herriot code. Audit warnings are about issues with documentation. System framework test has passed meaning that issue from downstream project is fixed. I'm gonna commit it.

          -1 overall.
          
              +1 @author.  The patch does not contain any @author tags.
          
              +1 tests included.  The patch appears to include 6 new or modified tests.
          
              +1 javadoc.  The javadoc tool did not generate any warning messages.
          
              +1 javac.  The applied patch does not increase the total number of javac compiler warnings.
          
              -1 findbugs.  The patch appears to introduce 13 new Findbugs warnings.
          
              -1 release audit.  The applied patch generated 2 release audit warnings (more than the trunk's current 1 warnings).
          
              +1 system test framework.  The patch passed system test framework compile.
          
          Show
          Konstantin Boudnik added a comment - I have ran test-patch manually. FindBugs warning is unrelated because FindBugs doesn't run against Herriot code. Audit warnings are about issues with documentation. System framework test has passed meaning that issue from downstream project is fixed. I'm gonna commit it. -1 overall. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 6 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. -1 findbugs. The patch appears to introduce 13 new Findbugs warnings. -1 release audit. The applied patch generated 2 release audit warnings (more than the trunk's current 1 warnings). +1 system test framework. The patch passed system test framework compile.
          Hide
          Konstantin Boudnik added a comment -

          Exact replica of HDFS-1408 - no review is needed.
          I have just committed it to trunk.

          Show
          Konstantin Boudnik added a comment - Exact replica of HDFS-1408 - no review is needed. I have just committed it to trunk.
          Konstantin Boudnik made changes -
          Status Patch Available [ 10002 ] Resolved [ 5 ]
          Resolution Fixed [ 1 ]
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Mapreduce-trunk-Commit #538 (See https://hudson.apache.org/hudson/job/Hadoop-Mapreduce-trunk-Commit/538/)
          MAPREDUCE-2093. Herriot JT and TT clients should vend statistics. Contributed by Konstantin Boudnik.

          Show
          Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk-Commit #538 (See https://hudson.apache.org/hudson/job/Hadoop-Mapreduce-trunk-Commit/538/ ) MAPREDUCE-2093 . Herriot JT and TT clients should vend statistics. Contributed by Konstantin Boudnik.
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Mapreduce-trunk #643 (See https://hudson.apache.org/hudson/job/Hadoop-Mapreduce-trunk/643/)

          Show
          Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk #643 (See https://hudson.apache.org/hudson/job/Hadoop-Mapreduce-trunk/643/ )
          Konstantin Shvachko made changes -
          Fix Version/s 0.22.0 [ 12314184 ]
          Konstantin Shvachko made changes -
          Status Resolved [ 5 ] Closed [ 6 ]
          Transition Time In Source Status Execution Times Last Executer Last Execution Date
          Open Open Patch Available Patch Available
          44d 4h 37m 1 Konstantin Boudnik 11/Nov/10 04:51
          Patch Available Patch Available Resolved Resolved
          15h 12m 1 Konstantin Boudnik 11/Nov/10 20:03
          Resolved Resolved Closed Closed
          395d 10h 16m 1 Konstantin Shvachko 12/Dec/11 06:20

            People

            • Assignee:
              Konstantin Boudnik
              Reporter:
              Konstantin Boudnik
            • Votes:
              0 Vote for this issue
              Watchers:
              0 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development