Hadoop Common
  1. Hadoop Common
  2. HADOOP-8231

Make topologies easier to set up and debug

    Details

    • Type: Improvement Improvement
    • Status: In Progress
    • Priority: Minor Minor
    • Resolution: Unresolved
    • Affects Version/s: 0.23.1
    • Fix Version/s: None
    • Component/s: util
    • Labels:
      None

      Description

      Topology scripts are a source of problems as they

      1. are site-specific.
      2. hard to get right.
      3. can have adverse consequences on cluster operation when they go wrong.
        This issue is to group up the features needed to make it easier for ops people to get their scripts up and running.

        Issue Links

          Activity

          Steve Loughran created issue -
          Steve Loughran made changes -
          Field Original Value New Value
          Link This issue relates to HDFS-2492 [ HDFS-2492 ]
          Steve Loughran made changes -
          Link This issue relates to HADOOP-7030 [ HADOOP-7030 ]
          Steve Loughran made changes -
          Link This issue relates to HADOOP-7777 [ HADOOP-7777 ]
          Hide
          Steve Loughran added a comment -

          This is the implementation class with some changes to make the output more readable.

          1. It does not include any tests; the process for testing entry points is new to me.
          2. it does not patch the bin/hadoop script
          Show
          Steve Loughran added a comment - This is the implementation class with some changes to make the output more readable. It does not include any tests; the process for testing entry points is new to me. it does not patch the bin/hadoop script
          Steve Loughran made changes -
          Attachment HADOOP-8232.patch [ 12520624 ]
          Hide
          Steve Loughran added a comment -

          not ready for committing yet, sending to jenkins

          Show
          Steve Loughran added a comment - not ready for committing yet, sending to jenkins
          Steve Loughran made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          Fix Version/s 3.0.0 [ 12320357 ]
          Hide
          Steve Loughran added a comment -

          patch was to go into sub task

          Show
          Steve Loughran added a comment - patch was to go into sub task
          Steve Loughran made changes -
          Status Patch Available [ 10002 ] Open [ 1 ]
          Steve Loughran made changes -
          Attachment HADOOP-8232.patch [ 12520624 ]
          Hide
          Steve Loughran added a comment -

          Another way to simplify the process would be add JSP pages to the NN and RM, so that the topologies can be listed. The output could be an HTML table or piece of JSON. Clean HTML is easier to generate with JSP than json; JSP easier to test by parsing the output.

          The simplest is just to output text/plain and print the topology information created by AbstractDNSToSwitchMapping.dumpTopology(), which is what is printed out at the end of every successful run of the HADOOP-8232 resolver runs.

          Show
          Steve Loughran added a comment - Another way to simplify the process would be add JSP pages to the NN and RM, so that the topologies can be listed. The output could be an HTML table or piece of JSON. Clean HTML is easier to generate with JSP than json; JSP easier to test by parsing the output. The simplest is just to output text/plain and print the topology information created by AbstractDNSToSwitchMapping.dumpTopology() , which is what is printed out at the end of every successful run of the HADOOP-8232 resolver runs.
          Steve Loughran made changes -
          Link This issue is depended upon by HADOOP-8232 [ HADOOP-8232 ]
          Steve Loughran made changes -
          Link This issue is depended upon by HADOOP-8928 [ HADOOP-8928 ]
          Steve Loughran made changes -
          Link This issue depends on HADOOP-7423 [ HADOOP-7423 ]
          Steve Loughran made changes -
          Link This issue depends on HADOOP-7149 [ HADOOP-7149 ]
          Steve Loughran made changes -
          Link This issue depends on MAPREDUCE-50 [ MAPREDUCE-50 ]
          Hide
          Liang Xie added a comment -

          "hdfs dfsadmin -printTopology" is a good way to dump topology state

          Show
          Liang Xie added a comment - "hdfs dfsadmin -printTopology" is a good way to dump topology state
          Gavin made changes -
          Link This issue depends on MAPREDUCE-50 [ MAPREDUCE-50 ]
          Gavin made changes -
          Link This issue depends upon MAPREDUCE-50 [ MAPREDUCE-50 ]
          Gavin made changes -
          Link This issue depends on HADOOP-7423 [ HADOOP-7423 ]
          Gavin made changes -
          Link This issue depends upon HADOOP-7423 [ HADOOP-7423 ]
          Gavin made changes -
          Link This issue depends on HADOOP-7149 [ HADOOP-7149 ]
          Gavin made changes -
          Link This issue depends upon HADOOP-7149 [ HADOOP-7149 ]
          Steve Loughran made changes -
          Assignee Steve Loughran [ stevel@apache.org ]
          Steve Loughran made changes -
          Status Open [ 1 ] In Progress [ 3 ]
          Allen Wittenauer made changes -
          Fix Version/s 3.0.0 [ 12320357 ]
          Transition Time In Source Status Execution Times Last Executer Last Execution Date
          Open Open Patch Available Patch Available
          34m 42s 1 Steve Loughran 30/Mar/12 17:54
          Patch Available Patch Available Open Open
          1m 42s 1 Steve Loughran 30/Mar/12 17:56
          Open Open In Progress In Progress
          678d 3h 57m 1 Steve Loughran 06/Feb/14 20:54

            People

            • Assignee:
              Steve Loughran
              Reporter:
              Steve Loughran
            • Votes:
              0 Vote for this issue
              Watchers:
              11 Start watching this issue

              Dates

              • Created:
                Updated:

                Development