Hadoop Common
  1. Hadoop Common
  2. HADOOP-8231

Make topologies easier to set up and debug

    Details

    • Type: Improvement Improvement
    • Status: In Progress
    • Priority: Minor Minor
    • Resolution: Unresolved
    • Affects Version/s: 0.23.1
    • Fix Version/s: 3.0.0
    • Component/s: util
    • Labels:
      None

      Description

      Topology scripts are a source of problems as they

      1. are site-specific.
      2. hard to get right.
      3. can have adverse consequences on cluster operation when they go wrong.
        This issue is to group up the features needed to make it easier for ops people to get their scripts up and running.

        Issue Links

          Activity

          Hide
          Liang Xie added a comment -

          "hdfs dfsadmin -printTopology" is a good way to dump topology state

          Show
          Liang Xie added a comment - "hdfs dfsadmin -printTopology" is a good way to dump topology state
          Hide
          Steve Loughran added a comment -

          Another way to simplify the process would be add JSP pages to the NN and RM, so that the topologies can be listed. The output could be an HTML table or piece of JSON. Clean HTML is easier to generate with JSP than json; JSP easier to test by parsing the output.

          The simplest is just to output text/plain and print the topology information created by AbstractDNSToSwitchMapping.dumpTopology(), which is what is printed out at the end of every successful run of the HADOOP-8232 resolver runs.

          Show
          Steve Loughran added a comment - Another way to simplify the process would be add JSP pages to the NN and RM, so that the topologies can be listed. The output could be an HTML table or piece of JSON. Clean HTML is easier to generate with JSP than json; JSP easier to test by parsing the output. The simplest is just to output text/plain and print the topology information created by AbstractDNSToSwitchMapping.dumpTopology() , which is what is printed out at the end of every successful run of the HADOOP-8232 resolver runs.
          Hide
          Steve Loughran added a comment -

          patch was to go into sub task

          Show
          Steve Loughran added a comment - patch was to go into sub task
          Hide
          Steve Loughran added a comment -

          not ready for committing yet, sending to jenkins

          Show
          Steve Loughran added a comment - not ready for committing yet, sending to jenkins
          Hide
          Steve Loughran added a comment -

          This is the implementation class with some changes to make the output more readable.

          1. It does not include any tests; the process for testing entry points is new to me.
          2. it does not patch the bin/hadoop script
          Show
          Steve Loughran added a comment - This is the implementation class with some changes to make the output more readable. It does not include any tests; the process for testing entry points is new to me. it does not patch the bin/hadoop script

            People

            • Assignee:
              Steve Loughran
              Reporter:
              Steve Loughran
            • Votes:
              0 Vote for this issue
              Watchers:
              10 Start watching this issue

              Dates

              • Created:
                Updated:

                Development