Hadoop Common
  1. Hadoop Common
  2. HADOOP-2898

HOD should allow setting MapReduce UI ports within a port range

    Details

    • Type: Improvement Improvement
    • Status: Closed
    • Priority: Blocker Blocker
    • Resolution: Fixed
    • Affects Version/s: 0.16.0
    • Fix Version/s: 0.21.0
    • Component/s: contrib/hod
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      HOD currently does now allow to explicitly specify ports or a port range in the MapReduce and HDFS sections, but this could be useful. A typical example would involve firewall settings that allow only a certain range of ports.

      1. hadoop-2898.txt
        5 kB
        Peeyush Bishnoi
      2. hadoop-2898v1.txt
        5 kB
        Peeyush Bishnoi
      3. hadoop-2898v2.txt
        6 kB
        Peeyush Bishnoi
      4. hadoop-2898v3.txt
        6 kB
        Hemanth Yamijala
      5. hadoop-2898-0.20.txt
        6 kB
        Robert Chansler

        Activity

        Hide
        Hemanth Yamijala added a comment -

        Marking this a blocker, since we recently got an urgent need to fix this.

        Show
        Hemanth Yamijala added a comment - Marking this a blocker, since we recently got an urgent need to fix this.
        Hide
        Peeyush Bishnoi added a comment -

        Attaching a patch to this issue. This patch will add functionality to configure the port range for Hadoop Services in HOD hodrc .

        Comments ?.

        Show
        Peeyush Bishnoi added a comment - Attaching a patch to this issue. This patch will add functionality to configure the port range for Hadoop Services in HOD hodrc . Comments ?. —
        Hide
        Hemanth Yamijala added a comment -

        Peeyush, a couple of comments:

        • In HadoopCommand, the member variable, hadoopPortRange should be private. So, by convention, it should be __hadoopPortRange.
        • The low and high ports are being printed at an info level in createXml. These seem like trace statements, hence can be removed, as we know the values are read properly.
        • We have some tests under src/contrib/hod/testing. One of them, test_createHadoopSiteXml, is failing. The reason is because we've now made HadoopCommand take an additional argument, the hadoop port range, which is not being passed from the test case. Please modify the test case. You can verify that tests are running by running python testing/main.py under the HOD_HOME. Passing the port range as the default 50000-60000 in the test case should be Ok.
        • Under documentation, please move the line item for the new configuration option under the hodring section, because it is not an option to be configured under any other section, and hence is not needed for the common configuration section.
        • Also, please change the documentation to read: "Range of ports, among which an available port shall be picked for use to run a Hadoop service, like JobTracker or TaskTracker."
        Show
        Hemanth Yamijala added a comment - Peeyush, a couple of comments: In HadoopCommand, the member variable, hadoopPortRange should be private. So, by convention, it should be __hadoopPortRange. The low and high ports are being printed at an info level in createXml. These seem like trace statements, hence can be removed, as we know the values are read properly. We have some tests under src/contrib/hod/testing. One of them, test_createHadoopSiteXml, is failing. The reason is because we've now made HadoopCommand take an additional argument, the hadoop port range, which is not being passed from the test case. Please modify the test case. You can verify that tests are running by running python testing/main.py under the HOD_HOME. Passing the port range as the default 50000-60000 in the test case should be Ok. Under documentation, please move the line item for the new configuration option under the hodring section, because it is not an option to be configured under any other section, and hence is not needed for the common configuration section. Also, please change the documentation to read: "Range of ports, among which an available port shall be picked for use to run a Hadoop service, like JobTracker or TaskTracker."
        Hide
        Peeyush Bishnoi added a comment -

        Thanks! Hemanth for suggestions. Attaching the new patch after incorporating all your suggestions.

        Show
        Peeyush Bishnoi added a comment - Thanks! Hemanth for suggestions. Attaching the new patch after incorporating all your suggestions. —
        Hide
        Hemanth Yamijala added a comment -

        Minor editorial tweak in the documentation. Everything else remains the same.

        Show
        Hemanth Yamijala added a comment - Minor editorial tweak in the documentation. Everything else remains the same.
        Hide
        Hadoop QA added a comment -

        +1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12400594/hadoop-2898v3.txt
        against trunk revision 746340.

        +1 @author. The patch does not contain any @author tags.

        +1 tests included. The patch appears to include 3 new or modified tests.

        +1 javadoc. The javadoc tool did not generate any warning messages.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 findbugs. The patch does not introduce any new Findbugs warnings.

        +1 Eclipse classpath. The patch retains Eclipse classpath integrity.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        +1 core tests. The patch passed core unit tests.

        +1 contrib tests. The patch passed contrib unit tests.

        Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/3894/testReport/
        Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/3894/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
        Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/3894/artifact/trunk/build/test/checkstyle-errors.html
        Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/3894/console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - +1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12400594/hadoop-2898v3.txt against trunk revision 746340. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 3 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 Eclipse classpath. The patch retains Eclipse classpath integrity. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed core unit tests. +1 contrib tests. The patch passed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/3894/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/3894/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/3894/artifact/trunk/build/test/checkstyle-errors.html Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/3894/console This message is automatically generated.
        Hide
        Hemanth Yamijala added a comment -

        I committed this to trunk. Thanks, Peeyush.

        Show
        Hemanth Yamijala added a comment - I committed this to trunk. Thanks, Peeyush.
        Hide
        Hudson added a comment -
        Show
        Hudson added a comment - Integrated in Hadoop-trunk #766 (See http://hudson.zones.apache.org/hudson/job/Hadoop-trunk/766/ )
        Hide
        Suman Sehgal added a comment -

        Verified the feature on trunk. It is working fine but this parameter is included as mandatory feature. Raised a Jira HADOOP-5393 to make it optional.

        Show
        Suman Sehgal added a comment - Verified the feature on trunk. It is working fine but this parameter is included as mandatory feature. Raised a Jira HADOOP-5393 to make it optional.
        Hide
        Robert Chansler added a comment -

        Example for 0.20, not to be committed to the 0.20 branch.

        Show
        Robert Chansler added a comment - Example for 0.20, not to be committed to the 0.20 branch.

          People

          • Assignee:
            Peeyush Bishnoi
            Reporter:
            Luca Telloli
          • Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development