Hadoop Common
  1. Hadoop Common
  2. HADOOP-2898

HOD should allow setting MapReduce UI ports within a port range

    Details

    • Type: Improvement Improvement
    • Status: Closed
    • Priority: Blocker Blocker
    • Resolution: Fixed
    • Affects Version/s: 0.16.0
    • Fix Version/s: 0.21.0
    • Component/s: contrib/hod
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      HOD currently does now allow to explicitly specify ports or a port range in the MapReduce and HDFS sections, but this could be useful. A typical example would involve firewall settings that allow only a certain range of ports.

      1. hadoop-2898v3.txt
        6 kB
        Hemanth Yamijala
      2. hadoop-2898v2.txt
        6 kB
        Peeyush Bishnoi
      3. hadoop-2898v1.txt
        5 kB
        Peeyush Bishnoi
      4. hadoop-2898-0.20.txt
        6 kB
        Robert Chansler
      5. hadoop-2898.txt
        5 kB
        Peeyush Bishnoi

        Activity

        Luca Telloli created issue -
        Hemanth Yamijala made changes -
        Field Original Value New Value
        Assignee Peeyush Bishnoi [ peeyushb ]
        Hide
        Hemanth Yamijala added a comment -

        Marking this a blocker, since we recently got an urgent need to fix this.

        Show
        Hemanth Yamijala added a comment - Marking this a blocker, since we recently got an urgent need to fix this.
        Hemanth Yamijala made changes -
        Priority Minor [ 4 ] Blocker [ 1 ]
        Hide
        Peeyush Bishnoi added a comment -

        Attaching a patch to this issue. This patch will add functionality to configure the port range for Hadoop Services in HOD hodrc .

        Comments ?.

        Show
        Peeyush Bishnoi added a comment - Attaching a patch to this issue. This patch will add functionality to configure the port range for Hadoop Services in HOD hodrc . Comments ?. —
        Peeyush Bishnoi made changes -
        Attachment hadoop-2898.txt [ 12400472 ]
        Hemanth Yamijala made changes -
        Fix Version/s 0.21.0 [ 12313563 ]
        Peeyush Bishnoi made changes -
        Attachment hadoop-2898v1.txt [ 12400495 ]
        Hide
        Hemanth Yamijala added a comment -

        Peeyush, a couple of comments:

        • In HadoopCommand, the member variable, hadoopPortRange should be private. So, by convention, it should be __hadoopPortRange.
        • The low and high ports are being printed at an info level in createXml. These seem like trace statements, hence can be removed, as we know the values are read properly.
        • We have some tests under src/contrib/hod/testing. One of them, test_createHadoopSiteXml, is failing. The reason is because we've now made HadoopCommand take an additional argument, the hadoop port range, which is not being passed from the test case. Please modify the test case. You can verify that tests are running by running python testing/main.py under the HOD_HOME. Passing the port range as the default 50000-60000 in the test case should be Ok.
        • Under documentation, please move the line item for the new configuration option under the hodring section, because it is not an option to be configured under any other section, and hence is not needed for the common configuration section.
        • Also, please change the documentation to read: "Range of ports, among which an available port shall be picked for use to run a Hadoop service, like JobTracker or TaskTracker."
        Show
        Hemanth Yamijala added a comment - Peeyush, a couple of comments: In HadoopCommand, the member variable, hadoopPortRange should be private. So, by convention, it should be __hadoopPortRange. The low and high ports are being printed at an info level in createXml. These seem like trace statements, hence can be removed, as we know the values are read properly. We have some tests under src/contrib/hod/testing. One of them, test_createHadoopSiteXml, is failing. The reason is because we've now made HadoopCommand take an additional argument, the hadoop port range, which is not being passed from the test case. Please modify the test case. You can verify that tests are running by running python testing/main.py under the HOD_HOME. Passing the port range as the default 50000-60000 in the test case should be Ok. Under documentation, please move the line item for the new configuration option under the hodring section, because it is not an option to be configured under any other section, and hence is not needed for the common configuration section. Also, please change the documentation to read: "Range of ports, among which an available port shall be picked for use to run a Hadoop service, like JobTracker or TaskTracker."
        Hide
        Peeyush Bishnoi added a comment -

        Thanks! Hemanth for suggestions. Attaching the new patch after incorporating all your suggestions.

        Show
        Peeyush Bishnoi added a comment - Thanks! Hemanth for suggestions. Attaching the new patch after incorporating all your suggestions. —
        Peeyush Bishnoi made changes -
        Attachment hadoop-2898v2.txt [ 12400582 ]
        Peeyush Bishnoi made changes -
        Status Open [ 1 ] Patch Available [ 10002 ]
        Hemanth Yamijala made changes -
        Status Patch Available [ 10002 ] Open [ 1 ]
        Hide
        Hemanth Yamijala added a comment -

        Minor editorial tweak in the documentation. Everything else remains the same.

        Show
        Hemanth Yamijala added a comment - Minor editorial tweak in the documentation. Everything else remains the same.
        Hemanth Yamijala made changes -
        Attachment hadoop-2898v3.txt [ 12400594 ]
        Hemanth Yamijala made changes -
        Status Open [ 1 ] Patch Available [ 10002 ]
        Hide
        Hadoop QA added a comment -

        +1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12400594/hadoop-2898v3.txt
        against trunk revision 746340.

        +1 @author. The patch does not contain any @author tags.

        +1 tests included. The patch appears to include 3 new or modified tests.

        +1 javadoc. The javadoc tool did not generate any warning messages.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 findbugs. The patch does not introduce any new Findbugs warnings.

        +1 Eclipse classpath. The patch retains Eclipse classpath integrity.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        +1 core tests. The patch passed core unit tests.

        +1 contrib tests. The patch passed contrib unit tests.

        Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/3894/testReport/
        Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/3894/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
        Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/3894/artifact/trunk/build/test/checkstyle-errors.html
        Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/3894/console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - +1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12400594/hadoop-2898v3.txt against trunk revision 746340. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 3 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 Eclipse classpath. The patch retains Eclipse classpath integrity. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed core unit tests. +1 contrib tests. The patch passed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/3894/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/3894/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/3894/artifact/trunk/build/test/checkstyle-errors.html Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/3894/console This message is automatically generated.
        Hide
        Hemanth Yamijala added a comment -

        I committed this to trunk. Thanks, Peeyush.

        Show
        Hemanth Yamijala added a comment - I committed this to trunk. Thanks, Peeyush.
        Hemanth Yamijala made changes -
        Resolution Fixed [ 1 ]
        Hadoop Flags [Reviewed]
        Status Patch Available [ 10002 ] Resolved [ 5 ]
        Hide
        Hudson added a comment -
        Show
        Hudson added a comment - Integrated in Hadoop-trunk #766 (See http://hudson.zones.apache.org/hudson/job/Hadoop-trunk/766/ )
        Hide
        Suman Sehgal added a comment -

        Verified the feature on trunk. It is working fine but this parameter is included as mandatory feature. Raised a Jira HADOOP-5393 to make it optional.

        Show
        Suman Sehgal added a comment - Verified the feature on trunk. It is working fine but this parameter is included as mandatory feature. Raised a Jira HADOOP-5393 to make it optional.
        Hide
        Robert Chansler added a comment -

        Example for 0.20, not to be committed to the 0.20 branch.

        Show
        Robert Chansler added a comment - Example for 0.20, not to be committed to the 0.20 branch.
        Robert Chansler made changes -
        Attachment hadoop-2898-0.20.txt [ 12409316 ]
        Tom White made changes -
        Status Resolved [ 5 ] Closed [ 6 ]
        Transition Time In Source Status Execution Times Last Executer Last Execution Date
        Patch Available Patch Available Open Open
        1h 21m 1 Hemanth Yamijala 20/Feb/09 09:23
        Open Open Patch Available Patch Available
        359d 17h 58m 2 Hemanth Yamijala 20/Feb/09 09:25
        Patch Available Patch Available Resolved Resolved
        4d 1h 17m 1 Hemanth Yamijala 24/Feb/09 10:43
        Resolved Resolved Closed Closed
        546d 9h 50m 1 Tom White 24/Aug/10 21:34

          People

          • Assignee:
            Peeyush Bishnoi
            Reporter:
            Luca Telloli
          • Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development