Hadoop Common
  1. Hadoop Common
  2. HADOOP-10136

Custom JMX server to avoid random port usage by default JMX Server

    Details

    • Type: Improvement Improvement
    • Status: Patch Available
    • Priority: Major Major
    • Resolution: Unresolved
    • Affects Version/s: None
    • Fix Version/s: None
    • Component/s: None
    • Labels:
      None

      Description

      If any of the java process want to enable the JMX MBean server then following VM arguments needs to be passed.

      -Dcom.sun.management.jmxremote
      -Dcom.sun.management.jmxremote.port=14005
      -Dcom.sun.management.jmxremote.local.only=false
      -Dcom.sun.management.jmxremote.authenticate=false
      -Dcom.sun.management.jmxremote.ssl=false

      But the issue here is this will use one more random port other than 14005 while starting JMX.
      This can be a problem if that random port is used for some other service.

      So support a custom JMX Server through which random port can be avoided.

      1. HADOOP-10136.patch
        15 kB
        Vinayakumar B
      2. HADOOP-10136.patch
        15 kB
        Vinayakumar B
      3. HADOOP-10136.patch
        15 kB
        Vinayakumar B

        Activity

        Hide
        Steve Loughran added a comment -

        Is the port the JMX server coming up on in use? Surely it should be asking to listen on port 0 and let the OS find a port > 1024 where there is space.

        If you are going to do a custom JMX server, I would like the option of letting it take a port 0 for an automatic port selection -but have a way to query the server for the port in use -which is something the standard one doesn't let you do. Maybe also take a range of allowed ports and scan through them until a free port can be used -which will allow ops teams to restrict the open ports on a firewall to the same range

        This is important for starting JMX in YARN containers, as you can't hard code any port bindings without risking port collision.

        Show
        Steve Loughran added a comment - Is the port the JMX server coming up on in use? Surely it should be asking to listen on port 0 and let the OS find a port > 1024 where there is space. If you are going to do a custom JMX server, I would like the option of letting it take a port 0 for an automatic port selection -but have a way to query the server for the port in use -which is something the standard one doesn't let you do. Maybe also take a range of allowed ports and scan through them until a free port can be used -which will allow ops teams to restrict the open ports on a firewall to the same range This is important for starting JMX in YARN containers, as you can't hard code any port bindings without risking port collision.
        Hide
        Vinayakumar B added a comment -

        Thanks Steve for the input.

        Is the port the JMX server coming up on in use?

        This is possible. But my concern was not this. Default JMX will use one more extra random port along with the port configured via -Dcom.sun.management.jmxremote.port.

        I will try to implement suggestions you given.

        have a way to query the server for the port in use

        Do you mean, querying this via RPC..?

        Show
        Vinayakumar B added a comment - Thanks Steve for the input. Is the port the JMX server coming up on in use? This is possible. But my concern was not this. Default JMX will use one more extra random port along with the port configured via -Dcom.sun.management.jmxremote.port . I will try to implement suggestions you given. have a way to query the server for the port in use Do you mean, querying this via RPC..?
        Hide
        Vinayakumar B added a comment -

        Attaching the initial patch.
        Please review and let me know your feedback

        Show
        Vinayakumar B added a comment - Attaching the initial patch. Please review and let me know your feedback
        Hide
        Colin Patrick McCabe added a comment -

        Steve Loughran] Is the port the JMX server coming up on in use?

        Unfortunately, some Hadoop daemons are using ports in the ephemeral range as if they were fixed ports. For example, the DataNode uses port 50070 by default, which is in the Linux ephemeral range of 32768 to 61000. In the long run, we'd like to migrate people away from using those ports. But in the short run, reducing the ephemeral port usage would be a good thing.

        Show
        Colin Patrick McCabe added a comment - Steve Loughran ] Is the port the JMX server coming up on in use? Unfortunately, some Hadoop daemons are using ports in the ephemeral range as if they were fixed ports. For example, the DataNode uses port 50070 by default, which is in the Linux ephemeral range of 32768 to 61000. In the long run, we'd like to migrate people away from using those ports. But in the short run, reducing the ephemeral port usage would be a good thing.
        Hide
        nijel added a comment -

        Unfortunately, some Hadoop daemons are using ports in the ephemeral range as if they were fixed ports.

        In this case we can change the default port right ?

        In case of JMX even if we need to configure it is not possible.

        So i think better to keep this JMX server as an option.

        Show
        nijel added a comment - Unfortunately, some Hadoop daemons are using ports in the ephemeral range as if they were fixed ports. In this case we can change the default port right ? In case of JMX even if we need to configure it is not possible. So i think better to keep this JMX server as an option.
        Hide
        Vinayakumar B added a comment -

        In this case we can change the default port right ?

        We could change default ports to some other range. But these ports are there from very long time. I feel changing needs to be mentioned as incompatible.

        In case of JMX even if we need to configure it is not possible.
        So i think better to keep this JMX server as an option.

        Yes.

        Show
        Vinayakumar B added a comment - In this case we can change the default port right ? We could change default ports to some other range. But these ports are there from very long time. I feel changing needs to be mentioned as incompatible. In case of JMX even if we need to configure it is not possible. So i think better to keep this JMX server as an option. Yes.
        Hide
        Steve Loughran added a comment -
        1. how would this get wired up in an application?
        2. what would be very nice would be for the JMXServer to extend AbstractService, and so share the same lifecycle. Any YARN composite service could simply add it as a child & hook it into its lifecycle
        Show
        Steve Loughran added a comment - how would this get wired up in an application? what would be very nice would be for the JMXServer to extend AbstractService , and so share the same lifecycle. Any YARN composite service could simply add it as a child & hook it into its lifecycle
        Hide
        Vinayakumar B added a comment -

        Hi steve, Thanks for taking a look here.

        how would this get wired up in an application?

        Current patch is just implementation of JMXServer. To include in an Application, need to create instance of JMXServer and start/stop in the application.

        what would be very nice would be for the JMXServer to extend AbstractService, and so share the same lifecycle. Any YARN composite service could simply add it as a child & hook it into its lifecycle

        Yes, good suggestion. I will update patch for this.

        Show
        Vinayakumar B added a comment - Hi steve, Thanks for taking a look here. how would this get wired up in an application? Current patch is just implementation of JMXServer. To include in an Application, need to create instance of JMXServer and start/stop in the application. what would be very nice would be for the JMXServer to extend AbstractService, and so share the same lifecycle. Any YARN composite service could simply add it as a child & hook it into its lifecycle Yes, good suggestion. I will update patch for this.
        Hide
        Vinayakumar B added a comment -

        Updated the patch with AbstractService implementation.

        Show
        Vinayakumar B added a comment - Updated the patch with AbstractService implementation.
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12627301/HADOOP-10136.patch
        against trunk revision .

        +1 @author. The patch does not contain any @author tags.

        +1 tests included. The patch appears to include 1 new or modified test files.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 javadoc. There were no new javadoc warning messages.

        +1 eclipse:eclipse. The patch built with eclipse:eclipse.

        -1 findbugs. The patch appears to introduce 1 new Findbugs (version 1.3.9) warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        +1 core tests. The patch passed unit tests in hadoop-common-project/hadoop-common.

        +1 contrib tests. The patch passed contrib unit tests.

        Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/3548//testReport/
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HADOOP-Build/3548//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-common.html
        Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/3548//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12627301/HADOOP-10136.patch against trunk revision . +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 1 new or modified test files. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . There were no new javadoc warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. -1 findbugs . The patch appears to introduce 1 new Findbugs (version 1.3.9) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. +1 core tests . The patch passed unit tests in hadoop-common-project/hadoop-common. +1 contrib tests . The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/3548//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-HADOOP-Build/3548//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-common.html Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/3548//console This message is automatically generated.
        Hide
        Vinayakumar B added a comment -

        Attaching latest patch fixing findbug

        Show
        Vinayakumar B added a comment - Attaching latest patch fixing findbug
        Hide
        Hadoop QA added a comment -

        +1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12627604/HADOOP-10136.patch
        against trunk revision .

        +1 @author. The patch does not contain any @author tags.

        +1 tests included. The patch appears to include 1 new or modified test files.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 javadoc. There were no new javadoc warning messages.

        +1 eclipse:eclipse. The patch built with eclipse:eclipse.

        +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        +1 core tests. The patch passed unit tests in hadoop-common-project/hadoop-common.

        +1 contrib tests. The patch passed contrib unit tests.

        Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/3549//testReport/
        Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/3549//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - +1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12627604/HADOOP-10136.patch against trunk revision . +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 1 new or modified test files. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . There were no new javadoc warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. +1 core tests . The patch passed unit tests in hadoop-common-project/hadoop-common. +1 contrib tests . The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/3549//testReport/ Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/3549//console This message is automatically generated.

          People

          • Assignee:
            Vinayakumar B
            Reporter:
            Vinayakumar B
          • Votes:
            0 Vote for this issue
            Watchers:
            7 Start watching this issue

            Dates

            • Created:
              Updated:

              Development