Whirr
  1. Whirr
  2. WHIRR-358

Enable remote JMX access for HBase

    Details

    • Type: Improvement Improvement
    • Status: Resolved
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.7.0
    • Component/s: service/hbase
    • Labels:
      None

      Description

      Enabling remote jmx access should make certain management tasks easier.

      1. WHIRR-358.patch
        6 kB
        Karel Vervaeke

        Issue Links

          Activity

          Hide
          Karel Vervaeke added a comment -

          This patch enables (passwordless) jmx remote access for the hbase master and regionservers (on ports 10101 and 10102 respectively).

          Show
          Karel Vervaeke added a comment - This patch enables (passwordless) jmx remote access for the hbase master and regionservers (on ports 10101 and 10102 respectively).
          Hide
          Andrei Savu added a comment -

          Looks good to me (not tested yet).

          Show
          Andrei Savu added a comment - Looks good to me (not tested yet).
          Hide
          Andrei Savu added a comment -

          Do you think we can add an integration test for this? (as part of the existing HBase integration tests)

          Show
          Andrei Savu added a comment - Do you think we can add an integration test for this? (as part of the existing HBase integration tests)
          Hide
          Karel Vervaeke added a comment -

          Sure. I can test the connection and the presence of the right MBeans. If there are any values worth testing I'll test those too.

          Show
          Karel Vervaeke added a comment - Sure. I can test the connection and the presence of the right MBeans. If there are any values worth testing I'll test those too.
          Hide
          Andrei Savu added a comment -

          IMHO it's enough to just check that we can make a JMX connection.

          Show
          Andrei Savu added a comment - IMHO it's enough to just check that we can make a JMX connection.
          Hide
          Karel Vervaeke added a comment -

          I started a new branch on github with a (IMO) better implementation:
          https://github.com/karel1980/whirr/tree/hbase-env
          (diff against trunk here: http://ubuntuone.com/p/1Aaf/)

          The new branch allows you to set 4 properties:
          hbase.master.java.opts
          hbase.master.fw.portsh
          base.regionserver.java.opts
          hbase.regionserver.fw.ports

          Via the *.java.opts properties you can't just set the jmx properties, but change memory usage as well.
          The *.fw.ports options allow opening additional ports on the firewall.

          Show
          Karel Vervaeke added a comment - I started a new branch on github with a (IMO) better implementation: https://github.com/karel1980/whirr/tree/hbase-env (diff against trunk here: http://ubuntuone.com/p/1Aaf/ ) The new branch allows you to set 4 properties: hbase.master.java.opts hbase.master.fw.portsh base.regionserver.java.opts hbase.regionserver.fw.ports Via the *.java.opts properties you can't just set the jmx properties, but change memory usage as well. The *.fw.ports options allow opening additional ports on the firewall.
          Hide
          Andrei Savu added a comment -

          hbase.master.fw.portsh

          hbase.regionserver.fw.ports

          Good idea but I think we should expose the firewall settings as a core Whirr feature not related to a single service. What do you think?

          Show
          Andrei Savu added a comment - hbase.master.fw.portsh hbase.regionserver.fw.ports Good idea but I think we should expose the firewall settings as a core Whirr feature not related to a single service. What do you think?
          Hide
          Karel Vervaeke added a comment -

          Absolutely a good idea. I just didn't want to blow up the scope

          Here's my suggestion:
          e.g. hbase.fw.rules=tcp/10101,tcp/-10102,udp/8650,80

          meaning:
          allow tcp traffic on 10101
          disallow tcp traffic on 10102 (overriding a clusteractionhandler's action)
          allow udp traffic on 8650
          allow tcp traffic on port 80 (tcp/ is the default and can be omitted)

          WDYT? Shall we create a separate ticket for this?

          Show
          Karel Vervaeke added a comment - Absolutely a good idea. I just didn't want to blow up the scope Here's my suggestion: e.g. hbase.fw.rules=tcp/10101,tcp/-10102,udp/8650,80 meaning: allow tcp traffic on 10101 disallow tcp traffic on 10102 (overriding a clusteractionhandler's action) allow udp traffic on 8650 allow tcp traffic on port 80 (tcp/ is the default and can be omitted) WDYT? Shall we create a separate ticket for this?
          Hide
          Karel Vervaeke added a comment -

          I did it again...
          hbase.fw.rules=... should be whirr.fw.rules.

          Show
          Karel Vervaeke added a comment - I did it again... hbase.fw.rules=... should be whirr.fw.rules.
          Hide
          Andrei Savu added a comment -

          Lets open another issue for this. I would go for something like whirr.firewall.rules - a bit more verbose but more clear.

          Show
          Andrei Savu added a comment - Lets open another issue for this. I would go for something like whirr.firewall.rules - a bit more verbose but more clear.
          Hide
          Karel Vervaeke added a comment -

          Created WHIRR-371 for implementing whirr.firewall.rules

          Show
          Karel Vervaeke added a comment - Created WHIRR-371 for implementing whirr.firewall.rules
          Hide
          Andrei Savu added a comment -

          JMX by default binds to the local interface. Is this the expected behavior in this case?

          [1] http://stackoverflow.com/questions/834581/remote-jmx-connection

          Show
          Andrei Savu added a comment - JMX by default binds to the local interface. Is this the expected behavior in this case? [1] http://stackoverflow.com/questions/834581/remote-jmx-connection
          Hide
          Karel Vervaeke added a comment -

          Are you sure? According to the second reply in that post, it binds to 0.0.0.0.
          IIRC I did have a problem connecting from a remote machine, and I ended using ssh -L 10101:localhost:10101 without further consideration to the issue.

          Honestly, I think solving WHIRR-368 or (as Tom pointed out) WHIRR-370 would be better than solving this bug. (But that doesn't solve the problem with configuring JMX for remote connections)

          Show
          Karel Vervaeke added a comment - Are you sure? According to the second reply in that post, it binds to 0.0.0.0. IIRC I did have a problem connecting from a remote machine, and I ended using ssh -L 10101:localhost:10101 without further consideration to the issue. Honestly, I think solving WHIRR-368 or (as Tom pointed out) WHIRR-370 would be better than solving this bug. (But that doesn't solve the problem with configuring JMX for remote connections)
          Hide
          Andrei Savu added a comment -

          From the second reply: "I think the content of this system property (java.rmi.server.hostname) is used somewhere at connection and compared with the actual IP address used by agent to communicate with jconsole. And if those address do not match, connection fails."

          I guess the JMX server is doing filtering after a connection is established. I will do more testing later today and try to add a simple unit test.

          Show
          Andrei Savu added a comment - From the second reply: "I think the content of this system property (java.rmi.server.hostname) is used somewhere at connection and compared with the actual IP address used by agent to communicate with jconsole. And if those address do not match, connection fails." I guess the JMX server is doing filtering after a connection is established. I will do more testing later today and try to add a simple unit test.
          Hide
          Karel Vervaeke added a comment -

          Had a look with wireshark. The value of java.rmi.server.hostname is sent to the client. I guess it's the client which decides to abort the connection when it doesn't match the hostname it used.

          The value of java.rmi.server.hostname can be anything really. As long as your client connects using that value for the hostname.

          The default value is java.rmi.server.hostname the ip address of the server (I guess that means whatever 'hostname -i' returns). For BYON, you get whatever is set in /etc/hosts for your hostname. For ec2, you get the internal ip address.

          Let's evaluate the possible values for java.rmi.server.hostname (with ec2 in mind):

          • internal hostname: Connecting from outside the cluster only works if you add an entry for it in /etc/hosts: {external ip address}

            .

          • internal ip address (==default): Connecting from outside the cluster won't work (unless do funky things with iptables locally)
          • external ip address: Connect using the external ip address
          • external hostname: Connect using the external hostname.

          What an annoying situation: you can't even use a http tunnel (because then the client would compare the received hostname with 'localhost')

          Show
          Karel Vervaeke added a comment - Had a look with wireshark. The value of java.rmi.server.hostname is sent to the client. I guess it's the client which decides to abort the connection when it doesn't match the hostname it used. The value of java.rmi.server.hostname can be anything really. As long as your client connects using that value for the hostname. The default value is java.rmi.server.hostname the ip address of the server (I guess that means whatever 'hostname -i' returns). For BYON, you get whatever is set in /etc/hosts for your hostname. For ec2, you get the internal ip address. Let's evaluate the possible values for java.rmi.server.hostname (with ec2 in mind): internal hostname: Connecting from outside the cluster only works if you add an entry for it in /etc/hosts: {external ip address} . internal ip address (==default): Connecting from outside the cluster won't work (unless do funky things with iptables locally) external ip address: Connect using the external ip address external hostname: Connect using the external hostname. What an annoying situation: you can't even use a http tunnel (because then the client would compare the received hostname with 'localhost')
          Hide
          Andrei Savu added a comment -

          I guess we should use the default value for java.rmi.server.hostname so that we can connect to JMX from Ganglia for monitoring.

          Show
          Andrei Savu added a comment - I guess we should use the default value for java.rmi.server.hostname so that we can connect to JMX from Ganglia for monitoring.
          Hide
          Andrei Savu added a comment -

          Karel any updates on this one? Are you guys using JMX? It looks like it applies cleanly on trunk.

          Show
          Andrei Savu added a comment - Karel any updates on this one? Are you guys using JMX? It looks like it applies cleanly on trunk.
          Hide
          Karel Vervaeke added a comment -

          We're not using this patch for enabling JMX. We use whirr run-script/parallel-ssh for enabling and disabling it during tests).
          I'm not a big fan of the attached patch. We should look at WHIRR-368 or WHIRR-370.
          I'll continue discussing it on mailing list.

          Show
          Karel Vervaeke added a comment - We're not using this patch for enabling JMX. We use whirr run-script/parallel-ssh for enabling and disabling it during tests). I'm not a big fan of the attached patch. We should look at WHIRR-368 or WHIRR-370 . I'll continue discussing it on mailing list.
          Hide
          Karel Vervaeke added a comment -

          I never got around to writing down my thoughts on the mailing list.
          Bottom line is that I don't think WHIRR-370 will be fixed short term. We should close this via WHIRR-368 for a short-term solution.

          Show
          Karel Vervaeke added a comment - I never got around to writing down my thoughts on the mailing list. Bottom line is that I don't think WHIRR-370 will be fixed short term. We should close this via WHIRR-368 for a short-term solution.
          Hide
          Andrei Savu added a comment -

          I agree. I'm planning to look into WHIRR-370 later this month but I can't provide a timeline for it.

          Show
          Andrei Savu added a comment - I agree. I'm planning to look into WHIRR-370 later this month but I can't provide a timeline for it.
          Hide
          Tom White added a comment -

          +1 for WHIRR-368 short-term, WHIRR-370 long-term.

          Show
          Tom White added a comment - +1 for WHIRR-368 short-term, WHIRR-370 long-term.
          Hide
          Karel Vervaeke added a comment -

          This is fixed via WHIRR-368.

          Show
          Karel Vervaeke added a comment - This is fixed via WHIRR-368 .

            People

            • Assignee:
              Unassigned
              Reporter:
              Karel Vervaeke
            • Votes:
              0 Vote for this issue
              Watchers:
              0 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development