HBase
  1. HBase
  2. HBASE-10692

The Multi TableMap job don't support the security HBase cluster

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Minor Minor
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.99.0, 0.98.3, 0.94.21
    • Component/s: mapreduce
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      HBASE-3996 adds the support of multiple tables and scanners as input to the mapper in map/reduce jobs. But it don't support the security HBase cluster.

      Eran Kutner Bryan Baugher

      Ps: HBASE-3996 only support multiple tables from the same HBase cluster. Should we support multiple tables from the different clusters?

      1. HBASE-10692-trunk-v2.diff
        2 kB
        Liu Shaohui
      2. HBASE-10692-0.94-v1.diff
        0.5 kB
        Liu Shaohui
      3. HBASE-10692-trunk-v1.diff
        0.6 kB
        Liu Shaohui

        Issue Links

          Activity

          Transition Time In Source Status Execution Times Last Executer Last Execution Date
          Open Open Patch Available Patch Available
          15h 40m 1 Ted Yu 06/Mar/14 18:07
          Patch Available Patch Available Resolved Resolved
          81d 23h 53m 1 Andrew Purtell 27/May/14 19:00
          Resolved Resolved Closed Closed
          44d 12h 25m 1 Lars Hofhansl 11/Jul/14 07:26
          Lars Hofhansl made changes -
          Status Resolved [ 5 ] Closed [ 6 ]
          Andrew Purtell made changes -
          Fix Version/s 0.98.3 [ 12326765 ]
          Fix Version/s 0.98.4 [ 12326810 ]
          Hide
          Hudson added a comment -

          FAILURE: Integrated in HBase-0.94-JDK7 #140 (See https://builds.apache.org/job/HBase-0.94-JDK7/140/)
          HBASE-10692 The Multi TableMap job don't support the security HBase cluster (Liu Shaohui) (apurtell: rev 0328fcd33f0e9d07c307f39db84ef4c6740cbd27)

          • src/main/java/org/apache/hadoop/hbase/mapreduce/TableMapReduceUtil.java
          Show
          Hudson added a comment - FAILURE: Integrated in HBase-0.94-JDK7 #140 (See https://builds.apache.org/job/HBase-0.94-JDK7/140/ ) HBASE-10692 The Multi TableMap job don't support the security HBase cluster (Liu Shaohui) (apurtell: rev 0328fcd33f0e9d07c307f39db84ef4c6740cbd27) src/main/java/org/apache/hadoop/hbase/mapreduce/TableMapReduceUtil.java
          Hide
          Hudson added a comment -

          SUCCESS: Integrated in HBase-0.94 #1372 (See https://builds.apache.org/job/HBase-0.94/1372/)
          HBASE-10692 The Multi TableMap job don't support the security HBase cluster (Liu Shaohui) (apurtell: rev 0328fcd33f0e9d07c307f39db84ef4c6740cbd27)

          • src/main/java/org/apache/hadoop/hbase/mapreduce/TableMapReduceUtil.java
          Show
          Hudson added a comment - SUCCESS: Integrated in HBase-0.94 #1372 (See https://builds.apache.org/job/HBase-0.94/1372/ ) HBASE-10692 The Multi TableMap job don't support the security HBase cluster (Liu Shaohui) (apurtell: rev 0328fcd33f0e9d07c307f39db84ef4c6740cbd27) src/main/java/org/apache/hadoop/hbase/mapreduce/TableMapReduceUtil.java
          Hide
          Hudson added a comment -

          SUCCESS: Integrated in HBase-0.94-security #488 (See https://builds.apache.org/job/HBase-0.94-security/488/)
          HBASE-10692 The Multi TableMap job don't support the security HBase cluster (Liu Shaohui) (apurtell: rev 0328fcd33f0e9d07c307f39db84ef4c6740cbd27)

          • src/main/java/org/apache/hadoop/hbase/mapreduce/TableMapReduceUtil.java
          Show
          Hudson added a comment - SUCCESS: Integrated in HBase-0.94-security #488 (See https://builds.apache.org/job/HBase-0.94-security/488/ ) HBASE-10692 The Multi TableMap job don't support the security HBase cluster (Liu Shaohui) (apurtell: rev 0328fcd33f0e9d07c307f39db84ef4c6740cbd27) src/main/java/org/apache/hadoop/hbase/mapreduce/TableMapReduceUtil.java
          Hide
          Hudson added a comment -

          SUCCESS: Integrated in HBase-TRUNK #5145 (See https://builds.apache.org/job/HBase-TRUNK/5145/)
          HBASE-10692 The Multi TableMap job don't support the security HBase cluster (Liu Shaohui) (apurtell: rev d9739b9e3f749d4190e5b8766dd0b5f6f6a9844c)

          • hbase-server/src/main/java/org/apache/hadoop/hbase/mapreduce/TableMapReduceUtil.java
          Show
          Hudson added a comment - SUCCESS: Integrated in HBase-TRUNK #5145 (See https://builds.apache.org/job/HBase-TRUNK/5145/ ) HBASE-10692 The Multi TableMap job don't support the security HBase cluster (Liu Shaohui) (apurtell: rev d9739b9e3f749d4190e5b8766dd0b5f6f6a9844c) hbase-server/src/main/java/org/apache/hadoop/hbase/mapreduce/TableMapReduceUtil.java
          Hide
          Hudson added a comment -

          SUCCESS: Integrated in HBase-0.98 #317 (See https://builds.apache.org/job/HBase-0.98/317/)
          HBASE-10692 The Multi TableMap job don't support the security HBase cluster (Liu Shaohui) (apurtell: rev d3ecf7523d0de335f0f95b36447c891365e18c6b)

          • hbase-server/src/main/java/org/apache/hadoop/hbase/mapreduce/TableMapReduceUtil.java
          Show
          Hudson added a comment - SUCCESS: Integrated in HBase-0.98 #317 (See https://builds.apache.org/job/HBase-0.98/317/ ) HBASE-10692 The Multi TableMap job don't support the security HBase cluster (Liu Shaohui) (apurtell: rev d3ecf7523d0de335f0f95b36447c891365e18c6b) hbase-server/src/main/java/org/apache/hadoop/hbase/mapreduce/TableMapReduceUtil.java
          Hide
          Hudson added a comment -

          FAILURE: Integrated in HBase-0.98-on-Hadoop-1.1 #298 (See https://builds.apache.org/job/HBase-0.98-on-Hadoop-1.1/298/)
          HBASE-10692 The Multi TableMap job don't support the security HBase cluster (Liu Shaohui) (apurtell: rev d3ecf7523d0de335f0f95b36447c891365e18c6b)

          • hbase-server/src/main/java/org/apache/hadoop/hbase/mapreduce/TableMapReduceUtil.java
          Show
          Hudson added a comment - FAILURE: Integrated in HBase-0.98-on-Hadoop-1.1 #298 (See https://builds.apache.org/job/HBase-0.98-on-Hadoop-1.1/298/ ) HBASE-10692 The Multi TableMap job don't support the security HBase cluster (Liu Shaohui) (apurtell: rev d3ecf7523d0de335f0f95b36447c891365e18c6b) hbase-server/src/main/java/org/apache/hadoop/hbase/mapreduce/TableMapReduceUtil.java
          Andrew Purtell made changes -
          Status Patch Available [ 10002 ] Resolved [ 5 ]
          Hadoop Flags Reviewed [ 10343 ]
          Resolution Fixed [ 1 ]
          Hide
          Andrew Purtell added a comment -

          Committed

          Show
          Andrew Purtell added a comment - Committed
          Andrew Purtell made changes -
          Fix Version/s 0.99.0 [ 12325675 ]
          Fix Version/s 0.94.21 [ 12326794 ]
          Fix Version/s 0.98.4 [ 12326810 ]
          Hide
          Andrew Purtell added a comment -

          Going to commit shortly to trunk, 0.98, and 0.94 unless objection

          Show
          Andrew Purtell added a comment - Going to commit shortly to trunk, 0.98, and 0.94 unless objection
          Hide
          Liu Shaohui added a comment -

          Lars Hofhansl
          Currently, it is impossible to add such test for the security HBase cluster.
          The patch is very obvious and I have tested it in our security cluster.
          Would you help to commit it? I lasted too long. Thanks.

          Show
          Liu Shaohui added a comment - Lars Hofhansl Currently, it is impossible to add such test for the security HBase cluster. The patch is very obvious and I have tested it in our security cluster. Would you help to commit it? I lasted too long. Thanks.
          Hide
          Lars Hofhansl added a comment -

          Looks good to me. If that's the whole patch. Let's commit.
          A test would be good. At the same time this clearly looks like a bug, that we can just fix.

          Show
          Lars Hofhansl added a comment - Looks good to me. If that's the whole patch. Let's commit. A test would be good. At the same time this clearly looks like a bug, that we can just fix.
          Hide
          Liu Shaohui added a comment -

          Ted Yu Nick Dimiduk Lars Hofhansl
          Please help to push this issue? Thanks.

          Show
          Liu Shaohui added a comment - Ted Yu Nick Dimiduk Lars Hofhansl Please help to push this issue? Thanks.
          Hide
          Liu Shaohui added a comment -

          After a hard try, i failed to enable hbase security with SIMPLE auth method in unit test env.

          Maybe I will add a test after HBASE-7781, but that issue does't have any progress.

          So I think if we can just let this issue go first and make multi table map jobs work with the security HBase cluster.

          I tested the patch in our security HBase cluster and It works fine.

          What's your opinions? Matteo Bertozzi Nick Dimiduk Andrew Purtell

          Show
          Liu Shaohui added a comment - After a hard try, i failed to enable hbase security with SIMPLE auth method in unit test env. Maybe I will add a test after HBASE-7781 , but that issue does't have any progress. So I think if we can just let this issue go first and make multi table map jobs work with the security HBase cluster. I tested the patch in our security HBase cluster and It works fine. What's your opinions? Matteo Bertozzi Nick Dimiduk Andrew Purtell
          Hide
          Liu Shaohui added a comment -

          Andrew Purtell
          Please figure out if I am wrong. I am not very familiar with security component.

          Show
          Liu Shaohui added a comment - Andrew Purtell Please figure out if I am wrong. I am not very familiar with security component.
          Hide
          Liu Shaohui added a comment -

          Andrew Purtell
          Thanks for your advice.
          But i found some conflicts in the code.
          RpcServer line #1455

                    if (isSecurityEnabled && authMethod == AuthMethod.SIMPLE) {
                      AccessControlException ae = new AccessControlException("Authentication is required");
                      setupResponse(authFailedResponse, authFailedCall, ae, ae.getMessage());
                      responder.doRespond(authFailedCall);
                      throw ae;
                    }
          

          It seems that authMethod can not be SIMPLE when security is enabled

          Show
          Liu Shaohui added a comment - Andrew Purtell Thanks for your advice. But i found some conflicts in the code. RpcServer line #1455 if (isSecurityEnabled && authMethod == AuthMethod.SIMPLE) { AccessControlException ae = new AccessControlException( "Authentication is required" ); setupResponse(authFailedResponse, authFailedCall, ae, ae.getMessage()); responder.doRespond(authFailedCall); throw ae; } It seems that authMethod can not be SIMPLE when security is enabled
          Liu Shaohui made changes -
          Link This issue depends upon HBASE-10774 [ HBASE-10774 ]
          Hide
          Andrew Purtell added a comment -

          You can enable security but specify an auth method of SIMPLE to avoid needing a KDC

          Show
          Andrew Purtell added a comment - You can enable security but specify an auth method of SIMPLE to avoid needing a KDC
          Hide
          Nick Dimiduk added a comment -

          I must admit I'm ignorant in the ways of our secure testing infrastructure. Andrew Purtell can you advise here?

          Show
          Nick Dimiduk added a comment - I must admit I'm ignorant in the ways of our secure testing infrastructure. Andrew Purtell can you advise here?
          Hide
          Liu Shaohui added a comment -

          Nick Dimiduk
          For isHBaseSecurityEnabled is false in HadoopSecurityEnabledUserProviderForTesting, TestSecure* test can not cover the issue of lack of initCredentials.

          On the other side, we can not set HBase security enabled, for SASL authentication requires a Kerberos ticket

          Show
          Liu Shaohui added a comment - Nick Dimiduk For isHBaseSecurityEnabled is false in HadoopSecurityEnabledUserProviderForTesting, TestSecure* test can not cover the issue of lack of initCredentials. On the other side, we can not set HBase security enabled, for SASL authentication requires a Kerberos ticket
          Hide
          Liu Shaohui added a comment -

          TestMultiTableInputFormat is removed in HBASE-9009.
          I will try to restore it, and then add the TestSecureMultiTableInputFormat.

          Show
          Liu Shaohui added a comment - TestMultiTableInputFormat is removed in HBASE-9009 . I will try to restore it, and then add the TestSecureMultiTableInputFormat.
          Hide
          Liu Shaohui added a comment -

          Thanks Nick Dimiduk.
          I will add a test following TestSecureExportSnapshot.

          Show
          Liu Shaohui added a comment - Thanks Nick Dimiduk . I will add a test following TestSecureExportSnapshot.
          Hide
          Nick Dimiduk added a comment -

          patch lgtm.

          Would be good to add test coverage. Secure tests aren't difficult to write, see TestSecureExportSnapshot for the necessary setup.

          Show
          Nick Dimiduk added a comment - patch lgtm. Would be good to add test coverage. Secure tests aren't difficult to write, see TestSecureExportSnapshot for the necessary setup.
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12633121/HBASE-10692-trunk-v2.diff
          against trunk revision .
          ATTACHMENT ID: 12633121

          +1 @author. The patch does not contain any @author tags.

          -1 tests included. The patch doesn't appear to include any new or modified tests.
          Please justify why no new tests are needed for this patch.
          Also please list what manual steps were performed to verify this patch.

          +1 hadoop1.0. The patch compiles against the hadoop 1.0 profile.

          +1 hadoop1.1. The patch compiles against the hadoop 1.1 profile.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 lineLengths. The patch does not introduce lines longer than 100

          +1 site. The mvn site goal succeeds with this patch.

          +1 core tests. The patch passed unit tests in .

          Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/8914//testReport/
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8914//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8914//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-thrift.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8914//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-client.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8914//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8914//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8914//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8914//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8914//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8914//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
          Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/8914//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12633121/HBASE-10692-trunk-v2.diff against trunk revision . ATTACHMENT ID: 12633121 +1 @author . The patch does not contain any @author tags. -1 tests included . The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 hadoop1.0 . The patch compiles against the hadoop 1.0 profile. +1 hadoop1.1 . The patch compiles against the hadoop 1.1 profile. +1 javadoc . The javadoc tool did not generate any warning messages. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 findbugs . The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. +1 lineLengths . The patch does not introduce lines longer than 100 +1 site . The mvn site goal succeeds with this patch. +1 core tests . The patch passed unit tests in . Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/8914//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8914//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8914//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-thrift.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8914//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-client.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8914//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8914//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8914//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8914//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8914//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8914//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/8914//console This message is automatically generated.
          Ted Yu made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          Liu Shaohui made changes -
          Attachment HBASE-10692-trunk-v2.diff [ 12633121 ]
          Hide
          Liu Shaohui added a comment -

          Add an argument initCredentials Matteo Bertozzi. THX

          Show
          Liu Shaohui added a comment - Add an argument initCredentials Matteo Bertozzi . THX
          Hide
          Matteo Bertozzi added a comment -

          Liu Shaohui thanks.
          since the multicluster is not in the scope of this patch, +1 for me

          only one minor note, the other initTableMapperJob() has an argument initCredentials, should we keep the same style?
          https://github.com/apache/hbase/blob/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/mapreduce/TableMapReduceUtil.java#L193

          Show
          Matteo Bertozzi added a comment - Liu Shaohui thanks. since the multicluster is not in the scope of this patch, +1 for me only one minor note, the other initTableMapperJob() has an argument initCredentials, should we keep the same style? https://github.com/apache/hbase/blob/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/mapreduce/TableMapReduceUtil.java#L193
          Hide
          Liu Shaohui added a comment -

          Matteo Bertozzi
          The patch just make the multi table-Map job support the security HBase cluster. And those tables should be at the same cluster, so we only see a single conf object for the mappers.

          I am not sure if we should support multiple tables from the different clusters. It's just a discussion.
          If it is needed, i will open another issue to implement it.

          Show
          Liu Shaohui added a comment - Matteo Bertozzi The patch just make the multi table-Map job support the security HBase cluster. And those tables should be at the same cluster, so we only see a single conf object for the mappers. I am not sure if we should support multiple tables from the different clusters. It's just a discussion. If it is needed, i will open another issue to implement it.
          Hide
          Matteo Bertozzi added a comment -

          Liu Shaohui sorry for the dumb question, but where do you specify the different "hbase.zookeeper.quorum" based on the table/cluster that you want to use? I see only a single conf object

          Show
          Matteo Bertozzi added a comment - Liu Shaohui sorry for the dumb question, but where do you specify the different "hbase.zookeeper.quorum" based on the table/cluster that you want to use? I see only a single conf object
          Hide
          Liu Shaohui added a comment -

          Lars Hofhansl
          Test it in 0.94 security cluster using a user case.
          No test in trunk security cluster for we don't have a security cluster whose version > 0.95.
          It's not easy to write an unit test for it.
          May someone else help to test it ? Thx

          Show
          Liu Shaohui added a comment - Lars Hofhansl Test it in 0.94 security cluster using a user case. No test in trunk security cluster for we don't have a security cluster whose version > 0.95. It's not easy to write an unit test for it. May someone else help to test it ? Thx
          Hide
          Lars Hofhansl added a comment -

          Heh, I was just looking at this yesterday and wondered whether this will work with a secure cluster.
          Nice find. Did you test it?

          Show
          Lars Hofhansl added a comment - Heh, I was just looking at this yesterday and wondered whether this will work with a secure cluster. Nice find. Did you test it?
          Liu Shaohui made changes -
          Assignee Liu Shaohui [ liushaohui ]
          Liu Shaohui made changes -
          Attachment HBASE-10692-0.94-v1.diff [ 12633018 ]
          Hide
          Liu Shaohui added a comment -

          Patch for 0.94 Lars Hofhansl

          Show
          Liu Shaohui added a comment - Patch for 0.94 Lars Hofhansl
          Hide
          Liu Shaohui added a comment -

          typo: oatch -> patch

          Show
          Liu Shaohui added a comment - typo: oatch -> patch
          Liu Shaohui made changes -
          Field Original Value New Value
          Attachment HBASE-10692-trunk-v1.diff [ 12633011 ]
          Hide
          Liu Shaohui added a comment -

          Simple oatch for the trunk

          Show
          Liu Shaohui added a comment - Simple oatch for the trunk
          Liu Shaohui created issue -

            People

            • Assignee:
              Liu Shaohui
              Reporter:
              Liu Shaohui
            • Votes:
              0 Vote for this issue
              Watchers:
              9 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development