Uploaded image for project: 'Sqoop'
  1. Sqoop
  2. SQOOP-1038

Add support for composite keys in HBase import

    Details

    • Type: New Feature
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 1.4.3
    • Fix Version/s: 1.4.4
    • Component/s: None
    • Labels:
      None

      Description

      Sqoop does not support importing data into hbase when table contains composite key.
      Ref: https://issues.cloudera.org/browse/SQOOP-107

      1. HBase sqoop import for table with composite key.docx
        18 kB
        Shruti Joshi
      2. sqoop_trunk.patch
        8 kB
        Shruti Joshi
      3. sqoop_trunk1.patch
        10 kB
        Shruti Joshi

        Issue Links

          Activity

          Hide
          shrutijoshi Shruti Joshi added a comment -

          I am unable to assign this JIRA to myself.. But I am working on this JIRA..

          Show
          shrutijoshi Shruti Joshi added a comment - I am unable to assign this JIRA to myself.. But I am working on this JIRA..
          Hide
          jarcec Jarek Jarcec Cecho added a comment -

          Assigned to Shruti Joshi.

          Show
          jarcec Jarek Jarcec Cecho added a comment - Assigned to Shruti Joshi .
          Hide
          jarcec Jarek Jarcec Cecho added a comment -

          Hi Shruti Joshi,
          the proposal seems fine to me. Would you mind uploading your patch to review board for additional review?

          Jarcec

          Show
          jarcec Jarek Jarcec Cecho added a comment - Hi Shruti Joshi , the proposal seems fine to me. Would you mind uploading your patch to review board for additional review? Jarcec
          Hide
          shrutijoshi Shruti Joshi added a comment -

          Hi Jarek,
          Thanks.. I have already uploaded the patch on review board..
          https://reviews.apache.org/r/11041/
          This is the link..

          Shruti.

          Show
          shrutijoshi Shruti Joshi added a comment - Hi Jarek, Thanks.. I have already uploaded the patch on review board.. https://reviews.apache.org/r/11041/ This is the link.. Shruti.
          Hide
          shrutijoshi Shruti Joshi added a comment -

          Hi Jarek,

          I have done following changes in code:
          1) While performing HBase import in Sqoop, if the row-key column or the column-family is null, a warning message will be logged and an IOException will be thrown to indicate failure.

          2) To support composite-key table import in HBase, user will specify comma separated list of keys. The ‘isCompositeKey’ method will check if the specified row-key is a composite key.

          3) The logic to write record data in HBase does not vary for composite-key or a single key table and hence I have created a new method to insert data in HBase which will be common for both the cases.
          (Please verify the method putRecordInHBase())

          4) Added a new method getCompositeKey() to construct hbase row-key from attribute values belonging to the composite key.

          5) Tested the code with both single and composite key tables. Also ran ‘ant checkstyle’ which results in successful build.

          Show
          shrutijoshi Shruti Joshi added a comment - Hi Jarek, I have done following changes in code: 1) While performing HBase import in Sqoop, if the row-key column or the column-family is null, a warning message will be logged and an IOException will be thrown to indicate failure. 2) To support composite-key table import in HBase, user will specify comma separated list of keys. The ‘isCompositeKey’ method will check if the specified row-key is a composite key. 3) The logic to write record data in HBase does not vary for composite-key or a single key table and hence I have created a new method to insert data in HBase which will be common for both the cases. (Please verify the method putRecordInHBase()) 4) Added a new method getCompositeKey() to construct hbase row-key from attribute values belonging to the composite key. 5) Tested the code with both single and composite key tables. Also ran ‘ant checkstyle’ which results in successful build.
          Hide
          shrutijoshi Shruti Joshi added a comment -

          Hi Jarek,
          I am done with writing HBase test-case for composite key support.
          Please refer the review board for the patch.
          Shruti.

          Show
          shrutijoshi Shruti Joshi added a comment - Hi Jarek, I am done with writing HBase test-case for composite key support. Please refer the review board for the patch. Shruti.
          Hide
          jarcec Jarek Jarcec Cecho added a comment -

          Hi Shruti Joshi,
          thank you! I've left couple of additional comments on the review board.

          Jarcec

          Show
          jarcec Jarek Jarcec Cecho added a comment - Hi Shruti Joshi , thank you! I've left couple of additional comments on the review board. Jarcec
          Hide
          shrutijoshi Shruti Joshi added a comment -

          Hi Jarek,

          I have uploaded the patch again with all the required changes and suggestions incorporated.
          Please refer the same on the Review Board.

          Shruti

          Show
          shrutijoshi Shruti Joshi added a comment - Hi Jarek, I have uploaded the patch again with all the required changes and suggestions incorporated. Please refer the same on the Review Board. Shruti
          Hide
          jira-bot ASF subversion and git services added a comment -

          Commit a2a02076a9c94f344332864471894fa866c11806 in branch refs/heads/trunk from Jarek Jarcec Cecho
          [ https://git-wip-us.apache.org/repos/asf?p=sqoop.git;h=a2a0207 ]

          SQOOP-1038: Add support for composite keys in HBase import

          (Shruti Joshi via Jarek Jarcec Cecho)

          Show
          jira-bot ASF subversion and git services added a comment - Commit a2a02076a9c94f344332864471894fa866c11806 in branch refs/heads/trunk from Jarek Jarcec Cecho [ https://git-wip-us.apache.org/repos/asf?p=sqoop.git;h=a2a0207 ] SQOOP-1038 : Add support for composite keys in HBase import (Shruti Joshi via Jarek Jarcec Cecho)
          Hide
          jarcec Jarek Jarcec Cecho added a comment -

          Thank you Shruti for your contribution, greatly appreciated!

          Show
          jarcec Jarek Jarcec Cecho added a comment - Thank you Shruti for your contribution, greatly appreciated!
          Hide
          hudson Hudson added a comment -

          Integrated in Sqoop-ant-jdk-1.6-hadoop23 #950 (See https://builds.apache.org/job/Sqoop-ant-jdk-1.6-hadoop23/950/)
          SQOOP-1038: Add support for composite keys in HBase import (Revision a2a02076a9c94f344332864471894fa866c11806)

          Result = SUCCESS
          jarcec : https://git-wip-us.apache.org/repos/asf?p=sqoop.git&a=commit&h=a2a02076a9c94f344332864471894fa866c11806
          Files :

          • src/test/com/cloudera/sqoop/hbase/HBaseImportAddRowKeyTest.java
          • src/java/org/apache/sqoop/hbase/ToStringPutTransformer.java
          • src/docs/man/hbase-args.txt
          • src/docs/user/hbase-args.txt
          • src/java/org/apache/sqoop/hbase/HBasePutProcessor.java
          • src/docs/user/hbase.txt
          Show
          hudson Hudson added a comment - Integrated in Sqoop-ant-jdk-1.6-hadoop23 #950 (See https://builds.apache.org/job/Sqoop-ant-jdk-1.6-hadoop23/950/ ) SQOOP-1038 : Add support for composite keys in HBase import (Revision a2a02076a9c94f344332864471894fa866c11806) Result = SUCCESS jarcec : https://git-wip-us.apache.org/repos/asf?p=sqoop.git&a=commit&h=a2a02076a9c94f344332864471894fa866c11806 Files : src/test/com/cloudera/sqoop/hbase/HBaseImportAddRowKeyTest.java src/java/org/apache/sqoop/hbase/ToStringPutTransformer.java src/docs/man/hbase-args.txt src/docs/user/hbase-args.txt src/java/org/apache/sqoop/hbase/HBasePutProcessor.java src/docs/user/hbase.txt
          Hide
          hudson Hudson added a comment -

          Integrated in Sqoop-ant-jdk-1.6-hadoop100 #724 (See https://builds.apache.org/job/Sqoop-ant-jdk-1.6-hadoop100/724/)
          SQOOP-1038: Add support for composite keys in HBase import (Revision a2a02076a9c94f344332864471894fa866c11806)

          Result = SUCCESS
          jarcec : https://git-wip-us.apache.org/repos/asf?p=sqoop.git&a=commit&h=a2a02076a9c94f344332864471894fa866c11806
          Files :

          • src/docs/user/hbase-args.txt
          • src/java/org/apache/sqoop/hbase/ToStringPutTransformer.java
          • src/test/com/cloudera/sqoop/hbase/HBaseImportAddRowKeyTest.java
          • src/docs/man/hbase-args.txt
          • src/java/org/apache/sqoop/hbase/HBasePutProcessor.java
          • src/docs/user/hbase.txt
          Show
          hudson Hudson added a comment - Integrated in Sqoop-ant-jdk-1.6-hadoop100 #724 (See https://builds.apache.org/job/Sqoop-ant-jdk-1.6-hadoop100/724/ ) SQOOP-1038 : Add support for composite keys in HBase import (Revision a2a02076a9c94f344332864471894fa866c11806) Result = SUCCESS jarcec : https://git-wip-us.apache.org/repos/asf?p=sqoop.git&a=commit&h=a2a02076a9c94f344332864471894fa866c11806 Files : src/docs/user/hbase-args.txt src/java/org/apache/sqoop/hbase/ToStringPutTransformer.java src/test/com/cloudera/sqoop/hbase/HBaseImportAddRowKeyTest.java src/docs/man/hbase-args.txt src/java/org/apache/sqoop/hbase/HBasePutProcessor.java src/docs/user/hbase.txt
          Hide
          hudson Hudson added a comment -

          Integrated in Sqoop-ant-jdk-1.6-hadoop200 #761 (See https://builds.apache.org/job/Sqoop-ant-jdk-1.6-hadoop200/761/)
          SQOOP-1038: Add support for composite keys in HBase import (Revision a2a02076a9c94f344332864471894fa866c11806)

          Result = SUCCESS
          jarcec : https://git-wip-us.apache.org/repos/asf?p=sqoop.git&a=commit&h=a2a02076a9c94f344332864471894fa866c11806
          Files :

          • src/test/com/cloudera/sqoop/hbase/HBaseImportAddRowKeyTest.java
          • src/java/org/apache/sqoop/hbase/ToStringPutTransformer.java
          • src/docs/user/hbase-args.txt
          • src/java/org/apache/sqoop/hbase/HBasePutProcessor.java
          • src/docs/user/hbase.txt
          • src/docs/man/hbase-args.txt
          Show
          hudson Hudson added a comment - Integrated in Sqoop-ant-jdk-1.6-hadoop200 #761 (See https://builds.apache.org/job/Sqoop-ant-jdk-1.6-hadoop200/761/ ) SQOOP-1038 : Add support for composite keys in HBase import (Revision a2a02076a9c94f344332864471894fa866c11806) Result = SUCCESS jarcec : https://git-wip-us.apache.org/repos/asf?p=sqoop.git&a=commit&h=a2a02076a9c94f344332864471894fa866c11806 Files : src/test/com/cloudera/sqoop/hbase/HBaseImportAddRowKeyTest.java src/java/org/apache/sqoop/hbase/ToStringPutTransformer.java src/docs/user/hbase-args.txt src/java/org/apache/sqoop/hbase/HBasePutProcessor.java src/docs/user/hbase.txt src/docs/man/hbase-args.txt
          Hide
          hudson Hudson added a comment -

          Integrated in Sqoop-ant-jdk-1.6-hadoop20 #744 (See https://builds.apache.org/job/Sqoop-ant-jdk-1.6-hadoop20/744/)
          SQOOP-1038: Add support for composite keys in HBase import (Revision a2a02076a9c94f344332864471894fa866c11806)

          Result = SUCCESS
          jarcec : https://git-wip-us.apache.org/repos/asf?p=sqoop.git&a=commit&h=a2a02076a9c94f344332864471894fa866c11806
          Files :

          • src/java/org/apache/sqoop/hbase/HBasePutProcessor.java
          • src/docs/user/hbase.txt
          • src/test/com/cloudera/sqoop/hbase/HBaseImportAddRowKeyTest.java
          • src/docs/user/hbase-args.txt
          • src/java/org/apache/sqoop/hbase/ToStringPutTransformer.java
          • src/docs/man/hbase-args.txt
          Show
          hudson Hudson added a comment - Integrated in Sqoop-ant-jdk-1.6-hadoop20 #744 (See https://builds.apache.org/job/Sqoop-ant-jdk-1.6-hadoop20/744/ ) SQOOP-1038 : Add support for composite keys in HBase import (Revision a2a02076a9c94f344332864471894fa866c11806) Result = SUCCESS jarcec : https://git-wip-us.apache.org/repos/asf?p=sqoop.git&a=commit&h=a2a02076a9c94f344332864471894fa866c11806 Files : src/java/org/apache/sqoop/hbase/HBasePutProcessor.java src/docs/user/hbase.txt src/test/com/cloudera/sqoop/hbase/HBaseImportAddRowKeyTest.java src/docs/user/hbase-args.txt src/java/org/apache/sqoop/hbase/ToStringPutTransformer.java src/docs/man/hbase-args.txt

            People

            • Assignee:
              shrutijoshi Shruti Joshi
              Reporter:
              shrutijoshi Shruti Joshi
            • Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development