Uploaded image for project: 'Sqoop'
  1. Sqoop
  2. SQOOP-1032

Add the --bulk-load-dir option to support the HBase doBulkLoad function

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 1.4.3
    • Fix Version/s: 1.4.5
    • Component/s: hbase-integration
    • Labels:
    • Environment:

      Linux

      Description

      HBase supply the LoadIncrementalHFiles.LoadIncrementalHFiles method for bulk load, so this feature added with --bulk-load-dir option to support the HBase doBulkLoad function

      1. SQOOP-1032.patch.2
        17 kB
        Zhancheng Deng
      2. SQOOP-1032.path
        18 kB
        Zhancheng Deng
      3. SQOOP-1032-20131010.patch
        20 kB
        Alexandre Normand

        Issue Links

          Activity

          Hide
          zhan8610189 Zhancheng Deng added a comment -

          the patch attached

          Show
          zhan8610189 Zhancheng Deng added a comment - the patch attached
          Hide
          jarcec Jarek Jarcec Cecho added a comment -

          Assigning to Zhancheng Deng.

          Show
          jarcec Jarek Jarcec Cecho added a comment - Assigning to Zhancheng Deng .
          Hide
          jarcec Jarek Jarcec Cecho added a comment -

          Zhancheng Deng, thank you for working on this JIRA, would you mind uploading your patch to the review board?

          Show
          jarcec Jarek Jarcec Cecho added a comment - Zhancheng Deng , thank you for working on this JIRA, would you mind uploading your patch to the review board ?
          Hide
          vasanthkumar Vasanth kumar RJ added a comment -

          Please upload patch to review board and get approvals/comments from reviewers. Once committed, I can include this in version 1.4.4.

          Show
          vasanthkumar Vasanth kumar RJ added a comment - Please upload patch to review board and get approvals/comments from reviewers. Once committed, I can include this in version 1.4.4.
          Hide
          zhan8610189 Zhancheng Deng added a comment - - edited

          hi, Vasanth and Jarek:

          sorry, i am a novice to the jira and the review board.

          OK, i upload the patch to the review board, but error happened:

          The file 'src/java/com/cloudera/sqoop/hbase/HBasePutProcessor.java' (rf1d4e99) could not be found in the repository

          i think i got the wrong version of sqoop on the github before long.
          you can tell me the right repository, and i checkout and repatch it.

          thanks very much

          Zhancheng

          Show
          zhan8610189 Zhancheng Deng added a comment - - edited hi, Vasanth and Jarek: sorry, i am a novice to the jira and the review board. OK, i upload the patch to the review board, but error happened: The file 'src/java/com/cloudera/sqoop/hbase/HBasePutProcessor.java' (rf1d4e99) could not be found in the repository i think i got the wrong version of sqoop on the github before long. you can tell me the right repository, and i checkout and repatch it. thanks very much Zhancheng
          Hide
          jarcec Jarek Jarcec Cecho added a comment -

          Hi Zhancheng Deng,
          you can check out the git repository directly from Apache, there is no need to take the mirror from Github:

          git clone https://git-wip-us.apache.org/repos/asf/sqoop.git
          
          Show
          jarcec Jarek Jarcec Cecho added a comment - Hi Zhancheng Deng , you can check out the git repository directly from Apache, there is no need to take the mirror from Github: git clone https: //git-wip-us.apache.org/repos/asf/sqoop.git
          Hide
          zhan8610189 Zhancheng Deng added a comment -

          modify the patch of the correct repository

          Show
          zhan8610189 Zhancheng Deng added a comment - modify the patch of the correct repository
          Hide
          zhan8610189 Zhancheng Deng added a comment -

          review board is committed

          Show
          zhan8610189 Zhancheng Deng added a comment - review board is committed
          Hide
          alexandre.normand Alexandre Normand added a comment -

          I'd really like to use bulk loading with sqoop. Is there a chance this could get reviewed/committed soon? I was going to handle it by sqooping to a sequence file than having a second job convert to hfiles for bulk loading but this looks like a better way.

          Show
          alexandre.normand Alexandre Normand added a comment - I'd really like to use bulk loading with sqoop. Is there a chance this could get reviewed/committed soon? I was going to handle it by sqooping to a sequence file than having a second job convert to hfiles for bulk loading but this looks like a better way.
          Hide
          alexandre.normand Alexandre Normand added a comment -

          I tried to address some of the comments on the original review request but since I can't add a diff to that one, I created a new request on reviewboard: https://reviews.apache.org/r/14240/

          Show
          alexandre.normand Alexandre Normand added a comment - I tried to address some of the comments on the original review request but since I can't add a diff to that one, I created a new request on reviewboard: https://reviews.apache.org/r/14240/
          Hide
          alexandre.normand Alexandre Normand added a comment -

          Here's the final patch as reviewed in https://reviews.apache.org/r/14240/.

          Thanks!

          Show
          alexandre.normand Alexandre Normand added a comment - Here's the final patch as reviewed in https://reviews.apache.org/r/14240/ . Thanks!
          Hide
          jira-bot ASF subversion and git services added a comment -

          Commit ddb81e185be72c7530498c379e6ad45e6d54a2d6 in branch refs/heads/trunk from Jarek Jarcec Cecho
          [ https://git-wip-us.apache.org/repos/asf?p=sqoop.git;h=ddb81e1 ]

          SQOOP-1032: Add the --bulk-load-dir option to support the HBase doBulkLoad function

          (Alexandre Normand via Jarek Jarcec Cecho)

          Show
          jira-bot ASF subversion and git services added a comment - Commit ddb81e185be72c7530498c379e6ad45e6d54a2d6 in branch refs/heads/trunk from Jarek Jarcec Cecho [ https://git-wip-us.apache.org/repos/asf?p=sqoop.git;h=ddb81e1 ] SQOOP-1032 : Add the --bulk-load-dir option to support the HBase doBulkLoad function (Alexandre Normand via Jarek Jarcec Cecho)
          Hide
          jarcec Jarek Jarcec Cecho added a comment -

          The patch is in, thank you Alexandre Normand for your time and effort!

          Show
          jarcec Jarek Jarcec Cecho added a comment - The patch is in, thank you Alexandre Normand for your time and effort!
          Hide
          hudson Hudson added a comment -

          SUCCESS: Integrated in Sqoop-ant-jdk-1.6-hadoop200 #862 (See https://builds.apache.org/job/Sqoop-ant-jdk-1.6-hadoop200/862/)
          SQOOP-1032: Add the --bulk-load-dir option to support the HBase doBulkLoad function (jarcec: https://git-wip-us.apache.org/repos/asf?p=sqoop.git&a=commit&h=ddb81e185be72c7530498c379e6ad45e6d54a2d6)

          • src/java/org/apache/sqoop/manager/SqlManager.java
          • src/java/org/apache/sqoop/SqoopOptions.java
          • src/java/org/apache/sqoop/tool/BaseSqoopTool.java
          • src/java/org/apache/sqoop/mapreduce/ImportJobBase.java
          • src/java/org/apache/sqoop/mapreduce/HBaseBulkImportMapper.java
          • src/test/com/cloudera/sqoop/TestSqoopOptions.java
          • src/java/org/apache/sqoop/hbase/ToStringPutTransformer.java
          • src/java/org/apache/sqoop/hbase/HBasePutProcessor.java
          • src/java/org/apache/sqoop/mapreduce/HBaseBulkImportJob.java
          • src/docs/user/hbase-args.txt
          • src/docs/user/hbase.txt
          Show
          hudson Hudson added a comment - SUCCESS: Integrated in Sqoop-ant-jdk-1.6-hadoop200 #862 (See https://builds.apache.org/job/Sqoop-ant-jdk-1.6-hadoop200/862/ ) SQOOP-1032 : Add the --bulk-load-dir option to support the HBase doBulkLoad function (jarcec: https://git-wip-us.apache.org/repos/asf?p=sqoop.git&a=commit&h=ddb81e185be72c7530498c379e6ad45e6d54a2d6 ) src/java/org/apache/sqoop/manager/SqlManager.java src/java/org/apache/sqoop/SqoopOptions.java src/java/org/apache/sqoop/tool/BaseSqoopTool.java src/java/org/apache/sqoop/mapreduce/ImportJobBase.java src/java/org/apache/sqoop/mapreduce/HBaseBulkImportMapper.java src/test/com/cloudera/sqoop/TestSqoopOptions.java src/java/org/apache/sqoop/hbase/ToStringPutTransformer.java src/java/org/apache/sqoop/hbase/HBasePutProcessor.java src/java/org/apache/sqoop/mapreduce/HBaseBulkImportJob.java src/docs/user/hbase-args.txt src/docs/user/hbase.txt
          Hide
          hudson Hudson added a comment -

          SUCCESS: Integrated in Sqoop-ant-jdk-1.6-hadoop20 #857 (See https://builds.apache.org/job/Sqoop-ant-jdk-1.6-hadoop20/857/)
          SQOOP-1032: Add the --bulk-load-dir option to support the HBase doBulkLoad function (jarcec: https://git-wip-us.apache.org/repos/asf?p=sqoop.git&a=commit&h=ddb81e185be72c7530498c379e6ad45e6d54a2d6)

          • src/docs/user/hbase-args.txt
          • src/java/org/apache/sqoop/manager/SqlManager.java
          • src/docs/user/hbase.txt
          • src/test/com/cloudera/sqoop/TestSqoopOptions.java
          • src/java/org/apache/sqoop/mapreduce/ImportJobBase.java
          • src/java/org/apache/sqoop/mapreduce/HBaseBulkImportJob.java
          • src/java/org/apache/sqoop/hbase/ToStringPutTransformer.java
          • src/java/org/apache/sqoop/hbase/HBasePutProcessor.java
          • src/java/org/apache/sqoop/SqoopOptions.java
          • src/java/org/apache/sqoop/tool/BaseSqoopTool.java
          • src/java/org/apache/sqoop/mapreduce/HBaseBulkImportMapper.java
          Show
          hudson Hudson added a comment - SUCCESS: Integrated in Sqoop-ant-jdk-1.6-hadoop20 #857 (See https://builds.apache.org/job/Sqoop-ant-jdk-1.6-hadoop20/857/ ) SQOOP-1032 : Add the --bulk-load-dir option to support the HBase doBulkLoad function (jarcec: https://git-wip-us.apache.org/repos/asf?p=sqoop.git&a=commit&h=ddb81e185be72c7530498c379e6ad45e6d54a2d6 ) src/docs/user/hbase-args.txt src/java/org/apache/sqoop/manager/SqlManager.java src/docs/user/hbase.txt src/test/com/cloudera/sqoop/TestSqoopOptions.java src/java/org/apache/sqoop/mapreduce/ImportJobBase.java src/java/org/apache/sqoop/mapreduce/HBaseBulkImportJob.java src/java/org/apache/sqoop/hbase/ToStringPutTransformer.java src/java/org/apache/sqoop/hbase/HBasePutProcessor.java src/java/org/apache/sqoop/SqoopOptions.java src/java/org/apache/sqoop/tool/BaseSqoopTool.java src/java/org/apache/sqoop/mapreduce/HBaseBulkImportMapper.java
          Hide
          hudson Hudson added a comment -

          SUCCESS: Integrated in Sqoop-ant-jdk-1.6-hadoop23 #1059 (See https://builds.apache.org/job/Sqoop-ant-jdk-1.6-hadoop23/1059/)
          SQOOP-1032: Add the --bulk-load-dir option to support the HBase doBulkLoad function (jarcec: https://git-wip-us.apache.org/repos/asf?p=sqoop.git&a=commit&h=ddb81e185be72c7530498c379e6ad45e6d54a2d6)

          • src/docs/user/hbase.txt
          • src/test/com/cloudera/sqoop/TestSqoopOptions.java
          • src/java/org/apache/sqoop/mapreduce/HBaseBulkImportMapper.java
          • src/docs/user/hbase-args.txt
          • src/java/org/apache/sqoop/hbase/HBasePutProcessor.java
          • src/java/org/apache/sqoop/tool/BaseSqoopTool.java
          • src/java/org/apache/sqoop/mapreduce/ImportJobBase.java
          • src/java/org/apache/sqoop/mapreduce/HBaseBulkImportJob.java
          • src/java/org/apache/sqoop/manager/SqlManager.java
          • src/java/org/apache/sqoop/SqoopOptions.java
          • src/java/org/apache/sqoop/hbase/ToStringPutTransformer.java
          Show
          hudson Hudson added a comment - SUCCESS: Integrated in Sqoop-ant-jdk-1.6-hadoop23 #1059 (See https://builds.apache.org/job/Sqoop-ant-jdk-1.6-hadoop23/1059/ ) SQOOP-1032 : Add the --bulk-load-dir option to support the HBase doBulkLoad function (jarcec: https://git-wip-us.apache.org/repos/asf?p=sqoop.git&a=commit&h=ddb81e185be72c7530498c379e6ad45e6d54a2d6 ) src/docs/user/hbase.txt src/test/com/cloudera/sqoop/TestSqoopOptions.java src/java/org/apache/sqoop/mapreduce/HBaseBulkImportMapper.java src/docs/user/hbase-args.txt src/java/org/apache/sqoop/hbase/HBasePutProcessor.java src/java/org/apache/sqoop/tool/BaseSqoopTool.java src/java/org/apache/sqoop/mapreduce/ImportJobBase.java src/java/org/apache/sqoop/mapreduce/HBaseBulkImportJob.java src/java/org/apache/sqoop/manager/SqlManager.java src/java/org/apache/sqoop/SqoopOptions.java src/java/org/apache/sqoop/hbase/ToStringPutTransformer.java
          Hide
          hudson Hudson added a comment -

          SUCCESS: Integrated in Sqoop-ant-jdk-1.6-hadoop100 #820 (See https://builds.apache.org/job/Sqoop-ant-jdk-1.6-hadoop100/820/)
          SQOOP-1032: Add the --bulk-load-dir option to support the HBase doBulkLoad function (jarcec: https://git-wip-us.apache.org/repos/asf?p=sqoop.git&a=commit&h=ddb81e185be72c7530498c379e6ad45e6d54a2d6)

          • src/java/org/apache/sqoop/tool/BaseSqoopTool.java
          • src/java/org/apache/sqoop/hbase/ToStringPutTransformer.java
          • src/java/org/apache/sqoop/hbase/HBasePutProcessor.java
          • src/java/org/apache/sqoop/manager/SqlManager.java
          • src/docs/user/hbase-args.txt
          • src/java/org/apache/sqoop/mapreduce/HBaseBulkImportMapper.java
          • src/test/com/cloudera/sqoop/TestSqoopOptions.java
          • src/java/org/apache/sqoop/mapreduce/ImportJobBase.java
          • src/java/org/apache/sqoop/SqoopOptions.java
          • src/java/org/apache/sqoop/mapreduce/HBaseBulkImportJob.java
          • src/docs/user/hbase.txt
          Show
          hudson Hudson added a comment - SUCCESS: Integrated in Sqoop-ant-jdk-1.6-hadoop100 #820 (See https://builds.apache.org/job/Sqoop-ant-jdk-1.6-hadoop100/820/ ) SQOOP-1032 : Add the --bulk-load-dir option to support the HBase doBulkLoad function (jarcec: https://git-wip-us.apache.org/repos/asf?p=sqoop.git&a=commit&h=ddb81e185be72c7530498c379e6ad45e6d54a2d6 ) src/java/org/apache/sqoop/tool/BaseSqoopTool.java src/java/org/apache/sqoop/hbase/ToStringPutTransformer.java src/java/org/apache/sqoop/hbase/HBasePutProcessor.java src/java/org/apache/sqoop/manager/SqlManager.java src/docs/user/hbase-args.txt src/java/org/apache/sqoop/mapreduce/HBaseBulkImportMapper.java src/test/com/cloudera/sqoop/TestSqoopOptions.java src/java/org/apache/sqoop/mapreduce/ImportJobBase.java src/java/org/apache/sqoop/SqoopOptions.java src/java/org/apache/sqoop/mapreduce/HBaseBulkImportJob.java src/docs/user/hbase.txt

            People

            • Assignee:
              alexandre.normand Alexandre Normand
              Reporter:
              zhan8610189 Zhancheng Deng
            • Votes:
              2 Vote for this issue
              Watchers:
              10 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development