Uploaded image for project: 'Sqoop'
  1. Sqoop
  2. SQOOP-1032

Add the --bulk-load-dir option to support the HBase doBulkLoad function

        Details

        • Type: Bug
        • Status: Resolved
        • Priority: Major
        • Resolution: Fixed
        • Affects Version/s: 1.4.3
        • Fix Version/s: 1.4.5
        • Component/s: hbase-integration
        • Labels:
        • Environment:

          Linux

          Description

          HBase supply the LoadIncrementalHFiles.LoadIncrementalHFiles method for bulk load, so this feature added with --bulk-load-dir option to support the HBase doBulkLoad function

          1. File
            SQOOP-1032.patch.2
            17 kB
            Zhancheng Deng
          2. File
            SQOOP-1032.path
            18 kB
            Zhancheng Deng
          3. Text File
            SQOOP-1032-20131010.patch
            20 kB
            Alexandre Normand

            Issue Links

            relates to

            Bug - A problem which impairs or prevents the functions of the product. SQOOP-1215 Add HBase bulk load tests

            • Major - Major loss of function.
            • Open

            Improvement - An improvement or enhancement to an existing feature or task. HBASE-9744 Set io.serializations correctly in HFileOutputFormat.configureIncrementalLoad

            • Minor - Minor loss of function, or other problem where easy workaround is present.
            • Closed
            links to

            Web Link Review board

            Web Link Review board (newer patch)

              Activity

              Ascending order - Click to sort in descending order
              Hide
              Permalink
              zhan8610189 Zhancheng Deng added a comment -

              the patch attached

              Show
              zhan8610189 Zhancheng Deng added a comment - the patch attached
              Hide
              Permalink
              jarcec Jarek Jarcec Cecho added a comment -

              Assigning to Zhancheng Deng.

              Show
              jarcec Jarek Jarcec Cecho added a comment - Assigning to Zhancheng Deng .
              Hide
              Permalink
              jarcec Jarek Jarcec Cecho added a comment -

              Zhancheng Deng, thank you for working on this JIRA, would you mind uploading your patch to the review board?

              Show
              jarcec Jarek Jarcec Cecho added a comment - Zhancheng Deng , thank you for working on this JIRA, would you mind uploading your patch to the review board ?
              Hide
              Permalink
              vasanthkumar Vasanth kumar RJ added a comment -

              Please upload patch to review board and get approvals/comments from reviewers. Once committed, I can include this in version 1.4.4.

              Show
              vasanthkumar Vasanth kumar RJ added a comment - Please upload patch to review board and get approvals/comments from reviewers. Once committed, I can include this in version 1.4.4.
              Hide
              Permalink
              zhan8610189 Zhancheng Deng added a comment - - edited

              hi, Vasanth and Jarek:

              sorry, i am a novice to the jira and the review board.

              OK, i upload the patch to the review board, but error happened:

              The file 'src/java/com/cloudera/sqoop/hbase/HBasePutProcessor.java' (rf1d4e99) could not be found in the repository

              i think i got the wrong version of sqoop on the github before long.
              you can tell me the right repository, and i checkout and repatch it.

              thanks very much

              Zhancheng

              Show
              zhan8610189 Zhancheng Deng added a comment - - edited hi, Vasanth and Jarek: sorry, i am a novice to the jira and the review board. OK, i upload the patch to the review board, but error happened: The file 'src/java/com/cloudera/sqoop/hbase/HBasePutProcessor.java' (rf1d4e99) could not be found in the repository i think i got the wrong version of sqoop on the github before long. you can tell me the right repository, and i checkout and repatch it. thanks very much Zhancheng
              Hide
              Permalink
              jarcec Jarek Jarcec Cecho added a comment -

              Hi Zhancheng Deng,
              you can check out the git repository directly from Apache, there is no need to take the mirror from Github:

              git clone https://git-wip-us.apache.org/repos/asf/sqoop.git
              Show
              jarcec Jarek Jarcec Cecho added a comment - Hi Zhancheng Deng , you can check out the git repository directly from Apache, there is no need to take the mirror from Github: git clone https: //git-wip-us.apache.org/repos/asf/sqoop.git
              Hide
              Permalink
              zhan8610189 Zhancheng Deng added a comment -

              modify the patch of the correct repository

              Show
              zhan8610189 Zhancheng Deng added a comment - modify the patch of the correct repository
              Hide
              Permalink
              zhan8610189 Zhancheng Deng added a comment -

              review board is committed

              Show
              zhan8610189 Zhancheng Deng added a comment - review board is committed
              Hide
              Permalink
              alexandre.normand Alexandre Normand added a comment -

              I'd really like to use bulk loading with sqoop. Is there a chance this could get reviewed/committed soon? I was going to handle it by sqooping to a sequence file than having a second job convert to hfiles for bulk loading but this looks like a better way.

              Show
              alexandre.normand Alexandre Normand added a comment - I'd really like to use bulk loading with sqoop. Is there a chance this could get reviewed/committed soon? I was going to handle it by sqooping to a sequence file than having a second job convert to hfiles for bulk loading but this looks like a better way.
              Hide
              Permalink
              alexandre.normand Alexandre Normand added a comment -

              I tried to address some of the comments on the original review request but since I can't add a diff to that one, I created a new request on reviewboard: https://reviews.apache.org/r/14240/

              Show
              alexandre.normand Alexandre Normand added a comment - I tried to address some of the comments on the original review request but since I can't add a diff to that one, I created a new request on reviewboard: https://reviews.apache.org/r/14240/
              Hide
              Permalink
              alexandre.normand Alexandre Normand added a comment -

              Here's the final patch as reviewed in https://reviews.apache.org/r/14240/.

              Thanks!

              Show
              alexandre.normand Alexandre Normand added a comment - Here's the final patch as reviewed in https://reviews.apache.org/r/14240/ . Thanks!
              Hide
              Permalink
              jira-bot ASF subversion and git services added a comment -

              Commit ddb81e185be72c7530498c379e6ad45e6d54a2d6 in branch refs/heads/trunk from Jarek Jarcec Cecho
              [ https://git-wip-us.apache.org/repos/asf?p=sqoop.git;h=ddb81e1 ]

              SQOOP-1032: Add the --bulk-load-dir option to support the HBase doBulkLoad function

              (Alexandre Normand via Jarek Jarcec Cecho)

              Show
              jira-bot ASF subversion and git services added a comment - Commit ddb81e185be72c7530498c379e6ad45e6d54a2d6 in branch refs/heads/trunk from Jarek Jarcec Cecho [ https://git-wip-us.apache.org/repos/asf?p=sqoop.git;h=ddb81e1 ] SQOOP-1032 : Add the --bulk-load-dir option to support the HBase doBulkLoad function (Alexandre Normand via Jarek Jarcec Cecho)
              Hide
              Permalink
              jarcec Jarek Jarcec Cecho added a comment -

              The patch is in, thank you Alexandre Normand for your time and effort!

              Show
              jarcec Jarek Jarcec Cecho added a comment - The patch is in, thank you Alexandre Normand for your time and effort!
              Hide
              Permalink
              hudson Hudson added a comment -

              SUCCESS: Integrated in Sqoop-ant-jdk-1.6-hadoop200 #862 (See https://builds.apache.org/job/Sqoop-ant-jdk-1.6-hadoop200/862/)
              SQOOP-1032: Add the --bulk-load-dir option to support the HBase doBulkLoad function (jarcec: https://git-wip-us.apache.org/repos/asf?p=sqoop.git&a=commit&h=ddb81e185be72c7530498c379e6ad45e6d54a2d6)

              • src/java/org/apache/sqoop/manager/SqlManager.java
              • src/java/org/apache/sqoop/SqoopOptions.java
              • src/java/org/apache/sqoop/tool/BaseSqoopTool.java
              • src/java/org/apache/sqoop/mapreduce/ImportJobBase.java
              • src/java/org/apache/sqoop/mapreduce/HBaseBulkImportMapper.java
              • src/test/com/cloudera/sqoop/TestSqoopOptions.java
              • src/java/org/apache/sqoop/hbase/ToStringPutTransformer.java
              • src/java/org/apache/sqoop/hbase/HBasePutProcessor.java
              • src/java/org/apache/sqoop/mapreduce/HBaseBulkImportJob.java
              • src/docs/user/hbase-args.txt
              • src/docs/user/hbase.txt
              Show
              hudson Hudson added a comment - SUCCESS: Integrated in Sqoop-ant-jdk-1.6-hadoop200 #862 (See https://builds.apache.org/job/Sqoop-ant-jdk-1.6-hadoop200/862/ ) SQOOP-1032 : Add the --bulk-load-dir option to support the HBase doBulkLoad function (jarcec: https://git-wip-us.apache.org/repos/asf?p=sqoop.git&a=commit&h=ddb81e185be72c7530498c379e6ad45e6d54a2d6 ) src/java/org/apache/sqoop/manager/SqlManager.java src/java/org/apache/sqoop/SqoopOptions.java src/java/org/apache/sqoop/tool/BaseSqoopTool.java src/java/org/apache/sqoop/mapreduce/ImportJobBase.java src/java/org/apache/sqoop/mapreduce/HBaseBulkImportMapper.java src/test/com/cloudera/sqoop/TestSqoopOptions.java src/java/org/apache/sqoop/hbase/ToStringPutTransformer.java src/java/org/apache/sqoop/hbase/HBasePutProcessor.java src/java/org/apache/sqoop/mapreduce/HBaseBulkImportJob.java src/docs/user/hbase-args.txt src/docs/user/hbase.txt
              Hide
              Permalink
              hudson Hudson added a comment -

              SUCCESS: Integrated in Sqoop-ant-jdk-1.6-hadoop20 #857 (See https://builds.apache.org/job/Sqoop-ant-jdk-1.6-hadoop20/857/)
              SQOOP-1032: Add the --bulk-load-dir option to support the HBase doBulkLoad function (jarcec: https://git-wip-us.apache.org/repos/asf?p=sqoop.git&a=commit&h=ddb81e185be72c7530498c379e6ad45e6d54a2d6)

              • src/docs/user/hbase-args.txt
              • src/java/org/apache/sqoop/manager/SqlManager.java
              • src/docs/user/hbase.txt
              • src/test/com/cloudera/sqoop/TestSqoopOptions.java
              • src/java/org/apache/sqoop/mapreduce/ImportJobBase.java
              • src/java/org/apache/sqoop/mapreduce/HBaseBulkImportJob.java
              • src/java/org/apache/sqoop/hbase/ToStringPutTransformer.java
              • src/java/org/apache/sqoop/hbase/HBasePutProcessor.java
              • src/java/org/apache/sqoop/SqoopOptions.java
              • src/java/org/apache/sqoop/tool/BaseSqoopTool.java
              • src/java/org/apache/sqoop/mapreduce/HBaseBulkImportMapper.java
              Show
              hudson Hudson added a comment - SUCCESS: Integrated in Sqoop-ant-jdk-1.6-hadoop20 #857 (See https://builds.apache.org/job/Sqoop-ant-jdk-1.6-hadoop20/857/ ) SQOOP-1032 : Add the --bulk-load-dir option to support the HBase doBulkLoad function (jarcec: https://git-wip-us.apache.org/repos/asf?p=sqoop.git&a=commit&h=ddb81e185be72c7530498c379e6ad45e6d54a2d6 ) src/docs/user/hbase-args.txt src/java/org/apache/sqoop/manager/SqlManager.java src/docs/user/hbase.txt src/test/com/cloudera/sqoop/TestSqoopOptions.java src/java/org/apache/sqoop/mapreduce/ImportJobBase.java src/java/org/apache/sqoop/mapreduce/HBaseBulkImportJob.java src/java/org/apache/sqoop/hbase/ToStringPutTransformer.java src/java/org/apache/sqoop/hbase/HBasePutProcessor.java src/java/org/apache/sqoop/SqoopOptions.java src/java/org/apache/sqoop/tool/BaseSqoopTool.java src/java/org/apache/sqoop/mapreduce/HBaseBulkImportMapper.java
              Hide
              Permalink
              hudson Hudson added a comment -

              SUCCESS: Integrated in Sqoop-ant-jdk-1.6-hadoop23 #1059 (See https://builds.apache.org/job/Sqoop-ant-jdk-1.6-hadoop23/1059/)
              SQOOP-1032: Add the --bulk-load-dir option to support the HBase doBulkLoad function (jarcec: https://git-wip-us.apache.org/repos/asf?p=sqoop.git&a=commit&h=ddb81e185be72c7530498c379e6ad45e6d54a2d6)

              • src/docs/user/hbase.txt
              • src/test/com/cloudera/sqoop/TestSqoopOptions.java
              • src/java/org/apache/sqoop/mapreduce/HBaseBulkImportMapper.java
              • src/docs/user/hbase-args.txt
              • src/java/org/apache/sqoop/hbase/HBasePutProcessor.java
              • src/java/org/apache/sqoop/tool/BaseSqoopTool.java
              • src/java/org/apache/sqoop/mapreduce/ImportJobBase.java
              • src/java/org/apache/sqoop/mapreduce/HBaseBulkImportJob.java
              • src/java/org/apache/sqoop/manager/SqlManager.java
              • src/java/org/apache/sqoop/SqoopOptions.java
              • src/java/org/apache/sqoop/hbase/ToStringPutTransformer.java
              Show
              hudson Hudson added a comment - SUCCESS: Integrated in Sqoop-ant-jdk-1.6-hadoop23 #1059 (See https://builds.apache.org/job/Sqoop-ant-jdk-1.6-hadoop23/1059/ ) SQOOP-1032 : Add the --bulk-load-dir option to support the HBase doBulkLoad function (jarcec: https://git-wip-us.apache.org/repos/asf?p=sqoop.git&a=commit&h=ddb81e185be72c7530498c379e6ad45e6d54a2d6 ) src/docs/user/hbase.txt src/test/com/cloudera/sqoop/TestSqoopOptions.java src/java/org/apache/sqoop/mapreduce/HBaseBulkImportMapper.java src/docs/user/hbase-args.txt src/java/org/apache/sqoop/hbase/HBasePutProcessor.java src/java/org/apache/sqoop/tool/BaseSqoopTool.java src/java/org/apache/sqoop/mapreduce/ImportJobBase.java src/java/org/apache/sqoop/mapreduce/HBaseBulkImportJob.java src/java/org/apache/sqoop/manager/SqlManager.java src/java/org/apache/sqoop/SqoopOptions.java src/java/org/apache/sqoop/hbase/ToStringPutTransformer.java
              Hide
              Permalink
              hudson Hudson added a comment -

              SUCCESS: Integrated in Sqoop-ant-jdk-1.6-hadoop100 #820 (See https://builds.apache.org/job/Sqoop-ant-jdk-1.6-hadoop100/820/)
              SQOOP-1032: Add the --bulk-load-dir option to support the HBase doBulkLoad function (jarcec: https://git-wip-us.apache.org/repos/asf?p=sqoop.git&a=commit&h=ddb81e185be72c7530498c379e6ad45e6d54a2d6)

              • src/java/org/apache/sqoop/tool/BaseSqoopTool.java
              • src/java/org/apache/sqoop/hbase/ToStringPutTransformer.java
              • src/java/org/apache/sqoop/hbase/HBasePutProcessor.java
              • src/java/org/apache/sqoop/manager/SqlManager.java
              • src/docs/user/hbase-args.txt
              • src/java/org/apache/sqoop/mapreduce/HBaseBulkImportMapper.java
              • src/test/com/cloudera/sqoop/TestSqoopOptions.java
              • src/java/org/apache/sqoop/mapreduce/ImportJobBase.java
              • src/java/org/apache/sqoop/SqoopOptions.java
              • src/java/org/apache/sqoop/mapreduce/HBaseBulkImportJob.java
              • src/docs/user/hbase.txt
              Show
              hudson Hudson added a comment - SUCCESS: Integrated in Sqoop-ant-jdk-1.6-hadoop100 #820 (See https://builds.apache.org/job/Sqoop-ant-jdk-1.6-hadoop100/820/ ) SQOOP-1032 : Add the --bulk-load-dir option to support the HBase doBulkLoad function (jarcec: https://git-wip-us.apache.org/repos/asf?p=sqoop.git&a=commit&h=ddb81e185be72c7530498c379e6ad45e6d54a2d6 ) src/java/org/apache/sqoop/tool/BaseSqoopTool.java src/java/org/apache/sqoop/hbase/ToStringPutTransformer.java src/java/org/apache/sqoop/hbase/HBasePutProcessor.java src/java/org/apache/sqoop/manager/SqlManager.java src/docs/user/hbase-args.txt src/java/org/apache/sqoop/mapreduce/HBaseBulkImportMapper.java src/test/com/cloudera/sqoop/TestSqoopOptions.java src/java/org/apache/sqoop/mapreduce/ImportJobBase.java src/java/org/apache/sqoop/SqoopOptions.java src/java/org/apache/sqoop/mapreduce/HBaseBulkImportJob.java src/docs/user/hbase.txt

                People

                • Assignee:
                  alexandre.normand Alexandre Normand
                  Reporter:
                  zhan8610189 Zhancheng Deng
                • Votes:
                  2 Vote for this issue
                  Watchers:
                  10 Start watching this issue

                  Dates

                  • Created:
                    Updated:
                    Resolved:

                    Development