Sqoop
  1. Sqoop
  2. SQOOP-830

HBase import formatting BigDecimal inconsistently

    Details

    • Type: Bug Bug
    • Status: Resolved
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 1.4.3
    • Component/s: None
    • Labels:
      None

      Description

      When importing into HBase the toString() method is called on every field via the ToStringPutTransformer class.
      When the field is mapped as a BigDecimal - as it is with number fields in Oracle - this results in inconsistent formats in HBase.
      For example - create the following in Oracle:

      CREATE TABLE employee(id number primary key, test_number number);
      INSERT INTO employee values(1, 0.000001);
      INSERT INTO employee values(2, 0.0000001);
      COMMIT;

      Then run an import:

      sqoop import --connect jdbc:oracle:thin:@//HOSTNAME/SERVICE --username USERNAME --table EMPLOYEE --password PASSWORD --hbase-table EMPLOYEE --column-family tst --hbase-create-table

      The value for row 1 is "0.000001" while row 2 is "1E-7".

      1. SQOOP-830.patch
        7 kB
        David Robson
      2. SQOOP-830.patch
        19 kB
        David Robson

        Issue Links

          Activity

          Hide
          Hudson added a comment -

          Integrated in Sqoop-ant-jdk-1.6-hadoop100 #538 (See https://builds.apache.org/job/Sqoop-ant-jdk-1.6-hadoop100/538/)
          SQOOP-830: HBase import formatting BigDecimal inconsistently (Revision 7c5b46fb2860e7401e84542a01a61ef415cbe519)

          Result = SUCCESS
          jarcec : https://git-wip-us.apache.org/repos/asf?p=sqoop.git&a=commit&h=7c5b46fb2860e7401e84542a01a61ef415cbe519
          Files :

          • src/test/org/apache/sqoop/TestBigDecimalImport.java
          • src/test/com/cloudera/sqoop/testutil/BaseSqoopTestCase.java
          • src/test/org/apache/sqoop/TestBigDecimalExport.java
          • src/docs/user/import.txt
          • src/java/org/apache/sqoop/orm/ClassWriter.java
          • src/java/org/apache/sqoop/hbase/HBasePutProcessor.java
          • src/java/org/apache/sqoop/mapreduce/ImportJobBase.java
          • src/java/org/apache/sqoop/hbase/ToStringPutTransformer.java
          • src/java/org/apache/sqoop/mapreduce/AvroImportMapper.java
          Show
          Hudson added a comment - Integrated in Sqoop-ant-jdk-1.6-hadoop100 #538 (See https://builds.apache.org/job/Sqoop-ant-jdk-1.6-hadoop100/538/ ) SQOOP-830 : HBase import formatting BigDecimal inconsistently (Revision 7c5b46fb2860e7401e84542a01a61ef415cbe519) Result = SUCCESS jarcec : https://git-wip-us.apache.org/repos/asf?p=sqoop.git&a=commit&h=7c5b46fb2860e7401e84542a01a61ef415cbe519 Files : src/test/org/apache/sqoop/TestBigDecimalImport.java src/test/com/cloudera/sqoop/testutil/BaseSqoopTestCase.java src/test/org/apache/sqoop/TestBigDecimalExport.java src/docs/user/import.txt src/java/org/apache/sqoop/orm/ClassWriter.java src/java/org/apache/sqoop/hbase/HBasePutProcessor.java src/java/org/apache/sqoop/mapreduce/ImportJobBase.java src/java/org/apache/sqoop/hbase/ToStringPutTransformer.java src/java/org/apache/sqoop/mapreduce/AvroImportMapper.java
          Hide
          Hudson added a comment -

          Integrated in Sqoop-ant-jdk-1.6-hadoop200 #537 (See https://builds.apache.org/job/Sqoop-ant-jdk-1.6-hadoop200/537/)
          SQOOP-830: HBase import formatting BigDecimal inconsistently (Revision 7c5b46fb2860e7401e84542a01a61ef415cbe519)

          Result = SUCCESS
          jarcec : https://git-wip-us.apache.org/repos/asf?p=sqoop.git&a=commit&h=7c5b46fb2860e7401e84542a01a61ef415cbe519
          Files :

          • src/java/org/apache/sqoop/hbase/HBasePutProcessor.java
          • src/java/org/apache/sqoop/mapreduce/ImportJobBase.java
          • src/test/com/cloudera/sqoop/testutil/BaseSqoopTestCase.java
          • src/docs/user/import.txt
          • src/test/org/apache/sqoop/TestBigDecimalImport.java
          • src/java/org/apache/sqoop/orm/ClassWriter.java
          • src/java/org/apache/sqoop/hbase/ToStringPutTransformer.java
          • src/test/org/apache/sqoop/TestBigDecimalExport.java
          • src/java/org/apache/sqoop/mapreduce/AvroImportMapper.java
          Show
          Hudson added a comment - Integrated in Sqoop-ant-jdk-1.6-hadoop200 #537 (See https://builds.apache.org/job/Sqoop-ant-jdk-1.6-hadoop200/537/ ) SQOOP-830 : HBase import formatting BigDecimal inconsistently (Revision 7c5b46fb2860e7401e84542a01a61ef415cbe519) Result = SUCCESS jarcec : https://git-wip-us.apache.org/repos/asf?p=sqoop.git&a=commit&h=7c5b46fb2860e7401e84542a01a61ef415cbe519 Files : src/java/org/apache/sqoop/hbase/HBasePutProcessor.java src/java/org/apache/sqoop/mapreduce/ImportJobBase.java src/test/com/cloudera/sqoop/testutil/BaseSqoopTestCase.java src/docs/user/import.txt src/test/org/apache/sqoop/TestBigDecimalImport.java src/java/org/apache/sqoop/orm/ClassWriter.java src/java/org/apache/sqoop/hbase/ToStringPutTransformer.java src/test/org/apache/sqoop/TestBigDecimalExport.java src/java/org/apache/sqoop/mapreduce/AvroImportMapper.java
          Hide
          Hudson added a comment -

          Integrated in Sqoop-ant-jdk-1.6-hadoop20 #537 (See https://builds.apache.org/job/Sqoop-ant-jdk-1.6-hadoop20/537/)
          SQOOP-830: HBase import formatting BigDecimal inconsistently (Revision 7c5b46fb2860e7401e84542a01a61ef415cbe519)

          Result = SUCCESS
          jarcec : https://git-wip-us.apache.org/repos/asf?p=sqoop.git&a=commit&h=7c5b46fb2860e7401e84542a01a61ef415cbe519
          Files :

          • src/java/org/apache/sqoop/hbase/HBasePutProcessor.java
          • src/java/org/apache/sqoop/orm/ClassWriter.java
          • src/test/com/cloudera/sqoop/testutil/BaseSqoopTestCase.java
          • src/java/org/apache/sqoop/mapreduce/ImportJobBase.java
          • src/test/org/apache/sqoop/TestBigDecimalExport.java
          • src/java/org/apache/sqoop/mapreduce/AvroImportMapper.java
          • src/java/org/apache/sqoop/hbase/ToStringPutTransformer.java
          • src/docs/user/import.txt
          • src/test/org/apache/sqoop/TestBigDecimalImport.java
          Show
          Hudson added a comment - Integrated in Sqoop-ant-jdk-1.6-hadoop20 #537 (See https://builds.apache.org/job/Sqoop-ant-jdk-1.6-hadoop20/537/ ) SQOOP-830 : HBase import formatting BigDecimal inconsistently (Revision 7c5b46fb2860e7401e84542a01a61ef415cbe519) Result = SUCCESS jarcec : https://git-wip-us.apache.org/repos/asf?p=sqoop.git&a=commit&h=7c5b46fb2860e7401e84542a01a61ef415cbe519 Files : src/java/org/apache/sqoop/hbase/HBasePutProcessor.java src/java/org/apache/sqoop/orm/ClassWriter.java src/test/com/cloudera/sqoop/testutil/BaseSqoopTestCase.java src/java/org/apache/sqoop/mapreduce/ImportJobBase.java src/test/org/apache/sqoop/TestBigDecimalExport.java src/java/org/apache/sqoop/mapreduce/AvroImportMapper.java src/java/org/apache/sqoop/hbase/ToStringPutTransformer.java src/docs/user/import.txt src/test/org/apache/sqoop/TestBigDecimalImport.java
          Hide
          Hudson added a comment -

          Integrated in Sqoop-ant-jdk-1.6-hadoop23 #732 (See https://builds.apache.org/job/Sqoop-ant-jdk-1.6-hadoop23/732/)
          SQOOP-830: HBase import formatting BigDecimal inconsistently (Revision 7c5b46fb2860e7401e84542a01a61ef415cbe519)

          Result = SUCCESS
          jarcec : https://git-wip-us.apache.org/repos/asf?p=sqoop.git&a=commit&h=7c5b46fb2860e7401e84542a01a61ef415cbe519
          Files :

          • src/java/org/apache/sqoop/hbase/ToStringPutTransformer.java
          • src/test/org/apache/sqoop/TestBigDecimalImport.java
          • src/test/org/apache/sqoop/TestBigDecimalExport.java
          • src/test/com/cloudera/sqoop/testutil/BaseSqoopTestCase.java
          • src/java/org/apache/sqoop/orm/ClassWriter.java
          • src/java/org/apache/sqoop/mapreduce/ImportJobBase.java
          • src/java/org/apache/sqoop/mapreduce/AvroImportMapper.java
          • src/java/org/apache/sqoop/hbase/HBasePutProcessor.java
          • src/docs/user/import.txt
          Show
          Hudson added a comment - Integrated in Sqoop-ant-jdk-1.6-hadoop23 #732 (See https://builds.apache.org/job/Sqoop-ant-jdk-1.6-hadoop23/732/ ) SQOOP-830 : HBase import formatting BigDecimal inconsistently (Revision 7c5b46fb2860e7401e84542a01a61ef415cbe519) Result = SUCCESS jarcec : https://git-wip-us.apache.org/repos/asf?p=sqoop.git&a=commit&h=7c5b46fb2860e7401e84542a01a61ef415cbe519 Files : src/java/org/apache/sqoop/hbase/ToStringPutTransformer.java src/test/org/apache/sqoop/TestBigDecimalImport.java src/test/org/apache/sqoop/TestBigDecimalExport.java src/test/com/cloudera/sqoop/testutil/BaseSqoopTestCase.java src/java/org/apache/sqoop/orm/ClassWriter.java src/java/org/apache/sqoop/mapreduce/ImportJobBase.java src/java/org/apache/sqoop/mapreduce/AvroImportMapper.java src/java/org/apache/sqoop/hbase/HBasePutProcessor.java src/docs/user/import.txt
          Jarek Jarcec Cecho made changes -
          Status Patch Available [ 10002 ] Resolved [ 5 ]
          Fix Version/s 1.4.3 [ 12322644 ]
          Resolution Fixed [ 1 ]
          Hide
          Jarek Jarcec Cecho added a comment -

          The patch is in: https://git-wip-us.apache.org/repos/asf?p=sqoop.git;a=commit;h=7c5b46fb2860e7401e84542a01a61ef415cbe519

          Thank you David very much for your time and effort.

          Jarcec

          Show
          Jarek Jarcec Cecho added a comment - The patch is in: https://git-wip-us.apache.org/repos/asf?p=sqoop.git;a=commit;h=7c5b46fb2860e7401e84542a01a61ef415cbe519 Thank you David very much for your time and effort. Jarcec
          Hide
          Jarek Jarcec Cecho added a comment -

          Hi David,
          thank you very much for doing all the changes, I've just +1 the patch on review board and I'll commit it soon.

          Jarcec

          Show
          Jarek Jarcec Cecho added a comment - Hi David, thank you very much for doing all the changes, I've just +1 the patch on review board and I'll commit it soon. Jarcec
          Hide
          David Robson added a comment -

          Hi Jarcec,

          I have updated the patch as per your comments.

          I added some unit tests - while doing so I changed BaseSqoopTestCase to accept null for values as I did not want to create any rows in the table.

          I added a new section to the documentation "Additional Import Configuration Properties" - this doesn't seem quite ideal either - but I'm guessing the documentation will have to have a lot of work done for Sqoop2 so I thought that will probably do for now. Obviously there is a lot of undocumented parameters like you mentioned - we could add these into the same section and create a similar section for export parameters and common ones. Or perhaps join them together?

          Anyway I thought doing too much work on the structure of the documentation was beyond this change so that's what I came up with.

          Let me know what you think.

          Thanks,

          David

          Show
          David Robson added a comment - Hi Jarcec, I have updated the patch as per your comments. I added some unit tests - while doing so I changed BaseSqoopTestCase to accept null for values as I did not want to create any rows in the table. I added a new section to the documentation "Additional Import Configuration Properties" - this doesn't seem quite ideal either - but I'm guessing the documentation will have to have a lot of work done for Sqoop2 so I thought that will probably do for now. Obviously there is a lot of undocumented parameters like you mentioned - we could add these into the same section and create a similar section for export parameters and common ones. Or perhaps join them together? Anyway I thought doing too much work on the structure of the documentation was beyond this change so that's what I came up with. Let me know what you think. Thanks, David
          David Robson made changes -
          Attachment SQOOP-830.patch [ 12567304 ]
          Hide
          Jarek Jarcec Cecho added a comment -

          Hi David,
          no worries. This definitely works! I've shared few comments on the review board if you don't mind taking a look.

          Jarcec

          Show
          Jarek Jarcec Cecho added a comment - Hi David, no worries. This definitely works! I've shared few comments on the review board if you don't mind taking a look. Jarcec
          David Robson made changes -
          Status In Progress [ 3 ] Patch Available [ 10002 ]
          Hide
          David Robson added a comment -

          Hi Jarcec,

          I was going to submit the patch after the review - sorry wasn't really 100% sure of the process - hopefully this should work now.

          David

          Show
          David Robson added a comment - Hi Jarcec, I was going to submit the patch after the review - sorry wasn't really 100% sure of the process - hopefully this should work now. David
          David Robson made changes -
          Attachment SQOOP-830.patch [ 12566663 ]
          Hide
          Jarek Jarcec Cecho added a comment -

          Hi David,
          please accept my apologies that I did not reviewed your patch sooner. I've missed the review board email and this jira is "only" in "In progress" state. Would you mind changing it to "Patch available", so that it will show up in our review waiting queue [1]?

          Jarcec

          Links:
          1: http://s.apache.org/h7

          Show
          Jarek Jarcec Cecho added a comment - Hi David, please accept my apologies that I did not reviewed your patch sooner. I've missed the review board email and this jira is "only" in "In progress" state. Would you mind changing it to "Patch available", so that it will show up in our review waiting queue [1] ? Jarcec Links: 1: http://s.apache.org/h7
          David Robson made changes -
          Status Open [ 1 ] In Progress [ 3 ]
          David Robson made changes -
          Assignee David Robson [ david.robson ]
          David Robson made changes -
          Field Original Value New Value
          Remote Link This issue links to "Review board (Web Link)" [ 11989 ]
          David Robson created issue -

            People

            • Assignee:
              David Robson
              Reporter:
              David Robson
            • Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development