Sqoop
  1. Sqoop
  2. SQOOP-1154

Sqoop2: Text partitioner might miss or include edge values

    Details

    • Type: Bug Bug
    • Status: Resolved
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 1.99.2
    • Fix Version/s: 1.99.3
    • Component/s: None
    • Labels:
      None

      Description

      Attached is test case that will use Varchar partitioner to generate 3, 5, 10 and 13 partitions for interval "Breezy Badger" to "Warty Warthog". For all cases generated partitions looks like the following:

      'Bree' <= VCCOL AND VCCOL < SOME_VALUE
      SOME_VALUE <= VCCOL AND VCCOL <= 'Wart'
      

      As 'Warty Warthog' > 'Wart', the last value will be never imported. Also similarly as 'Bree' < 'Breezy Badger', additional values might be imported as well (for example 'Breedy Budget'). I think that the varchar partitioner must have the interval boundaries there without any truncation, for example:

      'Breezy Badget' <= VCCOL AND VCCOL < SOME_VALUE
      SOME_VALUE <= VCCOL AND VCCOL <= 'Warty Warthog'
      
      1. import_test_case.patch
        2 kB
        Jarek Jarcec Cecho
      2. bugSQOOP-1154.patch
        5 kB
        Jarek Jarcec Cecho

        Issue Links

          Activity

          Hide
          Hudson added a comment -

          SUCCESS: Integrated in Sqoop2-hadoop200 #480 (See https://builds.apache.org/job/Sqoop2-hadoop200/480/)
          SQOOP-1154: Sqoop2: Text partitioner might miss or include edge values (n.r.v: https://git-wip-us.apache.org/repos/asf?p=sqoop.git&a=commit&h=9c7adb4e112e57e5d2c4720be71b8a5278ef7638)

          • connector/connector-generic-jdbc/src/test/java/org/apache/sqoop/connector/jdbc/TestImportPartitioner.java
          • connector/connector-generic-jdbc/src/main/java/org/apache/sqoop/connector/jdbc/GenericJdbcImportPartitioner.java
          Show
          Hudson added a comment - SUCCESS: Integrated in Sqoop2-hadoop200 #480 (See https://builds.apache.org/job/Sqoop2-hadoop200/480/ ) SQOOP-1154 : Sqoop2: Text partitioner might miss or include edge values (n.r.v: https://git-wip-us.apache.org/repos/asf?p=sqoop.git&a=commit&h=9c7adb4e112e57e5d2c4720be71b8a5278ef7638 ) connector/connector-generic-jdbc/src/test/java/org/apache/sqoop/connector/jdbc/TestImportPartitioner.java connector/connector-generic-jdbc/src/main/java/org/apache/sqoop/connector/jdbc/GenericJdbcImportPartitioner.java
          Hide
          Hudson added a comment -

          SUCCESS: Integrated in Sqoop2-hadoop100 #524 (See https://builds.apache.org/job/Sqoop2-hadoop100/524/)
          SQOOP-1154: Sqoop2: Text partitioner might miss or include edge values (n.r.v: https://git-wip-us.apache.org/repos/asf?p=sqoop.git&a=commit&h=9c7adb4e112e57e5d2c4720be71b8a5278ef7638)

          • connector/connector-generic-jdbc/src/test/java/org/apache/sqoop/connector/jdbc/TestImportPartitioner.java
          • connector/connector-generic-jdbc/src/main/java/org/apache/sqoop/connector/jdbc/GenericJdbcImportPartitioner.java
          Show
          Hudson added a comment - SUCCESS: Integrated in Sqoop2-hadoop100 #524 (See https://builds.apache.org/job/Sqoop2-hadoop100/524/ ) SQOOP-1154 : Sqoop2: Text partitioner might miss or include edge values (n.r.v: https://git-wip-us.apache.org/repos/asf?p=sqoop.git&a=commit&h=9c7adb4e112e57e5d2c4720be71b8a5278ef7638 ) connector/connector-generic-jdbc/src/test/java/org/apache/sqoop/connector/jdbc/TestImportPartitioner.java connector/connector-generic-jdbc/src/main/java/org/apache/sqoop/connector/jdbc/GenericJdbcImportPartitioner.java
          Hide
          Venkat Ranganathan added a comment -

          Thanks for fixing this Jarek Jarcec Cecho

          Show
          Venkat Ranganathan added a comment - Thanks for fixing this Jarek Jarcec Cecho
          Hide
          ASF subversion and git services added a comment -

          Commit 9c7adb4e112e57e5d2c4720be71b8a5278ef7638 in branch refs/heads/sqoop2 from Venkat Ranganathan
          [ https://git-wip-us.apache.org/repos/asf?p=sqoop.git;h=9c7adb4 ]

          SQOOP-1154: Sqoop2: Text partitioner might miss or include edge values

          (Jarek Jarcec Cecho via Venkat Ranganathan)

          Show
          ASF subversion and git services added a comment - Commit 9c7adb4e112e57e5d2c4720be71b8a5278ef7638 in branch refs/heads/sqoop2 from Venkat Ranganathan [ https://git-wip-us.apache.org/repos/asf?p=sqoop.git;h=9c7adb4 ] SQOOP-1154 : Sqoop2: Text partitioner might miss or include edge values (Jarek Jarcec Cecho via Venkat Ranganathan)
          Hide
          Jarek Jarcec Cecho added a comment -

          No worries Venkat Ranganathan, take your time!

          Show
          Jarek Jarcec Cecho added a comment - No worries Venkat Ranganathan , take your time!
          Hide
          Venkat Ranganathan added a comment -

          I am seeing some failures on my builds while verifying the patch (looks like some env issues). Will commit once I fix it - Sorry about the delay Jarek Jarcec Cecho. If any one else wants to commit it, please go ahead.

          Show
          Venkat Ranganathan added a comment - I am seeing some failures on my builds while verifying the patch (looks like some env issues). Will commit once I fix it - Sorry about the delay Jarek Jarcec Cecho . If any one else wants to commit it, please go ahead.
          Hide
          Sqoop QA bot added a comment -

          Here are the results of testing the latest attachment
          https://issues.apache.org/jira/secure/attachment/12594867/bugSQOOP-1154.patch against branch sqoop2.

          Overall: +1 all checks pass

          SUCCESS: Clean was successful
          SUCCESS: Patch applied correctly
          SUCCESS: Patch compiled
          SUCCESS: All tests passed

          Console output: https://builds.apache.org/job/PreCommit-SQOOP-Build/105/console

          This message is automatically generated.

          Show
          Sqoop QA bot added a comment - Here are the results of testing the latest attachment https://issues.apache.org/jira/secure/attachment/12594867/bugSQOOP-1154.patch against branch sqoop2. Overall: +1 all checks pass SUCCESS: Clean was successful SUCCESS: Patch applied correctly SUCCESS: Patch compiled SUCCESS: All tests passed Console output: https://builds.apache.org/job/PreCommit-SQOOP-Build/105/console This message is automatically generated.

            People

            • Assignee:
              Jarek Jarcec Cecho
              Reporter:
              Jarek Jarcec Cecho
            • Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development