[SQOOP-2811] Sqoop2: Extracting sequence files may result in duplicates

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 1.99.6
• Fix Version/s: 1.99.7
• Assignee: Abraham Fine
• Reporter: Abraham Fine
• Created: 29/Jan/16 22:15
• Updated: 01/Feb/16 20:19
• Resolved: 01/Feb/16 19:14

Description

In the hdfs extractor we use:

    if (start > filereader.getPosition()) {
      filereader.sync(start); // sync to start
    }

to jump to the correct point in the sequence file that we want to extract.

If the sequence file is small, multiple start points may `sync` to the same point and we could end up extracting the same record multiple times.

Issue Links

• is required by: SQOOP-2808 Sqoop2: Integration tests should test rows with null values
• links to: rb