[SQOOP-1936] Sqoop2: Sort by comparing IDF data in shuffle phase

• Type: Sub-task
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 1.99.5
• Assignee: Veena Basavaraj
• Reporter: Veena Basavaraj
• Created: 22/Dec/14 16:52
• Updated: 13/Jan/15 05:47
• Resolved: 13/Jan/15 05:18

Description

The compareTo method in SqoopWritable looks like below, this means we are relying on the entire CSVString to order elements in the output.

For instance, if we have 20 records in JDBC data source, the output will not be ordered by its primary key, especially when we need to do delta merges, we have to be cognizant that the ordering of records in the output is controlled by their CSV representation

  String getString() {
    return toIDF.getCSVTextData();
  }



  @Override
  public int compareTo(SqoopWritable o) {
    return getString().compareTo(o.getString());
  }

Issue Links

• links to: RB