[SQOOP-2001] Sqoop2 Kite connector might produce duplicate values when retrying failed tasks.

• Type: Bug
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Fix Version/s: 2.0.0
• Assignee: Unassigned
• Reporter: Ryan Blue
• Created: 10/Jan/15 03:46
• Updated: 10/Jan/15 17:15

Description

This happens (as I understand things) because Kite may make files visible before a task is completed or committed in the combined temporary dataset directory. We should be able to avoid this by setting the temporary dataset's writer cache limit to something huge - so files are not closed until the overall writer is closed, no matter how many files are open.