[CARBONDATA-2848] Task failed during 1st load and restarted on same executor

• Type: Bug
• Status: Open
• Priority: Minor
• Resolution: Unresolved
• Affects Version/s: 1.4.1
• Component/s: data-load
• Environment: Spark 2.1
• Assignee: Unassigned
• Reporter: Pawan Malwal
• Due: 10/Aug/18
• Created: 09/Aug/18 13:19
• Updated: 09/Aug/18 13:30

Description

Steps :

Huge data load performed. (table has 3.5 billion records)

 

Actual Issue :

One of the task failed during first load and restarted again in the same executor and took double the time to load first segment, rest of the 5 load ran properly.

 Due to this failure and task rerun Segment_0 has got double the number of .carbondata files and even the number of records loaded twice.

 

Expected :

Task should not fail in 1st load. Even after failed task restarts carbondata files and records loaded should not double.