[CARBONDATA-1176] Streaming Ingest: Write path streaming segment/file creation

• Type: Sub-task
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 1.2.0
• Fix Version/s: 1.3.0
• Component/s: core, data-load, hadoop-integration
• Assignee: Unassigned
• Reporter: Aniket Adnaik
• Created: 14/Jun/17 19:17
• Updated: 03/Feb/18 09:47
• Resolved: 03/Feb/18 09:47

Description

Streaming Ingest: Write path : segment /streaming file

• Streaming segment creation and streaming file creation
• resolve conflict with spark structured streaming file names. Spark structured streaming names streaming files with unique batch id to avoid overwriting
• maintain spark structured streaming recover-ability as streaming file names generated by spark structured streaming are unique and recorded in spark structured streaming metadata.