[SQOOP-3178] SQOOP PARQUET INCREMENTAL MERGE

• Type: Improvement
• Status: Resolved
• Priority: Blocker
• Resolution: Fixed
• Fix Version/s: 1.5.0
• Component/s: build, codegen, connectors
• Labels: features newbie sqoop
• Assignee: Sandish Kumar HN
• Reporter: Sandish Kumar HN
• Created: 03/May/17 16:37
• Updated: 22/Feb/18 15:01
• Resolved: 22/Jul/17 05:26

Description

Currently, sqoop-1 only supports merging of two Parquet format data sets but it doesn't support to do incremental merge, so I have written a Sqoop Incremental Merge MR for Parquet File Format and I have tested with million records of data with N number of iterations.

blocked by issue https://issues.apache.org/jira/browse/SQOOP-3192

Issue Links

• Blocked: SQOOP-3182 Sqoop1 (import + --incremental + --merge-key + --as-parquetfile) fails with (Can't parse input data: 'PAR1')
• is blocked by: SQOOP-3192 upgrade parquet
• is related to: SQOOP-3181 Sqoop1 (import + --incremental + --merge-key + --as-parquetfile) fails with (Could not find class <CLASS>.)
• is related to: SQOOP-3182 Sqoop1 (import + --incremental + --merge-key + --as-parquetfile) fails with (Can't parse input data: 'PAR1')
• relates to: PARQUET-1010 SKIP FILES STARTS WITH "." in parquet.hadoop.ParquetFileReader.readAllFootersInParallel
• links to: Review Board