[SQOOP-1705] SQOOP2: Support AVRO as a an implementation for the IDF interface

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 1.99.3
• Fix Version/s: 1.99.5
• Component/s: sqoop2-framework
• Assignee: Veena Basavaraj
• Reporter: Veena Basavaraj
• Created: 10/Nov/14 20:55
• Updated: 12/Jan/15 20:12
• Resolved: 12/Jan/15 20:12

Description

More details will be added .

CSV is a bare bones data representation format that SQOOP uses to translate the FROM to the TO data source.

AVRO has a well defined schema and hence might be more convenient to represent and map the FROM and TO data types in AVRO.

Issue Links

• depends upon: SQOOP-1882 JobManager currently ignores the TO connector IDF and assumed all IDF use String for the generic T
• is blocked by: SQOOP-1882 JobManager currently ignores the TO connector IDF and assumed all IDF use String for the generic T
• is depended upon by: SQOOP-1616 Sqoop2: Sqoop data type to Avro data type conversion
• relates to: SQOOP-1350 Sqoop2: Support all supported data types in the CSV Intermediate Data Format implementation

Sub-Tasks

• Fix the SqoopWritable read/ write method to delegate the toIDF methods: Resolved
• Sqoop2: Support DRY code in IDF impementations and add JSONIDF: Resolved
• Sqoop2: Avro IDF class and unit tests: Resolved
• Wrap up the design doc once implementation is done: Resolved
• Sqoop2: Add jackson as a dependency: Resolved
• sqoop2: parseCSV in CSVIDF doesn't correctly parse strings with commas: Resolved
• Sqoop2: Enforce a non empty schema name and column names: Resolved
• Tech debt : Alternative to isTest flag in the SqoopOutputFormatExecutor: Resolved
• Sqoop2: Merge SPI and connector-sdk packages: Resolved
• Sqoop2: Cleanup IDF implementations: Resolved
• Does CSVIDF implementation need to store data field and CSV text: Resolved
• Sqoop2: parseCSVString in SQOOPIDFUtils add more tests: Resolved