[SQOOP-1588] Sqoop2: Kite connector write data to HDFS

• Type: Sub-task
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 1.99.5
• Component/s: sqoop2-kite-connector
• Assignee: Qian Xu
• Reporter: Qian Xu
• Created: 15/Oct/14 08:25
• Updated: 19/Mar/15 19:35
• Resolved: 18/Nov/14 06:06

Description

Create a basic Kite connector that can write data (i.e. from a jdbc connection) to HDFS.

The scope is defined as follows:

• Destination: HDFS
• File Format: Avro Parquet and CSV.
• Compression Codec: Use default
• Partitioner Strategy: Not supported
• Column Mapping: Not supported

Exposed Configuration:

• [Link] File Format (Enum)
• [To] Dataset URI (String, has a validation check)

Workflow:

• Create a link to Kite Connector
• Create a job with valid configuration (see above)
• Start a job KiteToInitializer will check dataset existence
• Sqoop will create N KiteLoader instances.
• Kite requires an Avro schema for data manipulation, KiteLoader will create an Avro schema from Sqoop schema provided by LoaderContext. As Sqoop schema types are not identical to Avro types, some types will be mapped. The original Sqoop type information will be kept as SqoopType in schema field, which can be used for a reversed type mapping.
• KiteLoader will create a temporary dataset and writes data records into it. If any error occurs, the dataset will be deleted.
• KiteToDestroy will merge all temporary datasets as one dataset.

Further features will be implemented in follow-up JIRAs.

Issue Links

• links to: Review Board