[HIVE-20225] SerDe to support Teradata Binary Format

• Type: New Feature
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 2.4.0, 3.2.0, 4.0.0-alpha-1
• Component/s: Serializers/Deserializers
• Target Version/s: 2.4.0, 3.2.0
• Hadoop Flags: Reviewed
• Assignee: Lu Li
• Reporter: Lu Li
• Created: 23/Jul/18 21:19
• Updated: 17/Nov/22 08:55
• Resolved: 11/Sep/18 20:00

Description

When using TPT/BTEQ to export/import Data from Teradata, Teradata will generate/require binary files based on the schema.

A Customized SerDe is needed in order to directly read these files from Hive or write these files in order to load back to TD.

CREATE EXTERNAL TABLE `TABLE1`(
...)
PARTITIONED BY (
...)
ROW FORMAT SERDE
  'org.apache.hadoop.hive.contrib.serde2.TeradataBinarySerde'
STORED AS INPUTFORMAT
 'org.apache.hadoop.hive.contrib.fileformat.teradata.TeradataBinaryFileInputFormat'
OUTPUTFORMAT
 'org.apache.hadoop.hive.contrib.fileformat.teradata.TeradataBinaryFileOutputFormat'
LOCATION ...;

SELECT * FROM `TABLE1`;

Problem Statement:

Right now the fast way to export/import data from Teradata is using TPT. However, the Hive could not directly utilize/generate these binary format because it doesn't have a SerDe for these files.

Result:

Provided with the SerDe, Hive can operate upon/generate the exported Teradata Binary Format file transparently