[HIVE-360] Generalize the FileFormat Interface in Hive

• Type: Improvement
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 0.4.0
• Fix Version/s: 0.4.0
• Hadoop Flags: Reviewed
• Release Note: HIVE-360. Generalize the FileFormat Interface in Hive. (He Yongqiang via zshao)
• Assignee: He Yongqiang
• Reporter: Zheng Shao
• Created: 24/Mar/09 08:10
• Updated: 17/Dec/11 00:07
• Resolved: 10/Apr/09 06:48

Description

Currently the FileFormat support in Hive is not generalized - we do "if ... else" to support TextFileFormat and SequenceFileFormat. There is no way to support a 3rd one without changing the "if...else" structure. We should make an interface for the FileFormat need for Hive.

The OutputFileFormat interface that Hive requires will contain one more method than the Hadoop OutputFileFormat - create a File with a specific name.

Hive.g:409 (Hive.g already supports the custom file format but DDLSemanticAnalyzer.java is not recognizing it yet

KW_STORED KW_AS KW_INPUTFORMAT inFmt=StringLiteral KW_OUTPUTFORMAT outFmt=StringLiteral

Please add the handling of TOK_TABLEFILEFORMAT here:
DDLSemanticAnalyzer.java:223

        case HiveParser.TOK_TBLSEQUENCEFILE:
        ...

Please add the handling of custom outputFormat here by adding a new interface (and cast the user-provided file format to that interface), instead of doing "if ... else"
FileSinkOperator.java:129-174:

      if(outputFormat instanceof IgnoreKeyTextOutputFormat) {
        finalPath = new Path(Utilities.toTempPath(conf.getDirName()), Utilities.getTaskId(hconf) +
                             Utilities.getFileExtension(jc, isCompressed));
      ...

Issue Links

• blocks: HIVE-352 Make Hive support column based storage