[PIG-4086] Fix Orc e2e tests for tez

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 0.14.0
• Component/s: impl
• Hadoop Flags: Reviewed
• Assignee: Daniel Dai
• Reporter: Daniel Dai
• Created: 31/Jul/14 07:28
• Updated: 21/Nov/14 05:59
• Resolved: 05/Aug/14 00:11

Description

All Orc e2e tests fail on tez.

There are two issue:
1. hivelibdir etc is not set in tez.conf
2. OrcStorage produce empty output file

Digging into #2, the problem is in this code in PigProcessor:

                if (fileOutput.isCommitRequired()) {
                    fileOutput.commit();
                }

fileOutput.commit() invokes both RecordWriter.close() and committer.commitTask(). However, OrcNewOutputFormate will generate output file only after RecordWriter.close (if the output file is small), fileOutput.isCommitRequired will not detect this file, thus skip fileOutput.commit().

Changing the code to invoke fileOutput.close explicitly fix the issue. fileOutput.commit will invoke close again, but there is no side effect since close will check if it has been already called.

Issue Links

• depends upon: TEZ-1351 MROutput needs a flush method to ensure data is materialized for FileOutputCommitter