[TEZ-1351] MROutput needs a flush method to ensure data is materialized for FileOutputCommitter

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 0.5.0
• Hadoop Flags: Incompatible change, Reviewed
• Assignee: Bikas Saha
• Reporter: Daniel Dai
• Created: 31/Jul/14 18:51
• Updated: 06/Sep/14 01:35
• Resolved: 04/Aug/14 21:58

Description

In MROutput.commit, we need to check isCommitRequired before invoking commitTask.

Currently we did this check inside Pig:

                if (fileOutput.isCommitRequired()) {
                    fileOutput.commit();
                }

However, in some loader, output file is generated only after fileOutput.close, which is part of fileOutput.commit. The isCommitRequired check is too early. A walk around is to invoke fileOutput.close before isCommitRequired:

                fileOutput.close();
                if (fileOutput.isCommitRequired()) {
                    fileOutput.commit();
                }

But we are told there is a plan to make MROutput.close private.

Issue Links

• is depended upon by: PIG-4086 Fix Orc e2e tests for tez