Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.14.0
    • Component/s: impl
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      All Orc e2e tests fail on tez.

      There are two issue:
      1. hivelibdir etc is not set in tez.conf
      2. OrcStorage produce empty output file

      Digging into #2, the problem is in this code in PigProcessor:

                      if (fileOutput.isCommitRequired()) {
                          fileOutput.commit();
                      }
      

      fileOutput.commit() invokes both RecordWriter.close() and committer.commitTask(). However, OrcNewOutputFormate will generate output file only after RecordWriter.close (if the output file is small), fileOutput.isCommitRequired will not detect this file, thus skip fileOutput.commit().

      Changing the code to invoke fileOutput.close explicitly fix the issue. fileOutput.commit will invoke close again, but there is no side effect since close will check if it has been already called.

      1. PIG-4086-3.patch
        1 kB
        Daniel Dai
      2. PIG-4086-2.patch
        1 kB
        Daniel Dai
      3. PIG-4086-1.patch
        1 kB
        Daniel Dai

        Issue Links

          Activity

          Daniel Dai created issue -
          Daniel Dai made changes -
          Field Original Value New Value
          Attachment PIG-4086-1.patch [ 12658857 ]
          Hide
          Rohini Palaniswamy added a comment -

          Siddharth Seth told me when I did the patch to selectively start inputs that we should never call close() methods of input or output and only framework should do that. Daniel Dai, can you check with him?

          Show
          Rohini Palaniswamy added a comment - Siddharth Seth told me when I did the patch to selectively start inputs that we should never call close() methods of input or output and only framework should do that. Daniel Dai , can you check with him?
          Daniel Dai made changes -
          Link This issue depends upon TEZ-1351 [ TEZ-1351 ]
          Hide
          Daniel Dai added a comment -

          TEZ-1351 will give us a flush hook. Update the patch to reflect that. That should fix the test once TEZ-1351 checked in.

          Show
          Daniel Dai added a comment - TEZ-1351 will give us a flush hook. Update the patch to reflect that. That should fix the test once TEZ-1351 checked in.
          Daniel Dai made changes -
          Attachment PIG-4086-2.patch [ 12659288 ]
          Hide
          Rohini Palaniswamy added a comment -

          +1

          Show
          Rohini Palaniswamy added a comment - +1
          Hide
          Daniel Dai added a comment -

          Added "getContext().canCommit()" as discussed in TEZ-1351.

          Show
          Daniel Dai added a comment - Added "getContext().canCommit()" as discussed in TEZ-1351 .
          Daniel Dai made changes -
          Attachment PIG-4086-3.patch [ 12659768 ]
          Hide
          Daniel Dai added a comment -

          Patch committed to trunk. Thanks Rohini for review!

          Show
          Daniel Dai added a comment - Patch committed to trunk. Thanks Rohini for review!
          Daniel Dai made changes -
          Status Open [ 1 ] Resolved [ 5 ]
          Hadoop Flags Reviewed [ 10343 ]
          Resolution Fixed [ 1 ]
          Daniel Dai made changes -
          Status Resolved [ 5 ] Closed [ 6 ]

            People

            • Assignee:
              Daniel Dai
              Reporter:
              Daniel Dai
            • Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development