Uploaded image for project: 'Hive'
  1. Hive
  2. HIVE-557

Exception in FileSinkOperator's close should NOT be ignored

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 0.3.0
    • Fix Version/s: 0.4.0
    • Component/s: Query Processor
    • Labels:
      None

      Description

      FileSinkOperator currently ignores all IOExceptions from close() and commit(). We should not ignore them, or the output file can be incomplete or missing.

      1. HIVE-557.1.patch
        0.8 kB
        Zheng Shao
      2. HIVE-557.2.patch
        4 kB
        Zheng Shao
      3. HIVE-557.2.patch
        5 kB
        Zheng Shao

        Activity

        Hide
        jsensarma Joydeep Sen Sarma added a comment -

        just talked to Zheng - need to catch all Exceptions from mapoperator and not just hiveexception ..

        Show
        jsensarma Joydeep Sen Sarma added a comment - just talked to Zheng - need to catch all Exceptions from mapoperator and not just hiveexception ..
        Hide
        zshao Zheng Shao added a comment -

        Modified ExecMapper and ExecReducer to catch all Exceptions and set "abort".

        Show
        zshao Zheng Shao added a comment - Modified ExecMapper and ExecReducer to catch all Exceptions and set "abort".
        Hide
        jsensarma Joydeep Sen Sarma added a comment -

        committed. thanks Zheng!

        Show
        jsensarma Joydeep Sen Sarma added a comment - committed. thanks Zheng!
        Hide
        zshao Zheng Shao added a comment -

        The "close()" is called in the "finally" block in "MapRunner.run()". If there is an out-of-memory error, Hive won't be able to catch it (because we only catch "Exception" now), so abort is NOT set to false, which is wrong.

        We need to catch "Throwable" instead of "Exception".

        Show
        zshao Zheng Shao added a comment - The "close()" is called in the "finally" block in "MapRunner.run()". If there is an out-of-memory error, Hive won't be able to catch it (because we only catch "Exception" now), so abort is NOT set to false, which is wrong. We need to catch "Throwable" instead of "Exception".
        Hide
        zshao Zheng Shao added a comment -

        Changed Exception/HiveException to Throwable to places where we are setting "abort = true".

        Show
        zshao Zheng Shao added a comment - Changed Exception/HiveException to Throwable to places where we are setting "abort = true".
        Hide
        jsensarma Joydeep Sen Sarma added a comment -

        commited again. thanks Zheng.

        Show
        jsensarma Joydeep Sen Sarma added a comment - commited again. thanks Zheng.

          People

          • Assignee:
            zshao Zheng Shao
            Reporter:
            zshao Zheng Shao
          • Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development