Hive
  1. Hive
  2. HIVE-557

Exception in FileSinkOperator's close should NOT be ignored

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 0.3.0
    • Fix Version/s: 0.4.0
    • Component/s: Query Processor
    • Labels:
      None

      Description

      FileSinkOperator currently ignores all IOExceptions from close() and commit(). We should not ignore them, or the output file can be incomplete or missing.

      1. HIVE-557.1.patch
        0.8 kB
        Zheng Shao
      2. HIVE-557.2.patch
        5 kB
        Zheng Shao
      3. HIVE-557.2.patch
        4 kB
        Zheng Shao

        Activity

        Hide
        Joydeep Sen Sarma added a comment -

        commited again. thanks Zheng.

        Show
        Joydeep Sen Sarma added a comment - commited again. thanks Zheng.
        Hide
        Zheng Shao added a comment -

        Changed Exception/HiveException to Throwable to places where we are setting "abort = true".

        Show
        Zheng Shao added a comment - Changed Exception/HiveException to Throwable to places where we are setting "abort = true".
        Hide
        Zheng Shao added a comment -

        The "close()" is called in the "finally" block in "MapRunner.run()". If there is an out-of-memory error, Hive won't be able to catch it (because we only catch "Exception" now), so abort is NOT set to false, which is wrong.

        We need to catch "Throwable" instead of "Exception".

        Show
        Zheng Shao added a comment - The "close()" is called in the "finally" block in "MapRunner.run()". If there is an out-of-memory error, Hive won't be able to catch it (because we only catch "Exception" now), so abort is NOT set to false, which is wrong. We need to catch "Throwable" instead of "Exception".
        Hide
        Joydeep Sen Sarma added a comment -

        committed. thanks Zheng!

        Show
        Joydeep Sen Sarma added a comment - committed. thanks Zheng!
        Hide
        Zheng Shao added a comment -

        Modified ExecMapper and ExecReducer to catch all Exceptions and set "abort".

        Show
        Zheng Shao added a comment - Modified ExecMapper and ExecReducer to catch all Exceptions and set "abort".
        Hide
        Joydeep Sen Sarma added a comment -

        just talked to Zheng - need to catch all Exceptions from mapoperator and not just hiveexception ..

        Show
        Joydeep Sen Sarma added a comment - just talked to Zheng - need to catch all Exceptions from mapoperator and not just hiveexception ..

          People

          • Assignee:
            Zheng Shao
            Reporter:
            Zheng Shao
          • Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development