Hive
  1. Hive
  2. HIVE-557

Exception in FileSinkOperator's close should NOT be ignored

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 0.3.0
    • Fix Version/s: 0.4.0
    • Component/s: Query Processor
    • Labels:
      None

      Description

      FileSinkOperator currently ignores all IOExceptions from close() and commit(). We should not ignore them, or the output file can be incomplete or missing.

      1. HIVE-557.2.patch
        4 kB
        Zheng Shao
      2. HIVE-557.2.patch
        5 kB
        Zheng Shao
      3. HIVE-557.1.patch
        0.8 kB
        Zheng Shao

        Activity

        Transition Time In Source Status Execution Times Last Executer Last Execution Date
        Open Open Patch Available Patch Available
        1m 52s 1 Zheng Shao 11/Jun/09 18:10
        Patch Available Patch Available Resolved Resolved
        23h 2m 1 Joydeep Sen Sarma 12/Jun/09 17:12
        Resolved Resolved Reopened Reopened
        3d 3h 45m 1 Zheng Shao 15/Jun/09 20:58
        Reopened Reopened Resolved Resolved
        3h 52m 1 Joydeep Sen Sarma 16/Jun/09 00:50
        Resolved Resolved Closed Closed
        913d 23h 17m 1 Carl Steinbach 17/Dec/11 00:07
        Carl Steinbach made changes -
        Status Resolved [ 5 ] Closed [ 6 ]
        Carl Steinbach made changes -
        Affects Version/s 0.3.1 [ 12313845 ]
        Joydeep Sen Sarma made changes -
        Status Reopened [ 4 ] Resolved [ 5 ]
        Resolution Fixed [ 1 ]
        Hide
        Joydeep Sen Sarma added a comment -

        commited again. thanks Zheng.

        Show
        Joydeep Sen Sarma added a comment - commited again. thanks Zheng.
        Zheng Shao made changes -
        Attachment HIVE-557.2.patch [ 12410712 ]
        Hide
        Zheng Shao added a comment -

        Changed Exception/HiveException to Throwable to places where we are setting "abort = true".

        Show
        Zheng Shao added a comment - Changed Exception/HiveException to Throwable to places where we are setting "abort = true".
        Zheng Shao made changes -
        Resolution Fixed [ 1 ]
        Status Resolved [ 5 ] Reopened [ 4 ]
        Hide
        Zheng Shao added a comment -

        The "close()" is called in the "finally" block in "MapRunner.run()". If there is an out-of-memory error, Hive won't be able to catch it (because we only catch "Exception" now), so abort is NOT set to false, which is wrong.

        We need to catch "Throwable" instead of "Exception".

        Show
        Zheng Shao added a comment - The "close()" is called in the "finally" block in "MapRunner.run()". If there is an out-of-memory error, Hive won't be able to catch it (because we only catch "Exception" now), so abort is NOT set to false, which is wrong. We need to catch "Throwable" instead of "Exception".
        Joydeep Sen Sarma made changes -
        Affects Version/s 0.3.0 [ 12313637 ]
        Affects Version/s 0.3.1 [ 12313845 ]
        Affects Version/s 0.4.0 [ 12313714 ]
        Joydeep Sen Sarma made changes -
        Status Patch Available [ 10002 ] Resolved [ 5 ]
        Fix Version/s 0.4.0 [ 12313714 ]
        Resolution Fixed [ 1 ]
        Hide
        Joydeep Sen Sarma added a comment -

        committed. thanks Zheng!

        Show
        Joydeep Sen Sarma added a comment - committed. thanks Zheng!
        Zheng Shao made changes -
        Attachment HIVE-557.2.patch [ 12410439 ]
        Hide
        Zheng Shao added a comment -

        Modified ExecMapper and ExecReducer to catch all Exceptions and set "abort".

        Show
        Zheng Shao added a comment - Modified ExecMapper and ExecReducer to catch all Exceptions and set "abort".
        Hide
        Joydeep Sen Sarma added a comment -

        just talked to Zheng - need to catch all Exceptions from mapoperator and not just hiveexception ..

        Show
        Joydeep Sen Sarma added a comment - just talked to Zheng - need to catch all Exceptions from mapoperator and not just hiveexception ..
        Zheng Shao made changes -
        Status Open [ 1 ] Patch Available [ 10002 ]
        Zheng Shao made changes -
        Field Original Value New Value
        Attachment HIVE-557.1.patch [ 12410420 ]
        Zheng Shao created issue -

          People

          • Assignee:
            Zheng Shao
            Reporter:
            Zheng Shao
          • Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development