Hive
  1. Hive
  2. HIVE-1543

set abort in ExecMapper when Hive's record reader got an IOException

    Details

    • Type: Improvement Improvement
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.6.0
    • Component/s: Query Processor
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      When RecordReader got an IOException, ExecMapper does not know and will close the operators as if there is not error. We should catch this exception and avoid writing partial results to HDFS which will be removed later anyways.

      1. HIVE-1543.patch
        3 kB
        Ning Zhang
      2. HIVE-1543.1.patch
        5 kB
        Ning Zhang
      3. HIVE-1543_branch0.6.patch
        5 kB
        Ning Zhang
      4. HIVE-1543.2_branch0.6.patch
        4 kB
        Ning Zhang

        Activity

        Ning Zhang created issue -
        Hide
        Ning Zhang added a comment -

        passed all hadoop 0.20 tests.

        Show
        Ning Zhang added a comment - passed all hadoop 0.20 tests.
        Ning Zhang made changes -
        Field Original Value New Value
        Attachment HIVE-1543.patch [ 12452213 ]
        Ning Zhang made changes -
        Status Open [ 1 ] Patch Available [ 10002 ]
        Fix Version/s 0.7.0 [ 12315150 ]
        Hide
        He Yongqiang added a comment -

        let's do it in HiveContextAwareRecordReader. And maybe store the var in IOContext?

        Show
        He Yongqiang added a comment - let's do it in HiveContextAwareRecordReader. And maybe store the var in IOContext?
        Hide
        Ning Zhang added a comment -

        @yongqiang, HiveContextAwareRecordReader is only available in 0.7 (trunk) but not 0.6. I think this should be back ported to 0.6 as well (as we did for HIVE-1492)?

        Show
        Ning Zhang added a comment - @yongqiang, HiveContextAwareRecordReader is only available in 0.7 (trunk) but not 0.6. I think this should be back ported to 0.6 as well (as we did for HIVE-1492 )?
        Hide
        Namit Jain added a comment -

        What about BucketizedHiveRecordReader ?

        Show
        Namit Jain added a comment - What about BucketizedHiveRecordReader ?
        Hide
        He Yongqiang added a comment -

        we can do two different patches for trunk and 0.6.

        I think BucketizedHiveRecordReader also extends HiveContextAwareRecordReader.

        Show
        He Yongqiang added a comment - we can do two different patches for trunk and 0.6. I think BucketizedHiveRecordReader also extends HiveContextAwareRecordReader.
        Hide
        Ning Zhang added a comment -

        Uploading 2 patches for trunk and 0.6 respectively.

        Show
        Ning Zhang added a comment - Uploading 2 patches for trunk and 0.6 respectively.
        Ning Zhang made changes -
        Attachment HIVE-1543.1.patch [ 12452233 ]
        Attachment HIVE-1543_branch0.6.patch [ 12452234 ]
        Ning Zhang made changes -
        Fix Version/s 0.6.0 [ 12314524 ]
        Fix Version/s 0.7.0 [ 12315150 ]
        Hide
        Namit Jain added a comment -

        can you reload the patch for 0.6 - i got some merge conflicts

        Show
        Namit Jain added a comment - can you reload the patch for 0.6 - i got some merge conflicts
        Hide
        Namit Jain added a comment -

        +1 for trunk

        Show
        Namit Jain added a comment - +1 for trunk
        Hide
        Ning Zhang added a comment -

        Uploading a new patch for branch 0.6.

        Show
        Ning Zhang added a comment - Uploading a new patch for branch 0.6.
        Ning Zhang made changes -
        Attachment HIVE-1543.2_branch0.6.patch [ 12452240 ]
        Hide
        Namit Jain added a comment -

        +1 for 0.6 also

        Will commit in both once the tests pass

        Show
        Namit Jain added a comment - +1 for 0.6 also Will commit in both once the tests pass
        Hide
        Namit Jain added a comment -

        Committed. Thanks Ning

        Show
        Namit Jain added a comment - Committed. Thanks Ning
        Namit Jain made changes -
        Status Patch Available [ 10002 ] Resolved [ 5 ]
        Hadoop Flags [Reviewed]
        Resolution Fixed [ 1 ]
        Carl Steinbach made changes -
        Component/s Query Processor [ 12312586 ]
        Carl Steinbach made changes -
        Status Resolved [ 5 ] Closed [ 6 ]

          People

          • Assignee:
            Ning Zhang
            Reporter:
            Ning Zhang
          • Votes:
            0 Vote for this issue
            Watchers:
            0 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development