Hive
  1. Hive
  2. HIVE-1543

set abort in ExecMapper when Hive's record reader got an IOException

    Details

    • Type: Improvement Improvement
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.6.0
    • Component/s: Query Processor
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      When RecordReader got an IOException, ExecMapper does not know and will close the operators as if there is not error. We should catch this exception and avoid writing partial results to HDFS which will be removed later anyways.

      1. HIVE-1543.2_branch0.6.patch
        4 kB
        Ning Zhang
      2. HIVE-1543_branch0.6.patch
        5 kB
        Ning Zhang
      3. HIVE-1543.1.patch
        5 kB
        Ning Zhang
      4. HIVE-1543.patch
        3 kB
        Ning Zhang

        Activity

        Hide
        Ning Zhang added a comment -

        passed all hadoop 0.20 tests.

        Show
        Ning Zhang added a comment - passed all hadoop 0.20 tests.
        Hide
        He Yongqiang added a comment -

        let's do it in HiveContextAwareRecordReader. And maybe store the var in IOContext?

        Show
        He Yongqiang added a comment - let's do it in HiveContextAwareRecordReader. And maybe store the var in IOContext?
        Hide
        Ning Zhang added a comment -

        @yongqiang, HiveContextAwareRecordReader is only available in 0.7 (trunk) but not 0.6. I think this should be back ported to 0.6 as well (as we did for HIVE-1492)?

        Show
        Ning Zhang added a comment - @yongqiang, HiveContextAwareRecordReader is only available in 0.7 (trunk) but not 0.6. I think this should be back ported to 0.6 as well (as we did for HIVE-1492 )?
        Hide
        Namit Jain added a comment -

        What about BucketizedHiveRecordReader ?

        Show
        Namit Jain added a comment - What about BucketizedHiveRecordReader ?
        Hide
        He Yongqiang added a comment -

        we can do two different patches for trunk and 0.6.

        I think BucketizedHiveRecordReader also extends HiveContextAwareRecordReader.

        Show
        He Yongqiang added a comment - we can do two different patches for trunk and 0.6. I think BucketizedHiveRecordReader also extends HiveContextAwareRecordReader.
        Hide
        Ning Zhang added a comment -

        Uploading 2 patches for trunk and 0.6 respectively.

        Show
        Ning Zhang added a comment - Uploading 2 patches for trunk and 0.6 respectively.
        Hide
        Namit Jain added a comment -

        can you reload the patch for 0.6 - i got some merge conflicts

        Show
        Namit Jain added a comment - can you reload the patch for 0.6 - i got some merge conflicts
        Hide
        Namit Jain added a comment -

        +1 for trunk

        Show
        Namit Jain added a comment - +1 for trunk
        Hide
        Ning Zhang added a comment -

        Uploading a new patch for branch 0.6.

        Show
        Ning Zhang added a comment - Uploading a new patch for branch 0.6.
        Hide
        Namit Jain added a comment -

        +1 for 0.6 also

        Will commit in both once the tests pass

        Show
        Namit Jain added a comment - +1 for 0.6 also Will commit in both once the tests pass
        Hide
        Namit Jain added a comment -

        Committed. Thanks Ning

        Show
        Namit Jain added a comment - Committed. Thanks Ning

          People

          • Assignee:
            Ning Zhang
            Reporter:
            Ning Zhang
          • Votes:
            0 Vote for this issue
            Watchers:
            0 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development