Uploaded image for project: 'Hive'
  1. Hive
  2. HIVE-10209

FetchTask with VC may fail because ExecMapper.done is true

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Closed
    • Major
    • Resolution: Fixed
    • 1.1.0
    • 1.2.0
    • Query Processor
    • None

    Description

      ExecMapper.done is a static variable, and may cause issues in the following example:

      set hive.fetch.task.conversion=minimal;
      select * from src where key < 10 limit 1;
      set hive.fetch.task.conversion=more;
      select *, BLOCK__OFFSET_INSIDE__FILE from src where key < 10;
      

      The second select won't return any result, if running in local mode.

      The issue is, the first select query will be converted to a MapRedTask with only a mapper. And, when the task is done, because of the limit operator, ExecMapper.done will be set to true.

      Then, when the second select query begin to execute, it will call FetchOperator::getRecordReader(), and since here we have virtual column, an instance of HiveRecordReader will be returned. The problem is, HiveRecordReader::doNext() will check ExecMapper.done. In this case, since the value is true, it will quit immediately.

      In short, I think making ExecMapper.done static is a bad idea. The first query should in no way affect the second one.

      Attachments

        1. HIVE-10209.1-spark.patch
          1 kB
          Chao Sun
        2. HIVE-10209.2-spark.patch
          1 kB
          Chao Sun

        Issue Links

          Activity

            People

              csun Chao Sun
              csun Chao Sun
              Votes:
              0 Vote for this issue
              Watchers:
              6 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: