Uploaded image for project: 'Hive'
  1. Hive
  2. HIVE-10209

FetchTask with VC may fail because ExecMapper.done is true

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 1.1.0
    • Fix Version/s: 1.2.0
    • Component/s: Query Processor
    • Labels:
      None

      Description

      ExecMapper.done is a static variable, and may cause issues in the following example:

      set hive.fetch.task.conversion=minimal;
      select * from src where key < 10 limit 1;
      set hive.fetch.task.conversion=more;
      select *, BLOCK__OFFSET_INSIDE__FILE from src where key < 10;
      

      The second select won't return any result, if running in local mode.

      The issue is, the first select query will be converted to a MapRedTask with only a mapper. And, when the task is done, because of the limit operator, ExecMapper.done will be set to true.

      Then, when the second select query begin to execute, it will call FetchOperator::getRecordReader(), and since here we have virtual column, an instance of HiveRecordReader will be returned. The problem is, HiveRecordReader::doNext() will check ExecMapper.done. In this case, since the value is true, it will quit immediately.

      In short, I think making ExecMapper.done static is a bad idea. The first query should in no way affect the second one.

        Attachments

        1. HIVE-10209.1-spark.patch
          1 kB
          Chao Sun
        2. HIVE-10209.2-spark.patch
          1 kB
          Chao Sun

          Issue Links

            Activity

              People

              • Assignee:
                csun Chao Sun
                Reporter:
                csun Chao Sun
              • Votes:
                0 Vote for this issue
                Watchers:
                6 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: