Uploaded image for project: 'IMPALA'
  1. IMPALA
  2. IMPALA-1464

Bug in explain plan: Plan nodes in unpartitioned fragments should have hosts=1.

    Details

      Description

      Query: explain select * from functional.alltypes limit 10
      +-----------------------------------------------------------+
      | Explain String                                            |
      +-----------------------------------------------------------+
      | Estimated Per-Host Requirements: Memory=160.00MB VCores=1 |
      |                                                           |
      | 01:EXCHANGE [UNPARTITIONED]                               |
      | |  limit: 10                                              |
      | |  hosts=3 per-host-mem=unavailable                       | <-- should be hosts=1
      | |  tuple-ids=0 row-size=97B cardinality=10                |
      | |                                                         |
      | 00:SCAN HDFS [functional.alltypes, RANDOM]                |
      |    partitions=24/24 files=24 size=478.45KB                |
      |    table stats: 7300 rows total                           |
      |    column stats: all                                      |
      |    limit: 10                                              |
      |    hosts=3 per-host-mem=160.00MB                          |
      |    tuple-ids=0 row-size=97B cardinality=10                |
      +-----------------------------------------------------------+
      

        Activity

        Hide
        tarmstrong Tim Armstrong added a comment -
        [localhost:21000] > explain select * from functional.alltypes limit 10;
        Query: explain select * from functional.alltypes limit 10
        +-------------------------------------------------------+
        | Explain String                                        |
        +-------------------------------------------------------+
        | Per-Host Resource Reservation: Memory=0B              |
        | Per-Host Resource Estimates: Memory=16.00MB           |
        |                                                       |
        | F00:PLAN FRAGMENT [UNPARTITIONED] hosts=1 instances=1 |
        | PLAN-ROOT SINK                                        |
        | |  mem-estimate=0B mem-reservation=0B                 |
        | |                                                     |
        | 00:SCAN HDFS [functional.alltypes]                    |
        |    partitions=24/24 files=24 size=478.45KB            |
        |    stats-rows=7300 extrapolated-rows=disabled         |
        |    table stats: rows=7300 size=478.45KB               |
        |    column stats: all                                  |
        |    limit: 10                                          |
        |    mem-estimate=16.00MB mem-reservation=0B            |
        |    tuple-ids=0 row-size=97B cardinality=10            |
        +-------------------------------------------------------+
        

        Fixed by
        commit 9a29dfc91b1ff8bbae3c94b53bf2b6ac81a271e0
        Author: Tim Armstrong <tarmstrong@cloudera.com>
        Date: Wed Jan 25 15:19:35 2017 -0800

        IMPALA-3748: minimum buffer requirements in planner

        Change-Id: I1e358182bcf2bc5fe5c73883eb97878735b12d37
        Reviewed-on: http://gerrit.cloudera.org:8080/5847
        Reviewed-by: Tim Armstrong <tarmstrong@cloudera.com>
        Tested-by: Impala Public Jenkins

        Show
        tarmstrong Tim Armstrong added a comment - [localhost:21000] > explain select * from functional.alltypes limit 10; Query: explain select * from functional.alltypes limit 10 +-------------------------------------------------------+ | Explain String | +-------------------------------------------------------+ | Per-Host Resource Reservation: Memory=0B | | Per-Host Resource Estimates: Memory=16.00MB | | | | F00:PLAN FRAGMENT [UNPARTITIONED] hosts=1 instances=1 | | PLAN-ROOT SINK | | | mem-estimate=0B mem-reservation=0B | | | | | 00:SCAN HDFS [functional.alltypes] | | partitions=24/24 files=24 size=478.45KB | | stats-rows=7300 extrapolated-rows=disabled | | table stats: rows=7300 size=478.45KB | | column stats: all | | limit: 10 | | mem-estimate=16.00MB mem-reservation=0B | | tuple-ids=0 row-size=97B cardinality=10 | +-------------------------------------------------------+ Fixed by commit 9a29dfc91b1ff8bbae3c94b53bf2b6ac81a271e0 Author: Tim Armstrong <tarmstrong@cloudera.com> Date: Wed Jan 25 15:19:35 2017 -0800 IMPALA-3748 : minimum buffer requirements in planner Change-Id: I1e358182bcf2bc5fe5c73883eb97878735b12d37 Reviewed-on: http://gerrit.cloudera.org:8080/5847 Reviewed-by: Tim Armstrong <tarmstrong@cloudera.com> Tested-by: Impala Public Jenkins
        Hide
        ippokratis Ippokratis Pandis added a comment -

        Casey, assigning it to you, as you have looked at it. Feel free to close if won't fix or reassign.

        Show
        ippokratis Ippokratis Pandis added a comment - Casey, assigning it to you, as you have looked at it. Feel free to close if won't fix or reassign.
        Hide
        caseyc casey added a comment -

        I spent over a day looking at this and don't think it's worth fixing (though it was a good learning exercise). The fix is messy because the ExchangeNode.computeStats() function would need access to it's own fragment to determine if the exchange context is a broadcast or not, but that fragment is often null at the time the function is called. I started going down the path of ensuring the fragment is non-null but quickly got into situations that required refactoring core functionality. I just don't think it's worth the risk when the end benefit is just fixing a minor display issue in the plan.

        Show
        caseyc casey added a comment - I spent over a day looking at this and don't think it's worth fixing (though it was a good learning exercise). The fix is messy because the ExchangeNode.computeStats() function would need access to it's own fragment to determine if the exchange context is a broadcast or not, but that fragment is often null at the time the function is called. I started going down the path of ensuring the fragment is non-null but quickly got into situations that required refactoring core functionality. I just don't think it's worth the risk when the end benefit is just fixing a minor display issue in the plan.

          People

          • Assignee:
            tarmstrong Tim Armstrong
            Reporter:
            alex.behm Alexander Behm
          • Votes:
            0 Vote for this issue
            Watchers:
            6 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development