Uploaded image for project: 'Apache Drill'
  1. Apache Drill
  2. DRILL-2757 Verify operators correctly handle low memory conditions and cancellations
  3. DRILL-2893

ScanBatch throws a NullPointerException instead of returning OUT_OF_MEMORY

    XMLWordPrintableJSON

    Details

      Description

      • set drill.exec.memory.top.max in drill-override.conf to some low value (I used 75000000)
      • disable hash aggregate (set plannder.enable_hashagg to false)
      • disable exchanges (set plannder.disable_exchanges to true)
      • run the following query
        select count(*) from (select * from dfs.data.`tpch1/lineitem.parquet` order by l_orderkey);
        

        and you should get the following error message:

        Query failed: SYSTEM ERROR: null
        
        Fragment 0:0
        
        [e05ff3c2-e130-449e-b721-b3442796e29b on 172.30.1.1:31010]
        

      We have 2 problems here:

      1st:

      • ScanBatch detects that it can't allocate it's field value vectors and right before returning OUT_OF_MEMORY downstream it calls _clear() on the field vectors
      • one of those vectors actually threw a NullPointerException in it's allocateNew() methods after it cleared it's buffer and couldn't allocate a new one
      • when ScanBatch tries to clean that vector, it will throw a NullPointerException which will prevent the ScanBatch from returning OUT_OF_MEMORY and will cancel the query instead

      2nd problem:

      • once the query has been canceled, ScanBatch.cleanup() will throw another NullPointerException when cleaning the field vectors, which will prevent the cleanup of the remaining resources and will cause a memory leak

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                sphillips Steven Phillips
                Reporter:
                adeneche Abdel Hakim Deneche
              • Votes:
                0 Vote for this issue
                Watchers:
                1 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: