Uploaded image for project: 'IMPALA'
  1. IMPALA
  2. IMPALA-4816

There is no way to determine if a filter was dropped due to OOM based on profile

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • Impala 2.6.0, Impala 2.7.0, Impala 2.8.0, Impala 2.9.0
    • Impala 2.12.0
    • Backend

    Description

      I just spent some time debugging a test failure that was the result of a bloom filter not being sent due to out of memory. There's no obvious way to tell if this happened by looking at the profile - it looks like the filter arrived and was just ineffective. It looks like the filter can be dropped in RuntimeFilterBank::PublishGlobalFilter() and PhjBuilder::AllocateRuntimeFilters()

                Filter 0 (128.00 MB):
                   - Files processed: 8 (8)
                   - Files rejected: 0 (0)
                   - Files total: 8 (8)
                   - RowGroups processed: 8 (8)
                   - RowGroups rejected: 0 (0)
                   - RowGroups total: 8 (8)
                   - Rows processed: 2.46K (2460)
                   - Rows rejected: 0 (0)
                   - Rows total: 2.46K (2460)
                   - Splits processed: 8 (8)
                   - Splits rejected: 0 (0)
                   - Splits total: 8 (8)
      

      I ran into it with the follow query on my buffer pool branch. You can probably replicate it on master by tweaking mem_limit.

      [localhost:21000] > use functional_parquet; SET RUNTIME_FILTER_MODE=GLOBAL; SET RUNTIME_FILTER_WAIT_TIME_MS=30000; SET RUNTIME_FILTER_MIN_SIZE=128MB; SET RUNTIME_FILTER_MAX_SIZE=500MB; SET MEM_LIMIT=140MB; select STRAIGHT_JOIN * from alltypes a join [BROADCAST] alltypes b on a.month = b.id and b.int_col = -3;
      

      Attachments

        Activity

          People

            bikramjeet.vig Bikramjeet Vig
            tarmstrong Tim Armstrong
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: