Uploaded image for project: 'Apache Drill'
  1. Apache Drill
  2. DRILL-4786

Improve metadata cache performance for queries with multiple partitions

    XMLWordPrintableJSON

    Details

      Description

      Consider queries of the following type run against Parquet data with metadata caching:

      SELECT col FROM `A` WHERE dir0 = 'B`' AND dir1 IN ('1', '2', '3')
      

      For such queries, Drill will read the metadata cache file from the top level directory 'A', which is not very efficient since we are only interested in the files from some subdirectories of 'B'. DRILL-4530 improves the performance of such queries when the leaf level directory is a single partition. Here, there are 3 subpartitions due to the IN list. We can build upon the DRILL-4530 enhancement by at least reading the cache file from the immediate parent level `/A/B` instead of the top level.

      The goal of this JIRA is to improve performance for such types of queries.

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                amansinha100 Aman Sinha
                Reporter:
                amansinha100 Aman Sinha
                Reviewer:
                Rahul Kumar Challapalli
              • Votes:
                0 Vote for this issue
                Watchers:
                3 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: