Uploaded image for project: 'Apache Drill'
  1. Apache Drill
  2. DRILL-5991

Performance improvements for Hive tables with skip header / footer logic



    • Improvement
    • Status: Open
    • Major
    • Resolution: Unresolved
    • 1.12.0
    • None
    • Storage - Hive
    • None


      Currently when Hive table has header / footer all input split of the file are processed by one reader. This has performance impact better way would be to keep one reader per split and see if we can figure out a way to tell readers how many rows they should skip.

      To create reader for each input split and maintain skip header / footer functionality we need to know how many rows are in input split. Unfortunately, input split does not hold such information, only number of bytes. We can't apply skip header functionality for the first input split and skip footer for the last input either since we don't know how many rows will be skipped, it can be the situation that we need to skip the whole first input split and partially second. Also we use Hadoop reader for the data and don't have information about number of rows in input split.

      Possible improvements:
      1. For table with header only before creating readers we can start skipping header and when done, create reader at that position, for other untouched input splits create separate readers though all readers will be on the same node.
      2. Consider Drill text reader usage instead of Hadoop one (as we do for parquet files) which might provide more flexibility in terms of offsetting bytes etc. This should be investigated further.


        Issue Links



              Unassigned Unassigned
              arina Arina Ielchiieva
              0 Vote for this issue
              1 Start watching this issue