Uploaded image for project: 'Apache Drill'
  1. Apache Drill
  2. DRILL-3333

Add support for auto-partitioning in parquet writer

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 1.1.0
    • Component/s: None
    • Labels:
      None

      Description

      When a table is created with a partition by clause, the parquet writer will create separate files for the different partition values. The data will first be sorted by the partition keys, and the parquet writer will create new file when it encounters a new value for the partition columns.

      When data is queried against the data that was created this way, partition pruning will work if the filter contains a partition column. And unlike directory based partitioning, no view is required, nor is it necessary to reference the dir* column names.

        Attachments

        1. DRILL-3333_2015-06-22_15:22:11.patch
          57 kB
          Steven Phillips
        2. DRILL-3333_2015-06-23_17:38:32.patch
          85 kB
          Steven Phillips
        3. DRILL-3333_tests.patch
          10 kB
          Steven Phillips
        4. DRILL-3333.patch
          57 kB
          Steven Phillips
        5. DRILL-3333.patch
          65 kB
          Steven Phillips

          Activity

            People

            • Assignee:
              sphillips Steven Phillips
              Reporter:
              sphillips Steven Phillips
            • Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: