Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-20901

Feature parity for ORC with Parquet

    XMLWordPrintableJSON

Details

    • Improvement
    • Status: Open
    • Major
    • Resolution: Unresolved
    • 3.1.0
    • None
    • SQL
    • None

    Description

      This issue aims to track the feature parity for ORC with Parquet.

      Attachments

        Issue Links

          is blocked by

          Bug - A problem which impairs or prevents the functions of the product. SPARK-15705 Spark won't read ORC schema from metastore for partitioned tables

          • Critical - Crashes, loss of data, severe memory leak.
          • Resolved

          Bug - A problem which impairs or prevents the functions of the product. SPARK-22258 Writing empty dataset fails with ORC format

          • Critical - Crashes, loss of data, severe memory leak.
          • Resolved

          Bug - A problem which impairs or prevents the functions of the product. SPARK-25306 Avoid skewed filter trees to speed up `createFilter` in ORC

          • Critical - Crashes, loss of data, severe memory leak.
          • Resolved

          Bug - A problem which impairs or prevents the functions of the product. SPARK-14387 Enable Hive-1.x ORC compatibility with spark.sql.hive.convertMetastoreOrc

          • Major - Major loss of function.
          • Resolved

          Bug - A problem which impairs or prevents the functions of the product. SPARK-15347 Problem select empty ORC table

          • Major - Major loss of function.
          • Resolved

          Bug - A problem which impairs or prevents the functions of the product. SPARK-15474 ORC data source fails to write and read back empty dataframe

          • Major - Major loss of function.
          • Resolved

          Bug - A problem which impairs or prevents the functions of the product. SPARK-15731 orc writer directory permissions

          • Major - Major loss of function.
          • Resolved

          Bug - A problem which impairs or prevents the functions of the product. SPARK-15757 Error occurs when using Spark sql "select" statement on orc file after hive sql "insert overwrite tb1 select * from sourcTb" has been executed on this orc file

          • Major - Major loss of function.
          • Resolved

          Bug - A problem which impairs or prevents the functions of the product. SPARK-16628 OrcConversions should not convert an ORC table represented by MetastoreRelation to HadoopFsRelation if metastore schema does not match schema stored in ORC files

          • Major - Major loss of function.
          • Resolved

          Bug - A problem which impairs or prevents the functions of the product. SPARK-17047 Spark 2 cannot create table when CLUSTERED.

          • Major - Major loss of function.
          • Resolved

          Bug - A problem which impairs or prevents the functions of the product. SPARK-18355 Spark SQL fails to read data from a ORC hive table that has a new column added to it

          • Major - Major loss of function.
          • Resolved

          Bug - A problem which impairs or prevents the functions of the product. SPARK-19109 ORC metadata section can sometimes exceed protobuf message size limit

          • Major - Major loss of function.
          • Resolved

          Bug - A problem which impairs or prevents the functions of the product. SPARK-19430 Cannot read external tables with VARCHAR columns if they're backed by ORC files written by Hive 1.2.1

          • Major - Major loss of function.
          • Resolved

          Bug - A problem which impairs or prevents the functions of the product. SPARK-19809 NullPointerException on zero-size ORC file

          • Major - Major loss of function.
          • Resolved

          Bug - A problem which impairs or prevents the functions of the product. SPARK-20515 Issue with reading Hive ORC tables having char/varchar columns in Spark SQL

          • Major - Major loss of function.
          • Resolved

          Bug - A problem which impairs or prevents the functions of the product. SPARK-21422 Depend on Apache ORC 1.4.0

          • Major - Major loss of function.
          • Resolved

          Bug - A problem which impairs or prevents the functions of the product. SPARK-21686 spark.sql.hive.convertMetastoreOrc is causing NullPointerException while reading ORC tables

          • Major - Major loss of function.
          • Resolved

          Bug - A problem which impairs or prevents the functions of the product. SPARK-21762 FileFormatWriter/BasicWriteTaskStatsTracker metrics collection fails if a new file isn't yet visible

          • Major - Major loss of function.
          • Resolved

          Bug - A problem which impairs or prevents the functions of the product. SPARK-21912 ORC/Parquet table should not create invalid column names

          • Major - Major loss of function.
          • Resolved

          Bug - A problem which impairs or prevents the functions of the product. SPARK-21929 Support `ALTER TABLE table_name ADD COLUMNS(..)` for ORC data source

          • Major - Major loss of function.
          • Resolved

          Bug - A problem which impairs or prevents the functions of the product. SPARK-22158 convertMetastore should not ignore storage properties

          • Major - Major loss of function.
          • Resolved

          Bug - A problem which impairs or prevents the functions of the product. SPARK-22267 Spark SQL incorrectly reads ORC file when column order is different

          • Major - Major loss of function.
          • Resolved

          Bug - A problem which impairs or prevents the functions of the product. SPARK-22279 Turn on spark.sql.hive.convertMetastoreOrc by default

          • Major - Major loss of function.
          • Resolved

          Bug - A problem which impairs or prevents the functions of the product. SPARK-22300 Update ORC to 1.4.1

          • Major - Major loss of function.
          • Resolved

          Bug - A problem which impairs or prevents the functions of the product. SPARK-22712 Use `buildReaderWithPartitionValues` in native OrcFileFormat

          • Major - Major loss of function.
          • Resolved

          Bug - A problem which impairs or prevents the functions of the product. SPARK-23007 Add schema evolution test suite for file-based data sources

          • Major - Major loss of function.
          • Resolved

          Bug - A problem which impairs or prevents the functions of the product. SPARK-23049 `spark.sql.files.ignoreCorruptFiles` should work for ORC files

          • Major - Major loss of function.
          • Resolved

          Bug - A problem which impairs or prevents the functions of the product. SPARK-23340 Upgrade Apache ORC to 1.4.3

          • Major - Major loss of function.
          • Resolved

          Bug - A problem which impairs or prevents the functions of the product. SPARK-23355 convertMetastore should not ignore table properties

          • Major - Major loss of function.
          • Resolved

          Bug - A problem which impairs or prevents the functions of the product. SPARK-23399 Register a task completion listener first for OrcColumnarBatchReader

          • Major - Major loss of function.
          • Resolved

          Bug - A problem which impairs or prevents the functions of the product. SPARK-24322 Upgrade Apache ORC to 1.4.4

          • Major - Major loss of function.
          • Resolved

          Bug - A problem which impairs or prevents the functions of the product. SPARK-24472 Orc RecordReaderFactory throws IndexOutOfBoundsException

          • Major - Major loss of function.
          • Resolved

          Bug - A problem which impairs or prevents the functions of the product. SPARK-25175 Field resolution should fail if there's ambiguity for ORC native reader

          • Major - Major loss of function.
          • Resolved

          Bug - A problem which impairs or prevents the functions of the product. SPARK-25427 Add BloomFilter creation test cases

          • Major - Major loss of function.
          • Resolved

          Bug - A problem which impairs or prevents the functions of the product. SPARK-25438 Fix FilterPushdownBenchmark to use the same memory assumption

          • Major - Major loss of function.
          • Resolved

          Bug - A problem which impairs or prevents the functions of the product. SPARK-26427 Upgrade Apache ORC to 1.5.4

          • Major - Major loss of function.
          • Resolved

          Bug - A problem which impairs or prevents the functions of the product. SPARK-26437 Decimal data becomes bigint to query, unable to query

          • Major - Major loss of function.
          • Resolved

          Bug - A problem which impairs or prevents the functions of the product. SPARK-14286 Empty ORC table join throws exception

          • Minor - Minor loss of function, or other problem where easy workaround is present.
          • Resolved

          Bug - A problem which impairs or prevents the functions of the product. SPARK-22280 Improve StatisticsSuite to test `convertMetastore` properly

          • Minor - Minor loss of function, or other problem where easy workaround is present.
          • Resolved

          Bug - A problem which impairs or prevents the functions of the product. SPARK-22320 ORC should support VectorUDT/MatrixUDT

          • Minor - Minor loss of function, or other problem where easy workaround is present.
          • Resolved

          Bug - A problem which impairs or prevents the functions of the product. SPARK-25145 Buffer size too small on spark.sql query with filterPushdown predicate=True

          • Minor - Minor loss of function, or other problem where easy workaround is present.
          • Resolved

          Bug - A problem which impairs or prevents the functions of the product. SPARK-21791 ORC should support column names with dot

          • Major - Major loss of function.
          • Closed

          New Feature - A new feature of the product, which has yet to be developed. SPARK-11412 Support merge schema for ORC

          • Major - Major loss of function.
          • Resolved

          New Feature - A new feature of the product, which has yet to be developed. SPARK-16060 Vectorized ORC reader

          • Major - Major loss of function.
          • Resolved

          New Feature - A new feature of the product, which has yet to be developed. SPARK-22781 Support creating streaming dataset with ORC files

          • Major - Major loss of function.
          • Resolved

          Improvement - An improvement or enhancement to an existing feature or task. SPARK-18540 Wholestage code-gen for ORC Hive tables

          • Major - Major loss of function.
          • Resolved

          Improvement - An improvement or enhancement to an existing feature or task. SPARK-20682 Add new ORCFileFormat based on Apache ORC

          • Major - Major loss of function.
          • Resolved

          Improvement - An improvement or enhancement to an existing feature or task. SPARK-20728 Make ORCFileFormat configurable between sql/hive and sql/core

          • Major - Major loss of function.
          • Resolved

          Improvement - An improvement or enhancement to an existing feature or task. SPARK-21787 Support for pushing down filters for DateType in native OrcFileFormat

          • Major - Major loss of function.
          • Resolved

          Improvement - An improvement or enhancement to an existing feature or task. SPARK-21839 Support SQL config for ORC compression

          • Major - Major loss of function.
          • Resolved

          Improvement - An improvement or enhancement to an existing feature or task. SPARK-23456 Turn on `native` ORC implementation by default

          • Major - Major loss of function.
          • Resolved

          Improvement - An improvement or enhancement to an existing feature or task. SPARK-24576 Upgrade Apache ORC to 1.5.2

          • Major - Major loss of function.
          • Resolved

          Improvement - An improvement or enhancement to an existing feature or task. SPARK-34562 Leverage parquet bloom filters

          • Major - Major loss of function.
          • Resolved

          Improvement - An improvement or enhancement to an existing feature or task. SPARK-12417 Orc bloom filter options are not propagated during file write in spark

          • Minor - Minor loss of function, or other problem where easy workaround is present.
          • Resolved

          Improvement - An improvement or enhancement to an existing feature or task. SPARK-21783 Turn on ORC filter push-down by default

          • Minor - Minor loss of function, or other problem where easy workaround is present.
          • Resolved

          Improvement - An improvement or enhancement to an existing feature or task. SPARK-23276 Enable UDT tests in (Hive)OrcHadoopFsRelationSuite

          • Minor - Minor loss of function, or other problem where easy workaround is present.
          • Resolved

          Improvement - An improvement or enhancement to an existing feature or task. SPARK-23305 Test `spark.sql.files.ignoreMissingFiles` for all file-based data sources

          • Minor - Minor loss of function, or other problem where easy workaround is present.
          • Resolved

          Improvement - An improvement or enhancement to an existing feature or task. SPARK-23452 Extend test coverage to all ORC readers

          • Minor - Minor loss of function, or other problem where easy workaround is present.
          • Resolved

          Improvement - An improvement or enhancement to an existing feature or task. SPARK-24112 Add `spark.sql.hive.convertMetastoreTableProperty` for backward compatiblility

          • Minor - Minor loss of function, or other problem where easy workaround is present.
          • Closed

          Test - A new unit, integration or system test. SPARK-23072 Add a Unicode schema test for file-based data sources

          • Minor - Minor loss of function, or other problem where easy workaround is present.
          • Resolved

          Test - A new unit, integration or system test. SPARK-23342 Add ORC configuration tests for ORC data source

          • Minor - Minor loss of function, or other problem where easy workaround is present.
          • Resolved

          Task - A task that needs to be done. SPARK-23426 Use `hive` ORC impl and disable PPD for Spark 2.3.0

          • Blocker - Blocks development and/or testing work, production could not run
          • Resolved

          Task - A task that needs to be done. SPARK-22672 Refactor ORC Tests

          • Major - Major loss of function.
          • Resolved

          Task - A task that needs to be done. SPARK-22416 Move OrcOptions from `sql/hive` to `sql/core`

          • Minor - Minor loss of function, or other problem where easy workaround is present.
          • Resolved

          Documentation - Documentation or Website SPARK-23313 Add a migration guide for ORC

          • Major - Major loss of function.
          • Resolved
          is related to

          Bug - A problem which impairs or prevents the functions of the product. SPARK-19459 ORC tables cannot be read when they contain char/varchar columns

          • Major - Major loss of function.
          • Resolved

          Bug - A problem which impairs or prevents the functions of the product. ORC-233 Allow `orc.include.columns` to be empty

          • Major - Major loss of function.
          • Closed
          relates to

          Bug - A problem which impairs or prevents the functions of the product. HIVE-14007 Replace ORC module with ORC release

          • Major - Major loss of function.
          • Resolved

          Activity

            People

              Unassigned Unassigned
              dongjoon Dongjoon Hyun
              Votes:
              0 Vote for this issue
              Watchers:
              24 Start watching this issue

              Dates

                Created:
                Updated: