Uploaded image for project: 'Apache HAWQ (Retired)'
  1. Apache HAWQ (Retired)
  2. HAWQ-864

Support ORC as a native file format

    XMLWordPrintableJSON

Details

    • New Feature
    • Status: Closed
    • Major
    • Resolution: Fixed
    • 2.4.0.0
    • backlog
    • Storage
    • None

    Description

      ORC (Optimized Row Columnar) is a very popular open source format adopted in some major
      components in Hadoop eco­system. It is also used by a lot of users. The advantages of
      supporting ORC storage in HAWQ are in two folds: firstly, it makes HAWQ more Hadoop native
      which interacts with other components more easily; secondly, ORC stores some meta info for
      query optimization, thus, it might potentially outperform two native formats (i.e., AO, Parquet) if it
      is available.

      The implementation can be based on the framework proposed in HAWQ-786.

      Attachments

        Issue Links

          Blocked

          Sub-task - The sub-task of the issue HAWQ-1693 Build dbcommon,univplan,storage in hawq

          • Minor - Minor loss of function, or other problem where easy workaround is present.
          • Closed
          is related to

          Wish - General wishlist item. HAWQ-786 Framework to support pluggable formats and file systems

          • Major - Major loss of function.
          • Closed
          links to

          Web Link GitHub Pull Request #1423

          1.
          		 [DDL SUPPORT]CREATE TABLE FORMAT 'ORC' Sub-task Closed Ruilong Huo Actions
          2.
          		 [DDL SUPPORT]DROP ORC TABLE Sub-task Closed Ruilong Huo Actions
          3.
          		 [DML | SCAN] ORC READER INTERFACE Sub-task Closed Ruilong Huo Actions
          4.
          		 [DML|SCAN] ORC FORMAT READER IMPLEMENTATION Sub-task Closed Ruilong Huo Actions
          5.
          		 [DML|INSERT] ORC WRITER INTERFACE Sub-task Closed Ruilong Huo Actions
          6.
          		 [DML|INSERT] ORC WRITER IMPLEMENTATION Sub-task Closed Ruilong Huo Actions
          7.
          		 [DDL|MISC] ADD OPTION VALIDATE FOR ORC Sub-task Closed Ruilong Huo Actions
          8.
          		 [DDL|MISC] ADD DATATYPE CHECK Sub-task Closed Ruilong Huo Actions
          9.
          		 [DDL|MISC] ADD ENCODING CHECK FOR ORC Sub-task Closed Ruilong Huo Actions
          10.
          		 [MISC] ADD COMPILE SUPPORT FOR ORC FORMATTER AND WRAPPER Sub-task Closed oushu1longziyang1 Actions
          11.
          		 Disable ORC compilation until it is available Sub-task Closed Ruilong Huo Actions
          12.
          		 Missing header file in libyarn client Sub-task Closed Ruilong Huo Actions
          13.
          		 Typo in orc format and unnecessary error for hdfsprotocol_validate Sub-task Closed Ruilong Huo Actions
          14.
          		 Add common utilities used by orc format Sub-task Closed Ruilong Huo Actions
          15.
          		 Add universal plan used by orc format Sub-task Closed Ruilong Huo Actions
          16.
          		 Add orc format in apache hawq Sub-task Closed Ruilong Huo Actions
          17.
          		 Build dbcommon,univplan,storage in hawq Sub-task Closed Ruilong Huo Actions
          18.
          		 enable dbcommon,univplan,storage ut Sub-task Closed Ruilong Huo Actions
          19.
          		 Optimize hawq compilation structure Sub-task Closed Radar Da Lei Actions
          20.
          		 fix univplan cannot be build in linux bug Sub-task Closed Ruilong Huo Actions
          21.
          		 modify json's path in hawq build Sub-task Closed Ruilong Huo Actions
          22.
          		 fix hawq cannot be build in linux bug Sub-task Closed Ruilong Huo Actions
          23.
          		 enable feature test in apache hawq Sub-task Closed Jiali Yao Actions
          24.
          		 modify .ans file in hawq feature-test Sub-task Closed Jiali Yao Actions
          25.
          		 delete legacy orc in hawq Sub-task Closed Ruilong Huo Actions
          26.
          		 Add ORC protocol validators in hawq Sub-task Closed Radar Da Lei Actions
          27.
          		 fix orc build bug Sub-task Closed Ruilong Huo Actions
          28.
          		 enable orc build in makefile Sub-task Closed Ruilong Huo Actions
          29.
          		 install dbcommon,univplan,storage once they are made Sub-task Closed Ruilong Huo Actions
          30.
          		 Add ORC writer in hawq Sub-task Closed Ruilong Huo Actions
          31.
          		 Add ORC reader interface in hawq Sub-task Closed Ruilong Huo Actions
          32.
          		 Add ORC reader implement in hawq Sub-task Closed Ruilong Huo Actions
          33.
          		 Add feature test for ORC format Sub-task Closed Jiali Yao Actions
          34.
          		 enable apache hawq build in docker Sub-task Closed Ruilong Huo Actions
          35.
          		 Disable TestMbConverter.Canonical in hawq Sub-task Closed Radar Da Lei Actions
          36.
          		 bump hawq version to 2.5 in pom.xml for Apache HAWQ 2.5.0 Sub-task Closed Radar Da Lei Actions
          37.
          		 bump hawq version to 2.5 in contrib/hawq-ambari-plugin/build.properties for Apache HAWQ 2.5.0 Sub-task Closed Radar Da Lei Actions
          38.
          		 enable feature-test in docker Sub-task Closed Jiali Yao Actions
          39.
          		 Disable agg function unittest in dbcommon Sub-task Closed Radar Da Lei Actions
          40.
          		 Reopen unittest in hawq Sub-task Closed Radar Da Lei Actions
          41.
          		 support create table in orc format Sub-task Closed Ruilong Huo Actions
          42.
          		 add tools into thirdyparty Sub-task Closed Ruilong Huo Actions
          43.
          		 install cogapp in Dockerfile Sub-task Closed Ruilong Huo Actions
          44.
          		 install some necessary software in Dockerfile Sub-task Closed Radar Da Lei Actions
          45.
          		 fix dependency path hard code Sub-task Closed Radar Da Lei Actions
          46.
          		 fix orc table create information bug Sub-task Closed Radar Da Lei Actions
          47.
          		 Make writable table can read Sub-task Closed Ruilong Huo Actions
          48.
          		 Resolve select issue in external table of orc Sub-task Closed Ruilong Huo Actions
          49.
          		 Resolve insert issue in external table of orc Sub-task Closed Ruilong Huo Actions
          50.
          		 enable orc feature test Sub-task Closed Jiali Yao Actions

          Activity

            People

              huor Ruilong Huo
              lei_chang Lei Chang
              Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Slack

                  Issue deployment