Uploaded image for project: 'Apache HAWQ (Retired)'
  1. Apache HAWQ (Retired)
  2. HAWQ-864

Support ORC as a native file format

    XMLWordPrintableJSON

Details

    • New Feature
    • Status: Closed
    • Major
    • Resolution: Fixed
    • 2.4.0.0
    • backlog
    • Storage
    • None

    Description

      ORC (Optimized Row Columnar) is a very popular open source format adopted in some major
      components in Hadoop eco­system. It is also used by a lot of users. The advantages of
      supporting ORC storage in HAWQ are in two folds: firstly, it makes HAWQ more Hadoop native
      which interacts with other components more easily; secondly, ORC stores some meta info for
      query optimization, thus, it might potentially outperform two native formats (i.e., AO, Parquet) if it
      is available.

      The implementation can be based on the framework proposed in HAWQ-786.

      Attachments

        Issue Links

          Blocked

          Sub-task - The sub-task of the issue HAWQ-1693 Build dbcommon,univplan,storage in hawq

          • Minor - Minor loss of function, or other problem where easy workaround is present.
          • Closed
          is related to

          Wish - General wishlist item. HAWQ-786 Framework to support pluggable formats and file systems

          • Major - Major loss of function.
          • Closed
          links to

          Web Link GitHub Pull Request #1423

          1.
          		 [DDL SUPPORT]CREATE TABLE FORMAT 'ORC' Sub-task Closed Ruilong Huo
          2.
          		 [DDL SUPPORT]DROP ORC TABLE Sub-task Closed Ruilong Huo
          3.
          		 [DML | SCAN] ORC READER INTERFACE Sub-task Closed Ruilong Huo
          4.
          		 [DML|SCAN] ORC FORMAT READER IMPLEMENTATION Sub-task Closed Ruilong Huo
          5.
          		 [DML|INSERT] ORC WRITER INTERFACE Sub-task Closed Ruilong Huo
          6.
          		 [DML|INSERT] ORC WRITER IMPLEMENTATION Sub-task Closed Ruilong Huo
          7.
          		 [DDL|MISC] ADD OPTION VALIDATE FOR ORC Sub-task Closed Ruilong Huo
          8.
          		 [DDL|MISC] ADD DATATYPE CHECK Sub-task Closed Ruilong Huo
          9.
          		 [DDL|MISC] ADD ENCODING CHECK FOR ORC Sub-task Closed Ruilong Huo
          10.
          		 [MISC] ADD COMPILE SUPPORT FOR ORC FORMATTER AND WRAPPER Sub-task Closed oushu1longziyang1
          11.
          		 Disable ORC compilation until it is available Sub-task Closed Ruilong Huo
          12.
          		 Missing header file in libyarn client Sub-task Closed Ruilong Huo
          13.
          		 Typo in orc format and unnecessary error for hdfsprotocol_validate Sub-task Closed Ruilong Huo
          14.
          		 Add common utilities used by orc format Sub-task Closed Ruilong Huo
          15.
          		 Add universal plan used by orc format Sub-task Closed Ruilong Huo
          16.
          		 Add orc format in apache hawq Sub-task Closed Ruilong Huo
          17.
          		 Build dbcommon,univplan,storage in hawq Sub-task Closed Ruilong Huo
          18.
          		 enable dbcommon,univplan,storage ut Sub-task Closed Ruilong Huo
          19.
          		 Optimize hawq compilation structure Sub-task Closed Radar Da Lei
          20.
          		 fix univplan cannot be build in linux bug Sub-task Closed Ruilong Huo
          21.
          		 modify json's path in hawq build Sub-task Closed Ruilong Huo
          22.
          		 fix hawq cannot be build in linux bug Sub-task Closed Ruilong Huo
          23.
          		 enable feature test in apache hawq Sub-task Closed Jiali Yao
          24.
          		 modify .ans file in hawq feature-test Sub-task Closed Jiali Yao
          25.
          		 delete legacy orc in hawq Sub-task Closed Ruilong Huo
          26.
          		 Add ORC protocol validators in hawq Sub-task Closed Radar Da Lei
          27.
          		 fix orc build bug Sub-task Closed Ruilong Huo
          28.
          		 enable orc build in makefile Sub-task Closed Ruilong Huo
          29.
          		 install dbcommon,univplan,storage once they are made Sub-task Closed Ruilong Huo
          30.
          		 Add ORC writer in hawq Sub-task Closed Ruilong Huo
          31.
          		 Add ORC reader interface in hawq Sub-task Closed Ruilong Huo
          32.
          		 Add ORC reader implement in hawq Sub-task Closed Ruilong Huo
          33.
          		 Add feature test for ORC format Sub-task Closed Jiali Yao
          34.
          		 enable apache hawq build in docker Sub-task Closed Ruilong Huo
          35.
          		 Disable TestMbConverter.Canonical in hawq Sub-task Closed Radar Da Lei
          36.
          		 bump hawq version to 2.5 in pom.xml for Apache HAWQ 2.5.0 Sub-task Closed Radar Da Lei
          37.
          		 bump hawq version to 2.5 in contrib/hawq-ambari-plugin/build.properties for Apache HAWQ 2.5.0 Sub-task Closed Radar Da Lei
          38.
          		 enable feature-test in docker Sub-task Closed Jiali Yao
          39.
          		 Disable agg function unittest in dbcommon Sub-task Closed Radar Da Lei
          40.
          		 Reopen unittest in hawq Sub-task Closed Radar Da Lei
          41.
          		 support create table in orc format Sub-task Closed Ruilong Huo
          42.
          		 add tools into thirdyparty Sub-task Closed Ruilong Huo
          43.
          		 install cogapp in Dockerfile Sub-task Closed Ruilong Huo
          44.
          		 install some necessary software in Dockerfile Sub-task Closed Radar Da Lei
          45.
          		 fix dependency path hard code Sub-task Closed Radar Da Lei
          46.
          		 fix orc table create information bug Sub-task Closed Radar Da Lei
          47.
          		 Make writable table can read Sub-task Closed Ruilong Huo
          48.
          		 Resolve select issue in external table of orc Sub-task Closed Ruilong Huo
          49.
          		 Resolve insert issue in external table of orc Sub-task Closed Ruilong Huo
          50.
          		 enable orc feature test Sub-task Closed Jiali Yao

          Activity

            People

              huor Ruilong Huo
              lei_chang Lei Chang
              Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: