Uploaded image for project: 'Tajo'
  1. Tajo
  2. TAJO-714

Enable setting Parquet tuning parameters

    Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.8.0, 0.9.0
    • Component/s: None
    • Labels:
      None

      Description

      The first version of Parquet support does not support setting Parquet's tuning configuration parameters, such as compression, row group and page size, dictionary encoding, etc.

      1. TAJO-714_2.patch
        12 kB
        Hyunsik Choi
      2. TAJO-714_20140331_19:21:16.patch
        4 kB
        David Chen
      3. TAJO-714_20140331_21:05:42.patch
        13 kB
        David Chen
      4. TAJO-714.patch
        3 kB
        David Chen

        Issue Links

          Activity

          Hide
          davidzchen David Chen added a comment -

          Created a review request against branch master in reviewboard

          Show
          davidzchen David Chen added a comment - Created a review request against branch master in reviewboard
          Hide
          davidzchen David Chen added a comment -

          I have modified ParquetAppender to allow these settings to be set using the with clause in create table statements. The RB for this is here: https://reviews.apache.org/r/19786/

          I have tested this using the following query:

          create table table1_parquet (
              id int,
              name text,
              score float,
              type text)
            using parquet with (
              'parquet.page.size'='65536',
              'parquet.block.size'='536870912',
              'parquet.compression'='snappy')
            as select * from table1;
          

          I have a few questions:

          • The code to set default parameter values is a bit complex because TableMeta.getOption() only support String default values. I can overload this method to support int and boolean default values, but I noticed that this method returns null if p.hasParams() is false. Should this method still return the default value in this case regardless?
          • Enabling compression requires the compression codec jars to be in the classpath. Do you think we should add gzip, lzo, and snappy to the dependencies in the pom.xml or should we leave it up to users to install those jars?
          Show
          davidzchen David Chen added a comment - I have modified ParquetAppender to allow these settings to be set using the with clause in create table statements. The RB for this is here: https://reviews.apache.org/r/19786/ I have tested this using the following query: create table table1_parquet ( id int , name text, score float , type text) using parquet with ( 'parquet.page.size'='65536', 'parquet.block.size'='536870912', 'parquet.compression'='snappy') as select * from table1; I have a few questions: The code to set default parameter values is a bit complex because TableMeta.getOption() only support String default values. I can overload this method to support int and boolean default values, but I noticed that this method returns null if p.hasParams() is false. Should this method still return the default value in this case regardless? Enabling compression requires the compression codec jars to be in the classpath. Do you think we should add gzip, lzo, and snappy to the dependencies in the pom.xml or should we leave it up to users to install those jars?
          Hide
          tajoqa Tajo QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12637441/TAJO-714.patch
          against master revision 36007d7.

          +1 @author. The patch does not contain any @author tags.

          -1 tests included. The patch doesn't appear to include any new or modified tests.
          Please justify why no new tests are needed for this patch.
          Also please list what manual steps were performed to verify this patch.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javadoc. The applied patch does not increase the total number of javadoc warnings.

          +1 checkstyle. The patch generated 0 code style errors.

          +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed unit tests in tajo-storage.

          Test results: https://builds.apache.org/job/PreCommit-TAJO-Build/282//testReport/
          Console output: https://builds.apache.org/job/PreCommit-TAJO-Build/282//console

          This message is automatically generated.

          Show
          tajoqa Tajo QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12637441/TAJO-714.patch against master revision 36007d7. +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 javadoc. The applied patch does not increase the total number of javadoc warnings. +1 checkstyle. The patch generated 0 code style errors. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed unit tests in tajo-storage. Test results: https://builds.apache.org/job/PreCommit-TAJO-Build/282//testReport/ Console output: https://builds.apache.org/job/PreCommit-TAJO-Build/282//console This message is automatically generated.
          Hide
          hyunsik Hyunsik Choi added a comment -

          Hi David,

          I took a look at your patch. Your work looks nice to me. I leaved in-line comments on your questions.

          > The code to set default parameter values is a bit complex because TableMeta.getOption() only support String default values. I can overload this method to support int and boolean default values, but I noticed that this method returns null if p.hasParams() is false. Should this method still return the default value in this case regardless?

          Above all, thank you for pointing the limitation of TableMeta.getOption(). I'll think of its improvement. Also, getOption() needs to be changed to always return an empty instance even if hasParams() is false. This work seems out of scope of this work. So, I'll create and resolve them in additional issues.

          I agree that ParquetAppender should take default values, and the written parquet files should return default values even if with clause is not give. I think that the 1323 line in LogicalPlanner would be helpful for your consideration. It enables to set some default options values when a table is created.

              // Set default options to be created.
              Options options = CatalogUtil.newOptionsWithDefault(createTableNode.getStorageType());
              if (expr.hasParams()) {
                options.putAll(expr.getParams());
              }
          

          > Enabling compression requires the compression codec jars to be in the classpath. Do you think we should add gzip, lzo, and snappy to the dependencies in the pom.xml or should we leave it up to users to install those jars?

          Actually, I don't fully understand Parquet's compression mechanism. If Parquet's compression uses Hadoop's compression codec, Parquet compression will follow users' Hadoop setting because Tajo's startup script includes Hadoop's classpaths. Otherwise, we need to include the dependencies of compression codec jars.

          Warm regards,
          Hyunsik Choi

          Show
          hyunsik Hyunsik Choi added a comment - Hi David, I took a look at your patch. Your work looks nice to me. I leaved in-line comments on your questions. > The code to set default parameter values is a bit complex because TableMeta.getOption() only support String default values. I can overload this method to support int and boolean default values, but I noticed that this method returns null if p.hasParams() is false. Should this method still return the default value in this case regardless? Above all, thank you for pointing the limitation of TableMeta.getOption() . I'll think of its improvement. Also, getOption() needs to be changed to always return an empty instance even if hasParams() is false. This work seems out of scope of this work. So, I'll create and resolve them in additional issues. I agree that ParquetAppender should take default values, and the written parquet files should return default values even if with clause is not give. I think that the 1323 line in LogicalPlanner would be helpful for your consideration. It enables to set some default options values when a table is created. // Set default options to be created. Options options = CatalogUtil.newOptionsWithDefault(createTableNode.getStorageType()); if (expr.hasParams()) { options.putAll(expr.getParams()); } > Enabling compression requires the compression codec jars to be in the classpath. Do you think we should add gzip, lzo, and snappy to the dependencies in the pom.xml or should we leave it up to users to install those jars? Actually, I don't fully understand Parquet's compression mechanism. If Parquet's compression uses Hadoop's compression codec, Parquet compression will follow users' Hadoop setting because Tajo's startup script includes Hadoop's classpaths. Otherwise, we need to include the dependencies of compression codec jars. Warm regards, Hyunsik Choi
          Hide
          davidzchen David Chen added a comment -

          Updated the review request against branch master in reviewboard
          https://reviews.apache.org/r/19786/

          Show
          davidzchen David Chen added a comment - Updated the review request against branch master in reviewboard https://reviews.apache.org/r/19786/
          Hide
          davidzchen David Chen added a comment -

          Hi Hyunsik,

          Thanks for your helpful feedback!

          I think it would be fine as long as the compression codec jars are in the classpath, whether in the Hadoop or Tajo classpaths. Enabling compression is more on a per-file basis, which is set using the parquet.compression property, which the user would have to set explicitly using the with clause.

          Parquet has default values for each of its configuration options, which it uses if the configuration options are not set by the user. The code in ParquetAppender.init() simply checks which properties are set in the table metadata by the user and otherwise defaults to Parquet's own default values before passing them to the TajoParquetWriter constructor.

          Thanks, I can go ahead and add StoreType.PARQUET to CatalogUtil.newOptionsWithDefaults(). That should make the code in ParquetAppender.init() a little cleaner. In the future, when we decide to make storage types more pluggable, I think it might be a good idea to refactor the code for setting default options and move them code specific to that storage type. I will post another revision after I add Parquet's options to CatalogUtil.newOptionsWithDefaults().

          In the meantime, I have updated the patch with one small change which was to add missing calls to super.init() for ParquetAppender and ParquetScanner.

          Thanks,
          David

          Show
          davidzchen David Chen added a comment - Hi Hyunsik, Thanks for your helpful feedback! I think it would be fine as long as the compression codec jars are in the classpath, whether in the Hadoop or Tajo classpaths. Enabling compression is more on a per-file basis, which is set using the parquet.compression property, which the user would have to set explicitly using the with clause. Parquet has default values for each of its configuration options, which it uses if the configuration options are not set by the user. The code in ParquetAppender.init() simply checks which properties are set in the table metadata by the user and otherwise defaults to Parquet's own default values before passing them to the TajoParquetWriter constructor. Thanks, I can go ahead and add StoreType.PARQUET to CatalogUtil.newOptionsWithDefaults() . That should make the code in ParquetAppender.init() a little cleaner. In the future, when we decide to make storage types more pluggable, I think it might be a good idea to refactor the code for setting default options and move them code specific to that storage type. I will post another revision after I add Parquet's options to CatalogUtil.newOptionsWithDefaults() . In the meantime, I have updated the patch with one small change which was to add missing calls to super.init() for ParquetAppender and ParquetScanner . Thanks, David
          Hide
          tajoqa Tajo QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12637983/TAJO-714_20140331_19%3A21%3A16.patch
          against master revision bd418a5.

          +1 @author. The patch does not contain any @author tags.

          -1 tests included. The patch doesn't appear to include any new or modified tests.
          Please justify why no new tests are needed for this patch.
          Also please list what manual steps were performed to verify this patch.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javadoc. The applied patch does not increase the total number of javadoc warnings.

          +1 checkstyle. The patch generated 0 code style errors.

          +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed unit tests in tajo-storage.

          Test results: https://builds.apache.org/job/PreCommit-TAJO-Build/293//testReport/
          Console output: https://builds.apache.org/job/PreCommit-TAJO-Build/293//console

          This message is automatically generated.

          Show
          tajoqa Tajo QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12637983/TAJO-714_20140331_19%3A21%3A16.patch against master revision bd418a5. +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 javadoc. The applied patch does not increase the total number of javadoc warnings. +1 checkstyle. The patch generated 0 code style errors. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed unit tests in tajo-storage. Test results: https://builds.apache.org/job/PreCommit-TAJO-Build/293//testReport/ Console output: https://builds.apache.org/job/PreCommit-TAJO-Build/293//console This message is automatically generated.
          Hide
          davidzchen David Chen added a comment -

          Updated the review request against branch master in reviewboard
          https://reviews.apache.org/r/19786/

          Show
          davidzchen David Chen added a comment - Updated the review request against branch master in reviewboard https://reviews.apache.org/r/19786/
          Hide
          davidzchen David Chen added a comment -

          I have added the Parquet defaults to CatalogUtil.newOptionsWithDefaults() and updated the tests accordingly. This makes ParquetAppender.init much cleaner but this does add a dependency of tajo-catalog-common on Parquet. While this isn't ideal, I think it may be fine for now since parquet-catalog-common contains a fair amount of code specific for some of the storage types, which we may want to refactor into tajo-storage in the future.

          Show
          davidzchen David Chen added a comment - I have added the Parquet defaults to CatalogUtil.newOptionsWithDefaults() and updated the tests accordingly. This makes ParquetAppender.init much cleaner but this does add a dependency of tajo-catalog-common on Parquet. While this isn't ideal, I think it may be fine for now since parquet-catalog-common contains a fair amount of code specific for some of the storage types, which we may want to refactor into tajo-storage in the future.
          Hide
          tajoqa Tajo QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12637997/TAJO-714_20140331_21%3A05%3A42.patch
          against master revision bd418a5.

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 3 new or modified test files.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javadoc. The applied patch does not increase the total number of javadoc warnings.

          +1 checkstyle. The patch generated 0 code style errors.

          -1 findbugs. The patch appears to introduce 2 new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed unit tests in tajo-catalog/tajo-catalog-common tajo-storage.

          Test results: https://builds.apache.org/job/PreCommit-TAJO-Build/294//testReport/
          Findbugs warnings: https://builds.apache.org/job/PreCommit-TAJO-Build/294//artifact/incubator-tajo/patchprocess/newPatchFindbugsWarningstajo-catalog-common.html
          Console output: https://builds.apache.org/job/PreCommit-TAJO-Build/294//console

          This message is automatically generated.

          Show
          tajoqa Tajo QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12637997/TAJO-714_20140331_21%3A05%3A42.patch against master revision bd418a5. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 3 new or modified test files. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 javadoc. The applied patch does not increase the total number of javadoc warnings. +1 checkstyle. The patch generated 0 code style errors. -1 findbugs. The patch appears to introduce 2 new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed unit tests in tajo-catalog/tajo-catalog-common tajo-storage. Test results: https://builds.apache.org/job/PreCommit-TAJO-Build/294//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-TAJO-Build/294//artifact/incubator-tajo/patchprocess/newPatchFindbugsWarningstajo-catalog-common.html Console output: https://builds.apache.org/job/PreCommit-TAJO-Build/294//console This message is automatically generated.
          Hide
          hyunsik Hyunsik Choi added a comment -

          There was a trivial conflict due to TAJO-616, which was recently committed. So, I rebased the latest patch against the latest master revision.

          Show
          hyunsik Hyunsik Choi added a comment - There was a trivial conflict due to TAJO-616 , which was recently committed. So, I rebased the latest patch against the latest master revision.
          Hide
          hyunsik Hyunsik Choi added a comment -

          +1 for your latest patch.

          I reviewed the patch. The patch is straightforward and works well.

          The below is my test results.

          'With SNAPPY compression'
          tpch> create table orders_parquet8 using parquet with ('parquet.compression' = 'SNAPPY') as select * from orders;
          Progress: 0%, response time: 0.392 sec
          Progress: 0%, response time: 1.195 sec
          Progress: 66%, response time: 2.198 sec
          Progress: 100%, response time: 2.798 sec
          final state: QUERY_SUCCEEDED, response time: 2.798 sec
          OK
          
          tpch> \d orders_parquet6
          
          table name: tpch.orders_parquet6
          table path: hdfs://127.0.0.1:8020/tajo/warehouse/tpch/orders_parquet6
          store type: PARQUET
          number of rows: 1500000
          volume: 56.1 MB
          Options: 
          	'parquet.enable.dictionary'='true'
          	'parquet.compression'='SNAPPY'
          	'parquet.validation'='false'
          	'parquet.page.size'='1048576'
          	'parquet.block.size'='134217728'
          
          schema: 
          o_orderkey	INT8
          o_custkey	INT8
          o_orderstatus	TEXT
          o_totalprice	FLOAT8
          o_orderdate	TEXT
          o_orderpriority	TEXT
          o_clerk	TEXT
          o_shippriority	INT4
          o_comment	TEXT
          
          'default setting'
          tpch> create table orders_parquet7 using parquet as select * from orders;
          Progress: 0%, response time: 0.394 sec
          Progress: 0%, response time: 1.196 sec
          Progress: 66%, response time: 2.199 sec
          Progress: 100%, response time: 2.683 sec
          final state: QUERY_SUCCEEDED, response time: 2.683 sec
          OK
          tpch> \d orders_parquet7
          
          table name: tpch.orders_parquet7
          table path: hdfs://127.0.0.1:8020/tajo/warehouse/tpch/orders_parquet7
          store type: PARQUET
          number of rows: 1500000
          volume: 115.3 MB
          Options: 
          	'parquet.enable.dictionary'='true'
          	'parquet.compression'='uncompressed'
          	'parquet.validation'='false'
          	'parquet.page.size'='1048576'
          	'parquet.block.size'='134217728'
          
          schema: 
          o_orderkey	INT8
          o_custkey	INT8
          o_orderstatus	TEXT
          o_totalprice	FLOAT8
          o_orderdate	TEXT
          o_orderpriority	TEXT
          o_clerk	TEXT
          o_shippriority	INT4
          o_comment	TEXT
          
          Show
          hyunsik Hyunsik Choi added a comment - +1 for your latest patch. I reviewed the patch. The patch is straightforward and works well. The below is my test results. 'With SNAPPY compression' tpch> create table orders_parquet8 using parquet with ('parquet.compression' = 'SNAPPY') as select * from orders; Progress: 0%, response time: 0.392 sec Progress: 0%, response time: 1.195 sec Progress: 66%, response time: 2.198 sec Progress: 100%, response time: 2.798 sec final state: QUERY_SUCCEEDED, response time: 2.798 sec OK tpch> \d orders_parquet6 table name: tpch.orders_parquet6 table path: hdfs://127.0.0.1:8020/tajo/warehouse/tpch/orders_parquet6 store type: PARQUET number of rows: 1500000 volume: 56.1 MB Options: 'parquet.enable.dictionary'='true' 'parquet.compression'='SNAPPY' 'parquet.validation'='false' 'parquet.page.size'='1048576' 'parquet.block.size'='134217728' schema: o_orderkey INT8 o_custkey INT8 o_orderstatus TEXT o_totalprice FLOAT8 o_orderdate TEXT o_orderpriority TEXT o_clerk TEXT o_shippriority INT4 o_comment TEXT 'default setting' tpch> create table orders_parquet7 using parquet as select * from orders; Progress: 0%, response time: 0.394 sec Progress: 0%, response time: 1.196 sec Progress: 66%, response time: 2.199 sec Progress: 100%, response time: 2.683 sec final state: QUERY_SUCCEEDED, response time: 2.683 sec OK tpch> \d orders_parquet7 table name: tpch.orders_parquet7 table path: hdfs://127.0.0.1:8020/tajo/warehouse/tpch/orders_parquet7 store type: PARQUET number of rows: 1500000 volume: 115.3 MB Options: 'parquet.enable.dictionary'='true' 'parquet.compression'='uncompressed' 'parquet.validation'='false' 'parquet.page.size'='1048576' 'parquet.block.size'='134217728' schema: o_orderkey INT8 o_custkey INT8 o_orderstatus TEXT o_totalprice FLOAT8 o_orderdate TEXT o_orderpriority TEXT o_clerk TEXT o_shippriority INT4 o_comment TEXT
          Hide
          tajoqa Tajo QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12638182/TAJO-714_2.patch
          against master revision a3a178a.

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 3 new or modified test files.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javadoc. The applied patch does not increase the total number of javadoc warnings.

          +1 checkstyle. The patch generated 0 code style errors.

          -1 findbugs. The patch appears to introduce 2 new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed unit tests in tajo-catalog/tajo-catalog-common tajo-storage.

          Test results: https://builds.apache.org/job/PreCommit-TAJO-Build/302//testReport/
          Findbugs warnings: https://builds.apache.org/job/PreCommit-TAJO-Build/302//artifact/incubator-tajo/patchprocess/newPatchFindbugsWarningstajo-catalog-common.html
          Console output: https://builds.apache.org/job/PreCommit-TAJO-Build/302//console

          This message is automatically generated.

          Show
          tajoqa Tajo QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12638182/TAJO-714_2.patch against master revision a3a178a. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 3 new or modified test files. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 javadoc. The applied patch does not increase the total number of javadoc warnings. +1 checkstyle. The patch generated 0 code style errors. -1 findbugs. The patch appears to introduce 2 new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed unit tests in tajo-catalog/tajo-catalog-common tajo-storage. Test results: https://builds.apache.org/job/PreCommit-TAJO-Build/302//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-TAJO-Build/302//artifact/incubator-tajo/patchprocess/newPatchFindbugsWarningstajo-catalog-common.html Console output: https://builds.apache.org/job/PreCommit-TAJO-Build/302//console This message is automatically generated.
          Hide
          hudson Hudson added a comment -

          SUCCESS: Integrated in Tajo-master-build #153 (See https://builds.apache.org/job/Tajo-master-build/153/)
          TAJO-714: Enable setting Parquet tuning parameters. (David Chen via hyunsik) (hyunsik: rev dc40d849e4f52b93753627ae699e4c161e4ebdf1)

          • tajo-storage/src/test/java/org/apache/tajo/storage/TestStorages.java
          • tajo-storage/src/test/java/org/apache/tajo/storage/v2/TestStorages.java
          • tajo-storage/src/test/java/org/apache/tajo/storage/TestMergeScanner.java
          • tajo-catalog/tajo-catalog-common/pom.xml
          • tajo-catalog/tajo-catalog-common/src/main/java/org/apache/tajo/catalog/CatalogConstants.java
          • pom.xml
          • tajo-storage/src/main/java/org/apache/tajo/storage/parquet/ParquetAppender.java
          • tajo-storage/pom.xml
          • tajo-catalog/tajo-catalog-common/src/main/java/org/apache/tajo/catalog/CatalogUtil.java
          • CHANGES.txt
          • tajo-storage/src/main/java/org/apache/tajo/storage/parquet/ParquetScanner.java
          Show
          hudson Hudson added a comment - SUCCESS: Integrated in Tajo-master-build #153 (See https://builds.apache.org/job/Tajo-master-build/153/ ) TAJO-714 : Enable setting Parquet tuning parameters. (David Chen via hyunsik) (hyunsik: rev dc40d849e4f52b93753627ae699e4c161e4ebdf1) tajo-storage/src/test/java/org/apache/tajo/storage/TestStorages.java tajo-storage/src/test/java/org/apache/tajo/storage/v2/TestStorages.java tajo-storage/src/test/java/org/apache/tajo/storage/TestMergeScanner.java tajo-catalog/tajo-catalog-common/pom.xml tajo-catalog/tajo-catalog-common/src/main/java/org/apache/tajo/catalog/CatalogConstants.java pom.xml tajo-storage/src/main/java/org/apache/tajo/storage/parquet/ParquetAppender.java tajo-storage/pom.xml tajo-catalog/tajo-catalog-common/src/main/java/org/apache/tajo/catalog/CatalogUtil.java CHANGES.txt tajo-storage/src/main/java/org/apache/tajo/storage/parquet/ParquetScanner.java
          Hide
          hyunsik Hyunsik Choi added a comment -

          committed it to master branch and 0.8.0 branch. Thank you for your contribution!

          Show
          hyunsik Hyunsik Choi added a comment - committed it to master branch and 0.8.0 branch. Thank you for your contribution!
          Hide
          davidzchen David Chen added a comment -

          Excellent. Thanks, Hyunsik!

          Show
          davidzchen David Chen added a comment - Excellent. Thanks, Hyunsik!
          Hide
          hudson Hudson added a comment -

          SUCCESS: Integrated in Tajo-0.8.0-build #55 (See https://builds.apache.org/job/Tajo-0.8.0-build/55/)
          TAJO-714: Enable setting Parquet tuning parameters. (David Chen via hyunsik) (hyunsik: rev 5ca3dfc81b796a2a1298ff47f740e8dc3e074956)

          • tajo-catalog/tajo-catalog-common/pom.xml
          • tajo-storage/src/main/java/org/apache/tajo/storage/parquet/ParquetScanner.java
          • tajo-storage/src/main/java/org/apache/tajo/storage/parquet/ParquetAppender.java
          • tajo-storage/src/test/java/org/apache/tajo/storage/TestStorages.java
          • tajo-storage/src/test/java/org/apache/tajo/storage/TestMergeScanner.java
          • tajo-catalog/tajo-catalog-common/src/main/java/org/apache/tajo/catalog/CatalogConstants.java
          • tajo-catalog/tajo-catalog-common/src/main/java/org/apache/tajo/catalog/CatalogUtil.java
          • tajo-storage/src/test/java/org/apache/tajo/storage/v2/TestStorages.java
          • CHANGES.txt
          • pom.xml
          • tajo-storage/pom.xml
          Show
          hudson Hudson added a comment - SUCCESS: Integrated in Tajo-0.8.0-build #55 (See https://builds.apache.org/job/Tajo-0.8.0-build/55/ ) TAJO-714 : Enable setting Parquet tuning parameters. (David Chen via hyunsik) (hyunsik: rev 5ca3dfc81b796a2a1298ff47f740e8dc3e074956) tajo-catalog/tajo-catalog-common/pom.xml tajo-storage/src/main/java/org/apache/tajo/storage/parquet/ParquetScanner.java tajo-storage/src/main/java/org/apache/tajo/storage/parquet/ParquetAppender.java tajo-storage/src/test/java/org/apache/tajo/storage/TestStorages.java tajo-storage/src/test/java/org/apache/tajo/storage/TestMergeScanner.java tajo-catalog/tajo-catalog-common/src/main/java/org/apache/tajo/catalog/CatalogConstants.java tajo-catalog/tajo-catalog-common/src/main/java/org/apache/tajo/catalog/CatalogUtil.java tajo-storage/src/test/java/org/apache/tajo/storage/v2/TestStorages.java CHANGES.txt pom.xml tajo-storage/pom.xml
          Hide
          hyunsik Hyunsik Choi added a comment -

          > While this isn't ideal, I think it may be fine for now since parquet-catalog-common contains a fair amount of code specific for some of the storage types, which we may want to refactor into tajo-storage in the future.

          I'm thinking that I might commit it too earlier without your agreement. I'm very sorry for that. Actually, I missed one thing you suggested. I totally agree with your suggestion. It would be great to move the default setting part into tajo-storage.

          Best regards,
          Hyunsik Choi

          Show
          hyunsik Hyunsik Choi added a comment - > While this isn't ideal, I think it may be fine for now since parquet-catalog-common contains a fair amount of code specific for some of the storage types, which we may want to refactor into tajo-storage in the future. I'm thinking that I might commit it too earlier without your agreement. I'm very sorry for that. Actually, I missed one thing you suggested. I totally agree with your suggestion. It would be great to move the default setting part into tajo-storage. Best regards, Hyunsik Choi
          Hide
          davidzchen David Chen added a comment -

          Not a problem, Hyunsik. My suggestion was mostly for when we work on refactoring the storage code to be more pluggable in the future.

          Thanks for committing the patch!

          Best,
          David

          Show
          davidzchen David Chen added a comment - Not a problem, Hyunsik. My suggestion was mostly for when we work on refactoring the storage code to be more pluggable in the future. Thanks for committing the patch! Best, David

            People

            • Assignee:
              davidzchen David Chen
              Reporter:
              davidzchen David Chen
            • Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development