Hive
  1. Hive
  2. HIVE-7858

Parquet compression should be configurable via table property

    Details

    • Type: Improvement Improvement
    • Status: Resolved
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 1.1.0
    • Component/s: None
    • Labels:
    • Release Note:
      The property parquet.compression can now be configured as a table property.

      Description

      ORC supports the orc.compress table property:

      https://cwiki.apache.org/confluence/display/Hive/LanguageManual+ORC

      create table Addresses (
        name string,
        street string,
        city string,
        state string,
        zip int
      ) stored as orc tblproperties ("orc.compress"="NONE");
      

      I think it'd be great to support the same for Parquet.

      1. HIVE-7858.patch
        5 kB
        Ferdinand Xu
      2. HIVE-7858.patch
        5 kB
        Brock Noland
      3. HIVE-7858.1.patch
        6 kB
        Ferdinand Xu

        Issue Links

          Activity

          Hide
          Brock Noland added a comment -

          Hi,

          This change doesn't really have anything to do with internal or external tables. Parquet stores the compression algo used in the footer as such it's possible to have a mix of snappy, gzip, etc compression types within the same table. This change was to allow users to say "when I insert overwrite this table, I want to use compression format x". Before this change they could only use the global setting at a session or hive-site.xml level.

          Show
          Brock Noland added a comment - Hi, This change doesn't really have anything to do with internal or external tables. Parquet stores the compression algo used in the footer as such it's possible to have a mix of snappy, gzip, etc compression types within the same table. This change was to allow users to say "when I insert overwrite this table, I want to use compression format x". Before this change they could only use the global setting at a session or hive-site.xml level.
          Hide
          Alex McLintock added a comment -

          (I'm trying to understand the scope of this bug/bugfix)

          For internal tables within Hive we can only create parquet files with the default compression format - up until Hive version 0.15 when this fix will be generally available.

          Can I ask how this affects external tables?
          Assuming that Hive can read a parquet file supplied to it does it need to be told what compression format was used?

          Show
          Alex McLintock added a comment - (I'm trying to understand the scope of this bug/bugfix) For internal tables within Hive we can only create parquet files with the default compression format - up until Hive version 0.15 when this fix will be generally available. Can I ask how this affects external tables? Assuming that Hive can read a parquet file supplied to it does it need to be told what compression format was used?
          Hide
          Brock Noland added a comment -

          We should doc it.. I put a release note "The property parquet.compression can now be configured as a table property."

          I think we should put a item here: https://cwiki.apache.org/confluence/display/Hive/Parquet

          • Parquet Table Properties *

          with this one. We can update it as we add more..

          Show
          Brock Noland added a comment - We should doc it.. I put a release note "The property parquet.compression can now be configured as a table property." I think we should put a item here: https://cwiki.apache.org/confluence/display/Hive/Parquet Parquet Table Properties * with this one. We can update it as we add more..
          Hide
          Lefty Leverenz added a comment -

          Does this need documentation in 0.15.0 (or should we wait for HIVE-8469)?

          Show
          Lefty Leverenz added a comment - Does this need documentation in 0.15.0 (or should we wait for HIVE-8469 )?
          Hide
          Brock Noland added a comment -

          FYI that I created HIVE-8469 to add this as an annotated serde property post HIVE-8450

          Show
          Brock Noland added a comment - FYI that I created HIVE-8469 to add this as an annotated serde property post HIVE-8450
          Hide
          Brock Noland added a comment -

          Thank you so much! I have committed this to trunk!

          Show
          Brock Noland added a comment - Thank you so much! I have committed this to trunk!
          Hide
          Hive QA added a comment -

          Overall: -1 at least one tests failed

          Here are the results of testing the latest attachment:
          https://issues.apache.org/jira/secure/attachment/12674688/HIVE-7858.1.patch

          ERROR: -1 due to 3 failed/errored test(s), 6558 tests executed
          Failed tests:

          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_parallel
          org.apache.hive.beeline.TestSchemaTool.testSchemaInit
          org.apache.hive.beeline.TestSchemaTool.testSchemaUpgrade
          

          Test results: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/1268/testReport
          Console output: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/1268/console
          Test logs: http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-1268/

          Messages:

          Executing org.apache.hive.ptest.execution.PrepPhase
          Executing org.apache.hive.ptest.execution.ExecutionPhase
          Executing org.apache.hive.ptest.execution.ReportingPhase
          Tests exited with: TestsFailedException: 3 tests failed
          

          This message is automatically generated.

          ATTACHMENT ID: 12674688

          Show
          Hive QA added a comment - Overall : -1 at least one tests failed Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12674688/HIVE-7858.1.patch ERROR: -1 due to 3 failed/errored test(s), 6558 tests executed Failed tests: org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_parallel org.apache.hive.beeline.TestSchemaTool.testSchemaInit org.apache.hive.beeline.TestSchemaTool.testSchemaUpgrade Test results: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/1268/testReport Console output: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/1268/console Test logs: http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-1268/ Messages: Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase Tests exited with: TestsFailedException: 3 tests failed This message is automatically generated. ATTACHMENT ID: 12674688
          Hide
          Ferdinand Xu added a comment -

          Fix for the typo error

          Show
          Ferdinand Xu added a comment - Fix for the typo error
          Hide
          Brock Noland added a comment -

          +1 pending the fix of the spelling error.

          Show
          Brock Noland added a comment - +1 pending the fix of the spelling error.
          Hide
          Lefty Leverenz added a comment -

          TestMapredParquetOutputFormat.java has a typo (missing an s):

          +            Properties tablePropertie
          
          Show
          Lefty Leverenz added a comment - TestMapredParquetOutputFormat.java has a typo (missing an s): + Properties tablePropertie
          Hide
          Hive QA added a comment -

          Overall: -1 at least one tests failed

          Here are the results of testing the latest attachment:
          https://issues.apache.org/jira/secure/attachment/12673991/HIVE-7858.patch

          ERROR: -1 due to 1 failed/errored test(s), 4137 tests executed
          Failed tests:

          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_tez_smb_1
          

          Test results: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/1213/testReport
          Console output: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/1213/console
          Test logs: http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-1213/

          Messages:

          Executing org.apache.hive.ptest.execution.PrepPhase
          Executing org.apache.hive.ptest.execution.ExecutionPhase
          Executing org.apache.hive.ptest.execution.ReportingPhase
          Tests exited with: TestsFailedException: 1 tests failed
          

          This message is automatically generated.

          ATTACHMENT ID: 12673991

          Show
          Hive QA added a comment - Overall : -1 at least one tests failed Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12673991/HIVE-7858.patch ERROR: -1 due to 1 failed/errored test(s), 4137 tests executed Failed tests: org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_tez_smb_1 Test results: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/1213/testReport Console output: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/1213/console Test logs: http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-1213/ Messages: Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase Tests exited with: TestsFailedException: 1 tests failed This message is automatically generated. ATTACHMENT ID: 12673991
          Hide
          Ferdinand Xu added a comment -

          Yes, I verify it on a local cluster. Use the command "mvn exec:java -Dexec.mainClass="parquet.tools.Main" -Dexec.args="dump $

          {file_path}

          " with the the dump tool from https://github.com/apache/incubator-parquet-mr/blob/master/parquet-tools/src/main/java/parquet/tools/Main.java

          Show
          Ferdinand Xu added a comment - Yes, I verify it on a local cluster. Use the command "mvn exec:java -Dexec.mainClass="parquet.tools.Main" -Dexec.args="dump $ {file_path} " with the the dump tool from https://github.com/apache/incubator-parquet-mr/blob/master/parquet-tools/src/main/java/parquet/tools/Main.java
          Hide
          Brock Noland added a comment -

          Ok, could we manually verify this works by creating a compressed parquet file and then use the file dump util to see that's compressed?

          Show
          Brock Noland added a comment - Ok, could we manually verify this works by creating a compressed parquet file and then use the file dump util to see that's compressed?
          Hide
          Ferdinand Xu added a comment -

          Hi Brock Noland, I am afraid not. The Hive side just passes codec type to get a real writer. And compression information is encapsulated in the real writer from parquet side. Adding tests needs some changes on parquet side since the ParquetOutput is not exposing the codec information as publicity.
          https://github.com/apache/incubator-parquet-mr/blob/0148455170be07f89bd6b9230960a6cd510c7ca6/parquet-hadoop/src/main/java/parquet/hadoop/ParquetRecordWriter.java#L64

          Show
          Ferdinand Xu added a comment - Hi Brock Noland , I am afraid not. The Hive side just passes codec type to get a real writer. And compression information is encapsulated in the real writer from parquet side. Adding tests needs some changes on parquet side since the ParquetOutput is not exposing the codec information as publicity. https://github.com/apache/incubator-parquet-mr/blob/0148455170be07f89bd6b9230960a6cd510c7ca6/parquet-hadoop/src/main/java/parquet/hadoop/ParquetRecordWriter.java#L64
          Hide
          Brock Noland added a comment -

          This looks good! Is it possible to add a test for this?

          Show
          Brock Noland added a comment - This looks good! Is it possible to add a test for this?
          Hide
          Brock Noland added a comment -

          Attaching same patch.

          Show
          Brock Noland added a comment - Attaching same patch.
          Hide
          Hive QA added a comment -

          Overall: -1 no tests executed

          Here are the results of testing the latest attachment:
          https://issues.apache.org/jira/secure/attachment/12673513/HIVE-7858.patch

          Test results: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/1174/testReport
          Console output: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/1174/console
          Test logs: http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-1174/

          Messages:

          **** This message was trimmed, see log for full details ****
          warning(200): IdentifiersParser.g:68:4: 
          Decision can match input such as "LPAREN LPAREN KW_CASE" using multiple alternatives: 1, 2
          
          As a result, alternative(s) 2 were disabled for that input
          warning(200): IdentifiersParser.g:68:4: 
          Decision can match input such as "LPAREN KW_NOT KW_FALSE" using multiple alternatives: 1, 2
          
          As a result, alternative(s) 2 were disabled for that input
          warning(200): IdentifiersParser.g:68:4: 
          Decision can match input such as "LPAREN KW_DATE StringLiteral" using multiple alternatives: 1, 2
          
          As a result, alternative(s) 2 were disabled for that input
          warning(200): IdentifiersParser.g:68:4: 
          Decision can match input such as "LPAREN KW_NOT KW_TRUE" using multiple alternatives: 1, 2
          
          As a result, alternative(s) 2 were disabled for that input
          warning(200): IdentifiersParser.g:68:4: 
          Decision can match input such as "LPAREN KW_NOT KW_MAP" using multiple alternatives: 1, 2
          
          As a result, alternative(s) 2 were disabled for that input
          warning(200): IdentifiersParser.g:68:4: 
          Decision can match input such as "LPAREN KW_CASE KW_MAP" using multiple alternatives: 1, 2
          
          As a result, alternative(s) 2 were disabled for that input
          warning(200): IdentifiersParser.g:68:4: 
          Decision can match input such as "LPAREN LPAREN KW_MAP" using multiple alternatives: 1, 2
          
          As a result, alternative(s) 2 were disabled for that input
          warning(200): IdentifiersParser.g:68:4: 
          Decision can match input such as "LPAREN KW_CASE KW_UNIONTYPE" using multiple alternatives: 1, 2
          
          As a result, alternative(s) 2 were disabled for that input
          warning(200): IdentifiersParser.g:68:4: 
          Decision can match input such as "LPAREN KW_CASE KW_STRUCT" using multiple alternatives: 1, 2
          
          As a result, alternative(s) 2 were disabled for that input
          warning(200): IdentifiersParser.g:68:4: 
          Decision can match input such as "LPAREN KW_NOT KW_IF" using multiple alternatives: 1, 2
          
          As a result, alternative(s) 2 were disabled for that input
          warning(200): IdentifiersParser.g:68:4: 
          Decision can match input such as "LPAREN KW_CASE KW_IF" using multiple alternatives: 1, 2
          
          As a result, alternative(s) 2 were disabled for that input
          warning(200): IdentifiersParser.g:68:4: 
          Decision can match input such as "LPAREN LPAREN KW_IF" using multiple alternatives: 1, 2
          
          As a result, alternative(s) 2 were disabled for that input
          warning(200): IdentifiersParser.g:115:5: 
          Decision can match input such as "KW_CLUSTER KW_BY LPAREN" using multiple alternatives: 1, 2
          
          As a result, alternative(s) 2 were disabled for that input
          warning(200): IdentifiersParser.g:127:5: 
          Decision can match input such as "KW_PARTITION KW_BY LPAREN" using multiple alternatives: 1, 2
          
          As a result, alternative(s) 2 were disabled for that input
          warning(200): IdentifiersParser.g:138:5: 
          Decision can match input such as "KW_DISTRIBUTE KW_BY LPAREN" using multiple alternatives: 1, 2
          
          As a result, alternative(s) 2 were disabled for that input
          warning(200): IdentifiersParser.g:149:5: 
          Decision can match input such as "KW_SORT KW_BY LPAREN" using multiple alternatives: 1, 2
          
          As a result, alternative(s) 2 were disabled for that input
          warning(200): IdentifiersParser.g:166:7: 
          Decision can match input such as "STAR" using multiple alternatives: 1, 2
          
          As a result, alternative(s) 2 were disabled for that input
          warning(200): IdentifiersParser.g:179:5: 
          Decision can match input such as "KW_STRUCT" using multiple alternatives: 4, 6
          
          As a result, alternative(s) 6 were disabled for that input
          warning(200): IdentifiersParser.g:179:5: 
          Decision can match input such as "KW_ARRAY" using multiple alternatives: 2, 6
          
          As a result, alternative(s) 6 were disabled for that input
          warning(200): IdentifiersParser.g:179:5: 
          Decision can match input such as "KW_UNIONTYPE" using multiple alternatives: 5, 6
          
          As a result, alternative(s) 6 were disabled for that input
          warning(200): IdentifiersParser.g:261:5: 
          Decision can match input such as "KW_TRUE" using multiple alternatives: 3, 8
          
          As a result, alternative(s) 8 were disabled for that input
          warning(200): IdentifiersParser.g:261:5: 
          Decision can match input such as "KW_NULL" using multiple alternatives: 1, 8
          
          As a result, alternative(s) 8 were disabled for that input
          warning(200): IdentifiersParser.g:261:5: 
          Decision can match input such as "KW_FALSE" using multiple alternatives: 3, 8
          
          As a result, alternative(s) 8 were disabled for that input
          warning(200): IdentifiersParser.g:261:5: 
          Decision can match input such as "KW_DATE StringLiteral" using multiple alternatives: 2, 3
          
          As a result, alternative(s) 3 were disabled for that input
          warning(200): IdentifiersParser.g:393:5: 
          Decision can match input such as "KW_BETWEEN KW_MAP LPAREN" using multiple alternatives: 8, 9
          
          As a result, alternative(s) 9 were disabled for that input
          warning(200): IdentifiersParser.g:393:5: 
          Decision can match input such as "{KW_LIKE, KW_REGEXP, KW_RLIKE} KW_INSERT KW_INTO" using multiple alternatives: 2, 9
          
          As a result, alternative(s) 9 were disabled for that input
          warning(200): IdentifiersParser.g:393:5: 
          Decision can match input such as "{KW_LIKE, KW_REGEXP, KW_RLIKE} KW_UNION KW_ALL" using multiple alternatives: 2, 9
          
          As a result, alternative(s) 9 were disabled for that input
          warning(200): IdentifiersParser.g:393:5: 
          Decision can match input such as "{KW_LIKE, KW_REGEXP, KW_RLIKE} KW_SORT KW_BY" using multiple alternatives: 2, 9
          
          As a result, alternative(s) 9 were disabled for that input
          warning(200): IdentifiersParser.g:393:5: 
          Decision can match input such as "{KW_LIKE, KW_REGEXP, KW_RLIKE} KW_CLUSTER KW_BY" using multiple alternatives: 2, 9
          
          As a result, alternative(s) 9 were disabled for that input
          warning(200): IdentifiersParser.g:393:5: 
          Decision can match input such as "{KW_LIKE, KW_REGEXP, KW_RLIKE} KW_LATERAL KW_VIEW" using multiple alternatives: 2, 9
          
          As a result, alternative(s) 9 were disabled for that input
          warning(200): IdentifiersParser.g:393:5: 
          Decision can match input such as "{KW_LIKE, KW_REGEXP, KW_RLIKE} KW_MAP LPAREN" using multiple alternatives: 2, 9
          
          As a result, alternative(s) 9 were disabled for that input
          warning(200): IdentifiersParser.g:393:5: 
          Decision can match input such as "{KW_LIKE, KW_REGEXP, KW_RLIKE} KW_DISTRIBUTE KW_BY" using multiple alternatives: 2, 9
          
          As a result, alternative(s) 9 were disabled for that input
          warning(200): IdentifiersParser.g:393:5: 
          Decision can match input such as "{KW_LIKE, KW_REGEXP, KW_RLIKE} KW_ORDER KW_BY" using multiple alternatives: 2, 9
          
          As a result, alternative(s) 9 were disabled for that input
          warning(200): IdentifiersParser.g:393:5: 
          Decision can match input such as "{KW_LIKE, KW_REGEXP, KW_RLIKE} KW_GROUP KW_BY" using multiple alternatives: 2, 9
          
          As a result, alternative(s) 9 were disabled for that input
          warning(200): IdentifiersParser.g:393:5: 
          Decision can match input such as "{KW_LIKE, KW_REGEXP, KW_RLIKE} KW_INSERT KW_OVERWRITE" using multiple alternatives: 2, 9
          
          As a result, alternative(s) 9 were disabled for that input
          warning(200): IdentifiersParser.g:518:5: 
          Decision can match input such as "{AMPERSAND..BITWISEXOR, DIV..DIVIDE, EQUAL..EQUAL_NS, GREATERTHAN..GREATERTHANOREQUALTO, KW_AND, KW_ARRAY, KW_BETWEEN..KW_BOOLEAN, KW_CASE, KW_DOUBLE, KW_FLOAT, KW_IF, KW_IN, KW_INT, KW_LIKE, KW_MAP, KW_NOT, KW_OR, KW_REGEXP, KW_RLIKE, KW_SMALLINT, KW_STRING..KW_STRUCT, KW_TINYINT, KW_UNIONTYPE, KW_WHEN, LESSTHAN..LESSTHANOREQUALTO, MINUS..NOTEQUAL, PLUS, STAR, TILDE}" using multiple alternatives: 1, 3
          
          As a result, alternative(s) 3 were disabled for that input
          [INFO] 
          [INFO] --- maven-remote-resources-plugin:1.5:process (default) @ hive-exec ---
          Downloading: http://www.datanucleus.org/downloads/maven2/net/hydromatic/linq4j/0.4/linq4j-0.4.pom
          [INFO] ------------------------------------------------------------------------
          [INFO] Reactor Summary:
          [INFO] 
          [INFO] Hive .............................................. SUCCESS [14.529s]
          [INFO] Hive Shims Common ................................. SUCCESS [7.647s]
          [INFO] Hive Shims 0.20 ................................... SUCCESS [3.633s]
          [INFO] Hive Shims Secure Common .......................... SUCCESS [4.802s]
          [INFO] Hive Shims 0.20S .................................. SUCCESS [2.212s]
          [INFO] Hive Shims 0.23 ................................... SUCCESS [6.741s]
          [INFO] Hive Shims ........................................ SUCCESS [0.652s]
          [INFO] Hive Common ....................................... SUCCESS [8.891s]
          [INFO] Hive Serde ........................................ SUCCESS [15.604s]
          [INFO] Hive Metastore .................................... SUCCESS [37.244s]
          [INFO] Hive Ant Utilities ................................ SUCCESS [1.620s]
          [INFO] Hive Query Language ............................... FAILURE [29.486s]
          [INFO] Hive Service ...................................... SKIPPED
          [INFO] Hive Accumulo Handler ............................. SKIPPED
          [INFO] Hive JDBC ......................................... SKIPPED
          [INFO] Hive Beeline ...................................... SKIPPED
          [INFO] Hive CLI .......................................... SKIPPED
          [INFO] Hive Contrib ...................................... SKIPPED
          [INFO] Hive HBase Handler ................................ SKIPPED
          [INFO] Hive HCatalog ..................................... SKIPPED
          [INFO] Hive HCatalog Core ................................ SKIPPED
          [INFO] Hive HCatalog Pig Adapter ......................... SKIPPED
          [INFO] Hive HCatalog Server Extensions ................... SKIPPED
          [INFO] Hive HCatalog Webhcat Java Client ................. SKIPPED
          [INFO] Hive HCatalog Webhcat ............................. SKIPPED
          [INFO] Hive HCatalog Streaming ........................... SKIPPED
          [INFO] Hive HWI .......................................... SKIPPED
          [INFO] Hive ODBC ......................................... SKIPPED
          [INFO] Hive Shims Aggregator ............................. SKIPPED
          [INFO] Hive TestUtils .................................... SKIPPED
          [INFO] Hive Packaging .................................... SKIPPED
          [INFO] ------------------------------------------------------------------------
          [INFO] BUILD FAILURE
          [INFO] ------------------------------------------------------------------------
          [INFO] Total time: 2:15.907s
          [INFO] Finished at: Wed Oct 08 13:56:13 EDT 2014
          [INFO] Final Memory: 69M/413M
          [INFO] ------------------------------------------------------------------------
          [ERROR] Failed to execute goal org.apache.maven.plugins:maven-remote-resources-plugin:1.5:process (default) on project hive-exec: Error resolving project artifact: Could not transfer artifact net.hydromatic:linq4j:pom:0.4 from/to datanucleus (http://www.datanucleus.org/downloads/maven2): Access denied to: http://www.datanucleus.org/downloads/maven2/net/hydromatic/linq4j/0.4/linq4j-0.4.pom, ReasonPhrase: Forbidden. for project net.hydromatic:linq4j:jar:0.4 -> [Help 1]
          [ERROR] 
          [ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
          [ERROR] Re-run Maven using the -X switch to enable full debug logging.
          [ERROR] 
          [ERROR] For more information about the errors and possible solutions, please read the following articles:
          [ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoExecutionException
          [ERROR] 
          [ERROR] After correcting the problems, you can resume the build with the command
          [ERROR]   mvn <goals> -rf :hive-exec
          + exit 1
          '
          

          This message is automatically generated.

          ATTACHMENT ID: 12673513

          Show
          Hive QA added a comment - Overall : -1 no tests executed Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12673513/HIVE-7858.patch Test results: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/1174/testReport Console output: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/1174/console Test logs: http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-1174/ Messages: **** This message was trimmed, see log for full details **** warning(200): IdentifiersParser.g:68:4: Decision can match input such as "LPAREN LPAREN KW_CASE" using multiple alternatives: 1, 2 As a result, alternative(s) 2 were disabled for that input warning(200): IdentifiersParser.g:68:4: Decision can match input such as "LPAREN KW_NOT KW_FALSE" using multiple alternatives: 1, 2 As a result, alternative(s) 2 were disabled for that input warning(200): IdentifiersParser.g:68:4: Decision can match input such as "LPAREN KW_DATE StringLiteral" using multiple alternatives: 1, 2 As a result, alternative(s) 2 were disabled for that input warning(200): IdentifiersParser.g:68:4: Decision can match input such as "LPAREN KW_NOT KW_TRUE" using multiple alternatives: 1, 2 As a result, alternative(s) 2 were disabled for that input warning(200): IdentifiersParser.g:68:4: Decision can match input such as "LPAREN KW_NOT KW_MAP" using multiple alternatives: 1, 2 As a result, alternative(s) 2 were disabled for that input warning(200): IdentifiersParser.g:68:4: Decision can match input such as "LPAREN KW_CASE KW_MAP" using multiple alternatives: 1, 2 As a result, alternative(s) 2 were disabled for that input warning(200): IdentifiersParser.g:68:4: Decision can match input such as "LPAREN LPAREN KW_MAP" using multiple alternatives: 1, 2 As a result, alternative(s) 2 were disabled for that input warning(200): IdentifiersParser.g:68:4: Decision can match input such as "LPAREN KW_CASE KW_UNIONTYPE" using multiple alternatives: 1, 2 As a result, alternative(s) 2 were disabled for that input warning(200): IdentifiersParser.g:68:4: Decision can match input such as "LPAREN KW_CASE KW_STRUCT" using multiple alternatives: 1, 2 As a result, alternative(s) 2 were disabled for that input warning(200): IdentifiersParser.g:68:4: Decision can match input such as "LPAREN KW_NOT KW_IF" using multiple alternatives: 1, 2 As a result, alternative(s) 2 were disabled for that input warning(200): IdentifiersParser.g:68:4: Decision can match input such as "LPAREN KW_CASE KW_IF" using multiple alternatives: 1, 2 As a result, alternative(s) 2 were disabled for that input warning(200): IdentifiersParser.g:68:4: Decision can match input such as "LPAREN LPAREN KW_IF" using multiple alternatives: 1, 2 As a result, alternative(s) 2 were disabled for that input warning(200): IdentifiersParser.g:115:5: Decision can match input such as "KW_CLUSTER KW_BY LPAREN" using multiple alternatives: 1, 2 As a result, alternative(s) 2 were disabled for that input warning(200): IdentifiersParser.g:127:5: Decision can match input such as "KW_PARTITION KW_BY LPAREN" using multiple alternatives: 1, 2 As a result, alternative(s) 2 were disabled for that input warning(200): IdentifiersParser.g:138:5: Decision can match input such as "KW_DISTRIBUTE KW_BY LPAREN" using multiple alternatives: 1, 2 As a result, alternative(s) 2 were disabled for that input warning(200): IdentifiersParser.g:149:5: Decision can match input such as "KW_SORT KW_BY LPAREN" using multiple alternatives: 1, 2 As a result, alternative(s) 2 were disabled for that input warning(200): IdentifiersParser.g:166:7: Decision can match input such as "STAR" using multiple alternatives: 1, 2 As a result, alternative(s) 2 were disabled for that input warning(200): IdentifiersParser.g:179:5: Decision can match input such as "KW_STRUCT" using multiple alternatives: 4, 6 As a result, alternative(s) 6 were disabled for that input warning(200): IdentifiersParser.g:179:5: Decision can match input such as "KW_ARRAY" using multiple alternatives: 2, 6 As a result, alternative(s) 6 were disabled for that input warning(200): IdentifiersParser.g:179:5: Decision can match input such as "KW_UNIONTYPE" using multiple alternatives: 5, 6 As a result, alternative(s) 6 were disabled for that input warning(200): IdentifiersParser.g:261:5: Decision can match input such as "KW_TRUE" using multiple alternatives: 3, 8 As a result, alternative(s) 8 were disabled for that input warning(200): IdentifiersParser.g:261:5: Decision can match input such as "KW_NULL" using multiple alternatives: 1, 8 As a result, alternative(s) 8 were disabled for that input warning(200): IdentifiersParser.g:261:5: Decision can match input such as "KW_FALSE" using multiple alternatives: 3, 8 As a result, alternative(s) 8 were disabled for that input warning(200): IdentifiersParser.g:261:5: Decision can match input such as "KW_DATE StringLiteral" using multiple alternatives: 2, 3 As a result, alternative(s) 3 were disabled for that input warning(200): IdentifiersParser.g:393:5: Decision can match input such as "KW_BETWEEN KW_MAP LPAREN" using multiple alternatives: 8, 9 As a result, alternative(s) 9 were disabled for that input warning(200): IdentifiersParser.g:393:5: Decision can match input such as "{KW_LIKE, KW_REGEXP, KW_RLIKE} KW_INSERT KW_INTO" using multiple alternatives: 2, 9 As a result, alternative(s) 9 were disabled for that input warning(200): IdentifiersParser.g:393:5: Decision can match input such as "{KW_LIKE, KW_REGEXP, KW_RLIKE} KW_UNION KW_ALL" using multiple alternatives: 2, 9 As a result, alternative(s) 9 were disabled for that input warning(200): IdentifiersParser.g:393:5: Decision can match input such as "{KW_LIKE, KW_REGEXP, KW_RLIKE} KW_SORT KW_BY" using multiple alternatives: 2, 9 As a result, alternative(s) 9 were disabled for that input warning(200): IdentifiersParser.g:393:5: Decision can match input such as "{KW_LIKE, KW_REGEXP, KW_RLIKE} KW_CLUSTER KW_BY" using multiple alternatives: 2, 9 As a result, alternative(s) 9 were disabled for that input warning(200): IdentifiersParser.g:393:5: Decision can match input such as "{KW_LIKE, KW_REGEXP, KW_RLIKE} KW_LATERAL KW_VIEW" using multiple alternatives: 2, 9 As a result, alternative(s) 9 were disabled for that input warning(200): IdentifiersParser.g:393:5: Decision can match input such as "{KW_LIKE, KW_REGEXP, KW_RLIKE} KW_MAP LPAREN" using multiple alternatives: 2, 9 As a result, alternative(s) 9 were disabled for that input warning(200): IdentifiersParser.g:393:5: Decision can match input such as "{KW_LIKE, KW_REGEXP, KW_RLIKE} KW_DISTRIBUTE KW_BY" using multiple alternatives: 2, 9 As a result, alternative(s) 9 were disabled for that input warning(200): IdentifiersParser.g:393:5: Decision can match input such as "{KW_LIKE, KW_REGEXP, KW_RLIKE} KW_ORDER KW_BY" using multiple alternatives: 2, 9 As a result, alternative(s) 9 were disabled for that input warning(200): IdentifiersParser.g:393:5: Decision can match input such as "{KW_LIKE, KW_REGEXP, KW_RLIKE} KW_GROUP KW_BY" using multiple alternatives: 2, 9 As a result, alternative(s) 9 were disabled for that input warning(200): IdentifiersParser.g:393:5: Decision can match input such as "{KW_LIKE, KW_REGEXP, KW_RLIKE} KW_INSERT KW_OVERWRITE" using multiple alternatives: 2, 9 As a result, alternative(s) 9 were disabled for that input warning(200): IdentifiersParser.g:518:5: Decision can match input such as "{AMPERSAND..BITWISEXOR, DIV..DIVIDE, EQUAL..EQUAL_NS, GREATERTHAN..GREATERTHANOREQUALTO, KW_AND, KW_ARRAY, KW_BETWEEN..KW_BOOLEAN, KW_CASE, KW_DOUBLE, KW_FLOAT, KW_IF, KW_IN, KW_INT, KW_LIKE, KW_MAP, KW_NOT, KW_OR, KW_REGEXP, KW_RLIKE, KW_SMALLINT, KW_STRING..KW_STRUCT, KW_TINYINT, KW_UNIONTYPE, KW_WHEN, LESSTHAN..LESSTHANOREQUALTO, MINUS..NOTEQUAL, PLUS, STAR, TILDE}" using multiple alternatives: 1, 3 As a result, alternative(s) 3 were disabled for that input [INFO] [INFO] --- maven-remote-resources-plugin:1.5:process (default) @ hive-exec --- Downloading: http://www.datanucleus.org/downloads/maven2/net/hydromatic/linq4j/0.4/linq4j-0.4.pom [INFO] ------------------------------------------------------------------------ [INFO] Reactor Summary: [INFO] [INFO] Hive .............................................. SUCCESS [14.529s] [INFO] Hive Shims Common ................................. SUCCESS [7.647s] [INFO] Hive Shims 0.20 ................................... SUCCESS [3.633s] [INFO] Hive Shims Secure Common .......................... SUCCESS [4.802s] [INFO] Hive Shims 0.20S .................................. SUCCESS [2.212s] [INFO] Hive Shims 0.23 ................................... SUCCESS [6.741s] [INFO] Hive Shims ........................................ SUCCESS [0.652s] [INFO] Hive Common ....................................... SUCCESS [8.891s] [INFO] Hive Serde ........................................ SUCCESS [15.604s] [INFO] Hive Metastore .................................... SUCCESS [37.244s] [INFO] Hive Ant Utilities ................................ SUCCESS [1.620s] [INFO] Hive Query Language ............................... FAILURE [29.486s] [INFO] Hive Service ...................................... SKIPPED [INFO] Hive Accumulo Handler ............................. SKIPPED [INFO] Hive JDBC ......................................... SKIPPED [INFO] Hive Beeline ...................................... SKIPPED [INFO] Hive CLI .......................................... SKIPPED [INFO] Hive Contrib ...................................... SKIPPED [INFO] Hive HBase Handler ................................ SKIPPED [INFO] Hive HCatalog ..................................... SKIPPED [INFO] Hive HCatalog Core ................................ SKIPPED [INFO] Hive HCatalog Pig Adapter ......................... SKIPPED [INFO] Hive HCatalog Server Extensions ................... SKIPPED [INFO] Hive HCatalog Webhcat Java Client ................. SKIPPED [INFO] Hive HCatalog Webhcat ............................. SKIPPED [INFO] Hive HCatalog Streaming ........................... SKIPPED [INFO] Hive HWI .......................................... SKIPPED [INFO] Hive ODBC ......................................... SKIPPED [INFO] Hive Shims Aggregator ............................. SKIPPED [INFO] Hive TestUtils .................................... SKIPPED [INFO] Hive Packaging .................................... SKIPPED [INFO] ------------------------------------------------------------------------ [INFO] BUILD FAILURE [INFO] ------------------------------------------------------------------------ [INFO] Total time: 2:15.907s [INFO] Finished at: Wed Oct 08 13:56:13 EDT 2014 [INFO] Final Memory: 69M/413M [INFO] ------------------------------------------------------------------------ [ERROR] Failed to execute goal org.apache.maven.plugins:maven-remote-resources-plugin:1.5:process (default) on project hive-exec: Error resolving project artifact: Could not transfer artifact net.hydromatic:linq4j:pom:0.4 from/to datanucleus (http://www.datanucleus.org/downloads/maven2): Access denied to: http://www.datanucleus.org/downloads/maven2/net/hydromatic/linq4j/0.4/linq4j-0.4.pom, ReasonPhrase: Forbidden. for project net.hydromatic:linq4j:jar:0.4 -> [Help 1] [ERROR] [ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch. [ERROR] Re-run Maven using the -X switch to enable full debug logging. [ERROR] [ERROR] For more information about the errors and possible solutions, please read the following articles: [ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoExecutionException [ERROR] [ERROR] After correcting the problems, you can resume the build with the command [ERROR] mvn <goals> -rf :hive-exec + exit 1 ' This message is automatically generated. ATTACHMENT ID: 12673513
          Hide
          Ferdinand Xu added a comment -

          review board entry created in https://reviews.apache.org/r/26434/

          Show
          Ferdinand Xu added a comment - review board entry created in https://reviews.apache.org/r/26434/

            People

            • Assignee:
              Ferdinand Xu
              Reporter:
              Brock Noland
            • Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development