Hive
  1. Hive
  2. HIVE-6347

ZeroCopy read path for ORC RecordReader

    Details

    • Type: Bug Bug
    • Status: Resolved
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: tez-branch
    • Fix Version/s: tez-branch
    • Component/s: File Formats
    • Labels:
      None
    • Release Note:
      ZeroCopy readers for the ORC file format

      Description

      ORC can use the new HDFS Caching APIs and the ZeroCopy readers to avoid extra data copies into memory while scanning files.

      Implement ORC zcr codepath and a hive.orc.zerocopy flag.

      1. HIVE-6347.5-tez.patch
        24 kB
        Gopal V
      2. HIVE-6347.4-tez.patch
        26 kB
        Gopal V
      3. HIVE-6347.3-tez.patch
        27 kB
        Gopal V
      4. HIVE-6347.2-tez.patch
        27 kB
        Gopal V
      5. HIVE-6347.1.patch
        26 kB
        Gopal V

        Issue Links

          Activity

          Hide
          Gopal V added a comment -

          Patch which applies over HIVE-6346 on hive/tez branch

          Show
          Gopal V added a comment - Patch which applies over HIVE-6346 on hive/tez branch
          Hide
          Lefty Leverenz added a comment -

          The patch adds hive.orc.zerocopy to HiveConf.java, so it also needs to document the flag in hive-default.xml.template.

          Show
          Lefty Leverenz added a comment - The patch adds hive.orc.zerocopy to HiveConf.java, so it also needs to document the flag in hive-default.xml.template.
          Hide
          Hive QA added a comment -

          Overall: -1 no tests executed

          Here are the results of testing the latest attachment:
          https://issues.apache.org/jira/secure/attachment/12626451/HIVE-6347.1.patch

          Test results: http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/1151/testReport
          Console output: http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/1151/console

          Messages:

          **** This message was trimmed, see log for full details ****
          As a result, alternative(s) 9 were disabled for that input
          warning(200): IdentifiersParser.g:399:5: 
          Decision can match input such as "{KW_LIKE, KW_REGEXP, KW_RLIKE} KW_ORDER KW_BY" using multiple alternatives: 2, 9
          
          As a result, alternative(s) 9 were disabled for that input
          warning(200): IdentifiersParser.g:399:5: 
          Decision can match input such as "{KW_LIKE, KW_REGEXP, KW_RLIKE} KW_GROUP KW_BY" using multiple alternatives: 2, 9
          
          As a result, alternative(s) 9 were disabled for that input
          warning(200): IdentifiersParser.g:399:5: 
          Decision can match input such as "{KW_LIKE, KW_REGEXP, KW_RLIKE} KW_MAP LPAREN" using multiple alternatives: 2, 9
          
          As a result, alternative(s) 9 were disabled for that input
          warning(200): IdentifiersParser.g:399:5: 
          Decision can match input such as "{KW_LIKE, KW_REGEXP, KW_RLIKE} KW_INSERT KW_INTO" using multiple alternatives: 2, 9
          
          As a result, alternative(s) 9 were disabled for that input
          warning(200): IdentifiersParser.g:399:5: 
          Decision can match input such as "{KW_LIKE, KW_REGEXP, KW_RLIKE} KW_LATERAL KW_VIEW" using multiple alternatives: 2, 9
          
          As a result, alternative(s) 9 were disabled for that input
          warning(200): IdentifiersParser.g:524:5: 
          Decision can match input such as "{AMPERSAND..BITWISEXOR, DIV..DIVIDE, EQUAL..EQUAL_NS, GREATERTHAN..GREATERTHANOREQUALTO, KW_AND, KW_ARRAY, KW_BETWEEN..KW_BOOLEAN, KW_CASE, KW_DOUBLE, KW_FLOAT, KW_IF, KW_IN, KW_INT, KW_LIKE, KW_MAP, KW_NOT, KW_OR, KW_REGEXP, KW_RLIKE, KW_SMALLINT, KW_STRING..KW_STRUCT, KW_TINYINT, KW_UNIONTYPE, KW_WHEN, LESSTHAN..LESSTHANOREQUALTO, MINUS..NOTEQUAL, PLUS, STAR, TILDE}" using multiple alternatives: 1, 3
          
          As a result, alternative(s) 3 were disabled for that input
          [INFO] 
          [INFO] --- maven-resources-plugin:2.5:resources (default-resources) @ hive-exec ---
          [debug] execute contextualize
          [INFO] Using 'UTF-8' encoding to copy filtered resources.
          [INFO] Copying 1 resource
          [INFO] 
          [INFO] --- maven-antrun-plugin:1.7:run (define-classpath) @ hive-exec ---
          [INFO] Executing tasks
          
          main:
          [INFO] Executed tasks
          [INFO] 
          [INFO] --- maven-compiler-plugin:3.1:compile (default-compile) @ hive-exec ---
          [INFO] Compiling 1546 source files to /data/hive-ptest/working/apache-svn-trunk-source/ql/target/classes
          [INFO] -------------------------------------------------------------
          [WARNING] COMPILATION WARNING : 
          [INFO] -------------------------------------------------------------
          [WARNING] Note: Some input files use or override a deprecated API.
          [WARNING] Note: Recompile with -Xlint:deprecation for details.
          [WARNING] Note: Some input files use unchecked or unsafe operations.
          [WARNING] Note: Recompile with -Xlint:unchecked for details.
          [INFO] 4 warnings 
          [INFO] -------------------------------------------------------------
          [INFO] -------------------------------------------------------------
          [ERROR] COMPILATION ERROR : 
          [INFO] -------------------------------------------------------------
          [ERROR] /data/hive-ptest/working/apache-svn-trunk-source/ql/src/java/org/apache/hadoop/hive/ql/io/orc/SnappyCodec.java:[21,48] cannot find symbol
          symbol  : class DirectDecompressorShim
          location: interface org.apache.hadoop.hive.shims.HadoopShims
          [ERROR] /data/hive-ptest/working/apache-svn-trunk-source/ql/src/java/org/apache/hadoop/hive/ql/io/orc/SnappyCodec.java:[23,48] cannot find symbol
          symbol  : class DirectCompressionType
          location: interface org.apache.hadoop.hive.shims.HadoopShims
          [ERROR] /data/hive-ptest/working/apache-svn-trunk-source/ql/src/java/org/apache/hadoop/hive/ql/io/orc/RecordReaderImpl.java:[51,48] cannot find symbol
          symbol  : class ByteBufferPoolShim
          location: interface org.apache.hadoop.hive.shims.HadoopShims
          [ERROR] /data/hive-ptest/working/apache-svn-trunk-source/ql/src/java/org/apache/hadoop/hive/ql/io/orc/RecordReaderImpl.java:[52,48] cannot find symbol
          symbol  : class ZeroCopyReaderShim
          location: interface org.apache.hadoop.hive.shims.HadoopShims
          [ERROR] /data/hive-ptest/working/apache-svn-trunk-source/ql/src/java/org/apache/hadoop/hive/ql/io/orc/RecordReaderImpl.java:[100,64] cannot find symbol
          symbol  : class ByteBufferPoolShim
          location: class org.apache.hadoop.hive.ql.io.orc.RecordReaderImpl
          [ERROR] /data/hive-ptest/working/apache-svn-trunk-source/ql/src/java/org/apache/hadoop/hive/ql/io/orc/RecordReaderImpl.java:[96,17] cannot find symbol
          symbol  : class ZeroCopyReaderShim
          location: class org.apache.hadoop.hive.ql.io.orc.RecordReaderImpl
          [ERROR] /data/hive-ptest/working/apache-svn-trunk-source/ql/src/java/org/apache/hadoop/hive/ql/io/orc/ZlibCodec.java:[27,48] cannot find symbol
          symbol  : class DirectCompressionType
          location: interface org.apache.hadoop.hive.shims.HadoopShims
          [ERROR] /data/hive-ptest/working/apache-svn-trunk-source/ql/src/java/org/apache/hadoop/hive/ql/io/orc/ZlibCodec.java:[28,48] cannot find symbol
          symbol  : class DirectDecompressorShim
          location: interface org.apache.hadoop.hive.shims.HadoopShims
          [ERROR] /data/hive-ptest/working/apache-svn-trunk-source/ql/src/java/org/apache/hadoop/hive/ql/io/orc/SnappyCodec.java:[82,11] cannot find symbol
          symbol  : variable DirectCompressionType
          location: class org.apache.hadoop.hive.ql.io.orc.SnappyCodec
          [ERROR] /data/hive-ptest/working/apache-svn-trunk-source/ql/src/java/org/apache/hadoop/hive/ql/io/orc/SnappyCodec.java:[94,5] cannot find symbol
          symbol  : class DirectDecompressorShim
          location: class org.apache.hadoop.hive.ql.io.orc.SnappyCodec
          [ERROR] /data/hive-ptest/working/apache-svn-trunk-source/ql/src/java/org/apache/hadoop/hive/ql/io/orc/SnappyCodec.java:[95,32] cannot find symbol
          symbol  : variable DirectCompressionType
          location: class org.apache.hadoop.hive.ql.io.orc.SnappyCodec
          [ERROR] /data/hive-ptest/working/apache-svn-trunk-source/ql/src/java/org/apache/hadoop/hive/ql/io/orc/RecordReaderImpl.java:[153,5] method does not override or implement a method from a supertype
          [ERROR] /data/hive-ptest/working/apache-svn-trunk-source/ql/src/java/org/apache/hadoop/hive/ql/io/orc/RecordReaderImpl.java:[165,5] method does not override or implement a method from a supertype
          [ERROR] /data/hive-ptest/working/apache-svn-trunk-source/ql/src/java/org/apache/hadoop/hive/ql/io/orc/RecordReaderImpl.java:[228,45] cannot find symbol
          symbol  : method getZeroCopyReader(org.apache.hadoop.fs.FSDataInputStream,org.apache.hadoop.hive.ql.io.orc.RecordReaderImpl.ByteBufferAllocatorPool)
          location: interface org.apache.hadoop.hive.shims.HadoopShims
          [ERROR] /data/hive-ptest/working/apache-svn-trunk-source/ql/src/java/org/apache/hadoop/hive/ql/io/orc/ZlibCodec.java:[95,11] cannot find symbol
          symbol  : variable DirectCompressionType
          location: class org.apache.hadoop.hive.ql.io.orc.ZlibCodec
          [ERROR] /data/hive-ptest/working/apache-svn-trunk-source/ql/src/java/org/apache/hadoop/hive/ql/io/orc/ZlibCodec.java:[107,5] cannot find symbol
          symbol  : class DirectDecompressorShim
          location: class org.apache.hadoop.hive.ql.io.orc.ZlibCodec
          [ERROR] /data/hive-ptest/working/apache-svn-trunk-source/ql/src/java/org/apache/hadoop/hive/ql/io/orc/ZlibCodec.java:[108,32] cannot find symbol
          symbol  : variable DirectCompressionType
          location: class org.apache.hadoop.hive.ql.io.orc.ZlibCodec
          [INFO] 17 errors 
          [INFO] -------------------------------------------------------------
          [INFO] ------------------------------------------------------------------------
          [INFO] Reactor Summary:
          [INFO] 
          [INFO] Hive .............................................. SUCCESS [4.645s]
          [INFO] Hive Ant Utilities ................................ SUCCESS [6.787s]
          [INFO] Hive Shims Common ................................. SUCCESS [3.329s]
          [INFO] Hive Shims 0.20 ................................... SUCCESS [2.245s]
          [INFO] Hive Shims Secure Common .......................... SUCCESS [2.600s]
          [INFO] Hive Shims 0.20S .................................. SUCCESS [1.428s]
          [INFO] Hive Shims 0.23 ................................... SUCCESS [3.198s]
          [INFO] Hive Shims ........................................ SUCCESS [0.590s]
          [INFO] Hive Common ....................................... SUCCESS [7.734s]
          [INFO] Hive Serde ........................................ SUCCESS [8.743s]
          [INFO] Hive Metastore .................................... SUCCESS [26.982s]
          [INFO] Hive Query Language ............................... FAILURE [40.846s]
          [INFO] Hive Service ...................................... SKIPPED
          [INFO] Hive JDBC ......................................... SKIPPED
          [INFO] Hive Beeline ...................................... SKIPPED
          [INFO] Hive CLI .......................................... SKIPPED
          [INFO] Hive Contrib ...................................... SKIPPED
          [INFO] Hive HBase Handler ................................ SKIPPED
          [INFO] Hive HCatalog ..................................... SKIPPED
          [INFO] Hive HCatalog Core ................................ SKIPPED
          [INFO] Hive HCatalog Pig Adapter ......................... SKIPPED
          [INFO] Hive HCatalog Server Extensions ................... SKIPPED
          [INFO] Hive HCatalog Webhcat Java Client ................. SKIPPED
          [INFO] Hive HCatalog Webhcat ............................. SKIPPED
          [INFO] Hive HCatalog HBase Storage Handler ............... SKIPPED
          [INFO] Hive HWI .......................................... SKIPPED
          [INFO] Hive ODBC ......................................... SKIPPED
          [INFO] Hive Shims Aggregator ............................. SKIPPED
          [INFO] Hive TestUtils .................................... SKIPPED
          [INFO] Hive Packaging .................................... SKIPPED
          [INFO] ------------------------------------------------------------------------
          [INFO] BUILD FAILURE
          [INFO] ------------------------------------------------------------------------
          [INFO] Total time: 1:52.063s
          [INFO] Finished at: Sat Feb 01 08:39:12 EST 2014
          [INFO] Final Memory: 55M/422M
          [INFO] ------------------------------------------------------------------------
          [ERROR] Failed to execute goal org.apache.maven.plugins:maven-compiler-plugin:3.1:compile (default-compile) on project hive-exec: Compilation failure: Compilation failure:
          [ERROR] /data/hive-ptest/working/apache-svn-trunk-source/ql/src/java/org/apache/hadoop/hive/ql/io/orc/SnappyCodec.java:[21,48] cannot find symbol
          [ERROR] symbol  : class DirectDecompressorShim
          [ERROR] location: interface org.apache.hadoop.hive.shims.HadoopShims
          [ERROR] /data/hive-ptest/working/apache-svn-trunk-source/ql/src/java/org/apache/hadoop/hive/ql/io/orc/SnappyCodec.java:[23,48] cannot find symbol
          [ERROR] symbol  : class DirectCompressionType
          [ERROR] location: interface org.apache.hadoop.hive.shims.HadoopShims
          [ERROR] /data/hive-ptest/working/apache-svn-trunk-source/ql/src/java/org/apache/hadoop/hive/ql/io/orc/RecordReaderImpl.java:[51,48] cannot find symbol
          [ERROR] symbol  : class ByteBufferPoolShim
          [ERROR] location: interface org.apache.hadoop.hive.shims.HadoopShims
          [ERROR] /data/hive-ptest/working/apache-svn-trunk-source/ql/src/java/org/apache/hadoop/hive/ql/io/orc/RecordReaderImpl.java:[52,48] cannot find symbol
          [ERROR] symbol  : class ZeroCopyReaderShim
          [ERROR] location: interface org.apache.hadoop.hive.shims.HadoopShims
          [ERROR] /data/hive-ptest/working/apache-svn-trunk-source/ql/src/java/org/apache/hadoop/hive/ql/io/orc/RecordReaderImpl.java:[100,64] cannot find symbol
          [ERROR] symbol  : class ByteBufferPoolShim
          [ERROR] location: class org.apache.hadoop.hive.ql.io.orc.RecordReaderImpl
          [ERROR] /data/hive-ptest/working/apache-svn-trunk-source/ql/src/java/org/apache/hadoop/hive/ql/io/orc/RecordReaderImpl.java:[96,17] cannot find symbol
          [ERROR] symbol  : class ZeroCopyReaderShim
          [ERROR] location: class org.apache.hadoop.hive.ql.io.orc.RecordReaderImpl
          [ERROR] /data/hive-ptest/working/apache-svn-trunk-source/ql/src/java/org/apache/hadoop/hive/ql/io/orc/ZlibCodec.java:[27,48] cannot find symbol
          [ERROR] symbol  : class DirectCompressionType
          [ERROR] location: interface org.apache.hadoop.hive.shims.HadoopShims
          [ERROR] /data/hive-ptest/working/apache-svn-trunk-source/ql/src/java/org/apache/hadoop/hive/ql/io/orc/ZlibCodec.java:[28,48] cannot find symbol
          [ERROR] symbol  : class DirectDecompressorShim
          [ERROR] location: interface org.apache.hadoop.hive.shims.HadoopShims
          [ERROR] /data/hive-ptest/working/apache-svn-trunk-source/ql/src/java/org/apache/hadoop/hive/ql/io/orc/SnappyCodec.java:[82,11] cannot find symbol
          [ERROR] symbol  : variable DirectCompressionType
          [ERROR] location: class org.apache.hadoop.hive.ql.io.orc.SnappyCodec
          [ERROR] /data/hive-ptest/working/apache-svn-trunk-source/ql/src/java/org/apache/hadoop/hive/ql/io/orc/SnappyCodec.java:[94,5] cannot find symbol
          [ERROR] symbol  : class DirectDecompressorShim
          [ERROR] location: class org.apache.hadoop.hive.ql.io.orc.SnappyCodec
          [ERROR] /data/hive-ptest/working/apache-svn-trunk-source/ql/src/java/org/apache/hadoop/hive/ql/io/orc/SnappyCodec.java:[95,32] cannot find symbol
          [ERROR] symbol  : variable DirectCompressionType
          [ERROR] location: class org.apache.hadoop.hive.ql.io.orc.SnappyCodec
          [ERROR] /data/hive-ptest/working/apache-svn-trunk-source/ql/src/java/org/apache/hadoop/hive/ql/io/orc/RecordReaderImpl.java:[153,5] method does not override or implement a method from a supertype
          [ERROR] /data/hive-ptest/working/apache-svn-trunk-source/ql/src/java/org/apache/hadoop/hive/ql/io/orc/RecordReaderImpl.java:[165,5] method does not override or implement a method from a supertype
          [ERROR] /data/hive-ptest/working/apache-svn-trunk-source/ql/src/java/org/apache/hadoop/hive/ql/io/orc/RecordReaderImpl.java:[228,45] cannot find symbol
          [ERROR] symbol  : method getZeroCopyReader(org.apache.hadoop.fs.FSDataInputStream,org.apache.hadoop.hive.ql.io.orc.RecordReaderImpl.ByteBufferAllocatorPool)
          [ERROR] location: interface org.apache.hadoop.hive.shims.HadoopShims
          [ERROR] /data/hive-ptest/working/apache-svn-trunk-source/ql/src/java/org/apache/hadoop/hive/ql/io/orc/ZlibCodec.java:[95,11] cannot find symbol
          [ERROR] symbol  : variable DirectCompressionType
          [ERROR] location: class org.apache.hadoop.hive.ql.io.orc.ZlibCodec
          [ERROR] /data/hive-ptest/working/apache-svn-trunk-source/ql/src/java/org/apache/hadoop/hive/ql/io/orc/ZlibCodec.java:[107,5] cannot find symbol
          [ERROR] symbol  : class DirectDecompressorShim
          [ERROR] location: class org.apache.hadoop.hive.ql.io.orc.ZlibCodec
          [ERROR] /data/hive-ptest/working/apache-svn-trunk-source/ql/src/java/org/apache/hadoop/hive/ql/io/orc/ZlibCodec.java:[108,32] cannot find symbol
          [ERROR] symbol  : variable DirectCompressionType
          [ERROR] location: class org.apache.hadoop.hive.ql.io.orc.ZlibCodec
          [ERROR] -> [Help 1]
          [ERROR] 
          [ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
          [ERROR] Re-run Maven using the -X switch to enable full debug logging.
          [ERROR] 
          [ERROR] For more information about the errors and possible solutions, please read the following articles:
          [ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoFailureException
          [ERROR] 
          [ERROR] After correcting the problems, you can resume the build with the command
          [ERROR]   mvn <goals> -rf :hive-exec
          + exit 1
          '
          

          This message is automatically generated.

          ATTACHMENT ID: 12626451

          Show
          Hive QA added a comment - Overall : -1 no tests executed Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12626451/HIVE-6347.1.patch Test results: http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/1151/testReport Console output: http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/1151/console Messages: **** This message was trimmed, see log for full details **** As a result, alternative(s) 9 were disabled for that input warning(200): IdentifiersParser.g:399:5: Decision can match input such as "{KW_LIKE, KW_REGEXP, KW_RLIKE} KW_ORDER KW_BY" using multiple alternatives: 2, 9 As a result, alternative(s) 9 were disabled for that input warning(200): IdentifiersParser.g:399:5: Decision can match input such as "{KW_LIKE, KW_REGEXP, KW_RLIKE} KW_GROUP KW_BY" using multiple alternatives: 2, 9 As a result, alternative(s) 9 were disabled for that input warning(200): IdentifiersParser.g:399:5: Decision can match input such as "{KW_LIKE, KW_REGEXP, KW_RLIKE} KW_MAP LPAREN" using multiple alternatives: 2, 9 As a result, alternative(s) 9 were disabled for that input warning(200): IdentifiersParser.g:399:5: Decision can match input such as "{KW_LIKE, KW_REGEXP, KW_RLIKE} KW_INSERT KW_INTO" using multiple alternatives: 2, 9 As a result, alternative(s) 9 were disabled for that input warning(200): IdentifiersParser.g:399:5: Decision can match input such as "{KW_LIKE, KW_REGEXP, KW_RLIKE} KW_LATERAL KW_VIEW" using multiple alternatives: 2, 9 As a result, alternative(s) 9 were disabled for that input warning(200): IdentifiersParser.g:524:5: Decision can match input such as "{AMPERSAND..BITWISEXOR, DIV..DIVIDE, EQUAL..EQUAL_NS, GREATERTHAN..GREATERTHANOREQUALTO, KW_AND, KW_ARRAY, KW_BETWEEN..KW_BOOLEAN, KW_CASE, KW_DOUBLE, KW_FLOAT, KW_IF, KW_IN, KW_INT, KW_LIKE, KW_MAP, KW_NOT, KW_OR, KW_REGEXP, KW_RLIKE, KW_SMALLINT, KW_STRING..KW_STRUCT, KW_TINYINT, KW_UNIONTYPE, KW_WHEN, LESSTHAN..LESSTHANOREQUALTO, MINUS..NOTEQUAL, PLUS, STAR, TILDE}" using multiple alternatives: 1, 3 As a result, alternative(s) 3 were disabled for that input [INFO] [INFO] --- maven-resources-plugin:2.5:resources (default-resources) @ hive-exec --- [debug] execute contextualize [INFO] Using 'UTF-8' encoding to copy filtered resources. [INFO] Copying 1 resource [INFO] [INFO] --- maven-antrun-plugin:1.7:run (define-classpath) @ hive-exec --- [INFO] Executing tasks main: [INFO] Executed tasks [INFO] [INFO] --- maven-compiler-plugin:3.1:compile (default-compile) @ hive-exec --- [INFO] Compiling 1546 source files to /data/hive-ptest/working/apache-svn-trunk-source/ql/target/classes [INFO] ------------------------------------------------------------- [WARNING] COMPILATION WARNING : [INFO] ------------------------------------------------------------- [WARNING] Note: Some input files use or override a deprecated API. [WARNING] Note: Recompile with -Xlint:deprecation for details. [WARNING] Note: Some input files use unchecked or unsafe operations. [WARNING] Note: Recompile with -Xlint:unchecked for details. [INFO] 4 warnings [INFO] ------------------------------------------------------------- [INFO] ------------------------------------------------------------- [ERROR] COMPILATION ERROR : [INFO] ------------------------------------------------------------- [ERROR] /data/hive-ptest/working/apache-svn-trunk-source/ql/src/java/org/apache/hadoop/hive/ql/io/orc/SnappyCodec.java:[21,48] cannot find symbol symbol : class DirectDecompressorShim location: interface org.apache.hadoop.hive.shims.HadoopShims [ERROR] /data/hive-ptest/working/apache-svn-trunk-source/ql/src/java/org/apache/hadoop/hive/ql/io/orc/SnappyCodec.java:[23,48] cannot find symbol symbol : class DirectCompressionType location: interface org.apache.hadoop.hive.shims.HadoopShims [ERROR] /data/hive-ptest/working/apache-svn-trunk-source/ql/src/java/org/apache/hadoop/hive/ql/io/orc/RecordReaderImpl.java:[51,48] cannot find symbol symbol : class ByteBufferPoolShim location: interface org.apache.hadoop.hive.shims.HadoopShims [ERROR] /data/hive-ptest/working/apache-svn-trunk-source/ql/src/java/org/apache/hadoop/hive/ql/io/orc/RecordReaderImpl.java:[52,48] cannot find symbol symbol : class ZeroCopyReaderShim location: interface org.apache.hadoop.hive.shims.HadoopShims [ERROR] /data/hive-ptest/working/apache-svn-trunk-source/ql/src/java/org/apache/hadoop/hive/ql/io/orc/RecordReaderImpl.java:[100,64] cannot find symbol symbol : class ByteBufferPoolShim location: class org.apache.hadoop.hive.ql.io.orc.RecordReaderImpl [ERROR] /data/hive-ptest/working/apache-svn-trunk-source/ql/src/java/org/apache/hadoop/hive/ql/io/orc/RecordReaderImpl.java:[96,17] cannot find symbol symbol : class ZeroCopyReaderShim location: class org.apache.hadoop.hive.ql.io.orc.RecordReaderImpl [ERROR] /data/hive-ptest/working/apache-svn-trunk-source/ql/src/java/org/apache/hadoop/hive/ql/io/orc/ZlibCodec.java:[27,48] cannot find symbol symbol : class DirectCompressionType location: interface org.apache.hadoop.hive.shims.HadoopShims [ERROR] /data/hive-ptest/working/apache-svn-trunk-source/ql/src/java/org/apache/hadoop/hive/ql/io/orc/ZlibCodec.java:[28,48] cannot find symbol symbol : class DirectDecompressorShim location: interface org.apache.hadoop.hive.shims.HadoopShims [ERROR] /data/hive-ptest/working/apache-svn-trunk-source/ql/src/java/org/apache/hadoop/hive/ql/io/orc/SnappyCodec.java:[82,11] cannot find symbol symbol : variable DirectCompressionType location: class org.apache.hadoop.hive.ql.io.orc.SnappyCodec [ERROR] /data/hive-ptest/working/apache-svn-trunk-source/ql/src/java/org/apache/hadoop/hive/ql/io/orc/SnappyCodec.java:[94,5] cannot find symbol symbol : class DirectDecompressorShim location: class org.apache.hadoop.hive.ql.io.orc.SnappyCodec [ERROR] /data/hive-ptest/working/apache-svn-trunk-source/ql/src/java/org/apache/hadoop/hive/ql/io/orc/SnappyCodec.java:[95,32] cannot find symbol symbol : variable DirectCompressionType location: class org.apache.hadoop.hive.ql.io.orc.SnappyCodec [ERROR] /data/hive-ptest/working/apache-svn-trunk-source/ql/src/java/org/apache/hadoop/hive/ql/io/orc/RecordReaderImpl.java:[153,5] method does not override or implement a method from a supertype [ERROR] /data/hive-ptest/working/apache-svn-trunk-source/ql/src/java/org/apache/hadoop/hive/ql/io/orc/RecordReaderImpl.java:[165,5] method does not override or implement a method from a supertype [ERROR] /data/hive-ptest/working/apache-svn-trunk-source/ql/src/java/org/apache/hadoop/hive/ql/io/orc/RecordReaderImpl.java:[228,45] cannot find symbol symbol : method getZeroCopyReader(org.apache.hadoop.fs.FSDataInputStream,org.apache.hadoop.hive.ql.io.orc.RecordReaderImpl.ByteBufferAllocatorPool) location: interface org.apache.hadoop.hive.shims.HadoopShims [ERROR] /data/hive-ptest/working/apache-svn-trunk-source/ql/src/java/org/apache/hadoop/hive/ql/io/orc/ZlibCodec.java:[95,11] cannot find symbol symbol : variable DirectCompressionType location: class org.apache.hadoop.hive.ql.io.orc.ZlibCodec [ERROR] /data/hive-ptest/working/apache-svn-trunk-source/ql/src/java/org/apache/hadoop/hive/ql/io/orc/ZlibCodec.java:[107,5] cannot find symbol symbol : class DirectDecompressorShim location: class org.apache.hadoop.hive.ql.io.orc.ZlibCodec [ERROR] /data/hive-ptest/working/apache-svn-trunk-source/ql/src/java/org/apache/hadoop/hive/ql/io/orc/ZlibCodec.java:[108,32] cannot find symbol symbol : variable DirectCompressionType location: class org.apache.hadoop.hive.ql.io.orc.ZlibCodec [INFO] 17 errors [INFO] ------------------------------------------------------------- [INFO] ------------------------------------------------------------------------ [INFO] Reactor Summary: [INFO] [INFO] Hive .............................................. SUCCESS [4.645s] [INFO] Hive Ant Utilities ................................ SUCCESS [6.787s] [INFO] Hive Shims Common ................................. SUCCESS [3.329s] [INFO] Hive Shims 0.20 ................................... SUCCESS [2.245s] [INFO] Hive Shims Secure Common .......................... SUCCESS [2.600s] [INFO] Hive Shims 0.20S .................................. SUCCESS [1.428s] [INFO] Hive Shims 0.23 ................................... SUCCESS [3.198s] [INFO] Hive Shims ........................................ SUCCESS [0.590s] [INFO] Hive Common ....................................... SUCCESS [7.734s] [INFO] Hive Serde ........................................ SUCCESS [8.743s] [INFO] Hive Metastore .................................... SUCCESS [26.982s] [INFO] Hive Query Language ............................... FAILURE [40.846s] [INFO] Hive Service ...................................... SKIPPED [INFO] Hive JDBC ......................................... SKIPPED [INFO] Hive Beeline ...................................... SKIPPED [INFO] Hive CLI .......................................... SKIPPED [INFO] Hive Contrib ...................................... SKIPPED [INFO] Hive HBase Handler ................................ SKIPPED [INFO] Hive HCatalog ..................................... SKIPPED [INFO] Hive HCatalog Core ................................ SKIPPED [INFO] Hive HCatalog Pig Adapter ......................... SKIPPED [INFO] Hive HCatalog Server Extensions ................... SKIPPED [INFO] Hive HCatalog Webhcat Java Client ................. SKIPPED [INFO] Hive HCatalog Webhcat ............................. SKIPPED [INFO] Hive HCatalog HBase Storage Handler ............... SKIPPED [INFO] Hive HWI .......................................... SKIPPED [INFO] Hive ODBC ......................................... SKIPPED [INFO] Hive Shims Aggregator ............................. SKIPPED [INFO] Hive TestUtils .................................... SKIPPED [INFO] Hive Packaging .................................... SKIPPED [INFO] ------------------------------------------------------------------------ [INFO] BUILD FAILURE [INFO] ------------------------------------------------------------------------ [INFO] Total time: 1:52.063s [INFO] Finished at: Sat Feb 01 08:39:12 EST 2014 [INFO] Final Memory: 55M/422M [INFO] ------------------------------------------------------------------------ [ERROR] Failed to execute goal org.apache.maven.plugins:maven-compiler-plugin:3.1:compile (default-compile) on project hive-exec: Compilation failure: Compilation failure: [ERROR] /data/hive-ptest/working/apache-svn-trunk-source/ql/src/java/org/apache/hadoop/hive/ql/io/orc/SnappyCodec.java:[21,48] cannot find symbol [ERROR] symbol : class DirectDecompressorShim [ERROR] location: interface org.apache.hadoop.hive.shims.HadoopShims [ERROR] /data/hive-ptest/working/apache-svn-trunk-source/ql/src/java/org/apache/hadoop/hive/ql/io/orc/SnappyCodec.java:[23,48] cannot find symbol [ERROR] symbol : class DirectCompressionType [ERROR] location: interface org.apache.hadoop.hive.shims.HadoopShims [ERROR] /data/hive-ptest/working/apache-svn-trunk-source/ql/src/java/org/apache/hadoop/hive/ql/io/orc/RecordReaderImpl.java:[51,48] cannot find symbol [ERROR] symbol : class ByteBufferPoolShim [ERROR] location: interface org.apache.hadoop.hive.shims.HadoopShims [ERROR] /data/hive-ptest/working/apache-svn-trunk-source/ql/src/java/org/apache/hadoop/hive/ql/io/orc/RecordReaderImpl.java:[52,48] cannot find symbol [ERROR] symbol : class ZeroCopyReaderShim [ERROR] location: interface org.apache.hadoop.hive.shims.HadoopShims [ERROR] /data/hive-ptest/working/apache-svn-trunk-source/ql/src/java/org/apache/hadoop/hive/ql/io/orc/RecordReaderImpl.java:[100,64] cannot find symbol [ERROR] symbol : class ByteBufferPoolShim [ERROR] location: class org.apache.hadoop.hive.ql.io.orc.RecordReaderImpl [ERROR] /data/hive-ptest/working/apache-svn-trunk-source/ql/src/java/org/apache/hadoop/hive/ql/io/orc/RecordReaderImpl.java:[96,17] cannot find symbol [ERROR] symbol : class ZeroCopyReaderShim [ERROR] location: class org.apache.hadoop.hive.ql.io.orc.RecordReaderImpl [ERROR] /data/hive-ptest/working/apache-svn-trunk-source/ql/src/java/org/apache/hadoop/hive/ql/io/orc/ZlibCodec.java:[27,48] cannot find symbol [ERROR] symbol : class DirectCompressionType [ERROR] location: interface org.apache.hadoop.hive.shims.HadoopShims [ERROR] /data/hive-ptest/working/apache-svn-trunk-source/ql/src/java/org/apache/hadoop/hive/ql/io/orc/ZlibCodec.java:[28,48] cannot find symbol [ERROR] symbol : class DirectDecompressorShim [ERROR] location: interface org.apache.hadoop.hive.shims.HadoopShims [ERROR] /data/hive-ptest/working/apache-svn-trunk-source/ql/src/java/org/apache/hadoop/hive/ql/io/orc/SnappyCodec.java:[82,11] cannot find symbol [ERROR] symbol : variable DirectCompressionType [ERROR] location: class org.apache.hadoop.hive.ql.io.orc.SnappyCodec [ERROR] /data/hive-ptest/working/apache-svn-trunk-source/ql/src/java/org/apache/hadoop/hive/ql/io/orc/SnappyCodec.java:[94,5] cannot find symbol [ERROR] symbol : class DirectDecompressorShim [ERROR] location: class org.apache.hadoop.hive.ql.io.orc.SnappyCodec [ERROR] /data/hive-ptest/working/apache-svn-trunk-source/ql/src/java/org/apache/hadoop/hive/ql/io/orc/SnappyCodec.java:[95,32] cannot find symbol [ERROR] symbol : variable DirectCompressionType [ERROR] location: class org.apache.hadoop.hive.ql.io.orc.SnappyCodec [ERROR] /data/hive-ptest/working/apache-svn-trunk-source/ql/src/java/org/apache/hadoop/hive/ql/io/orc/RecordReaderImpl.java:[153,5] method does not override or implement a method from a supertype [ERROR] /data/hive-ptest/working/apache-svn-trunk-source/ql/src/java/org/apache/hadoop/hive/ql/io/orc/RecordReaderImpl.java:[165,5] method does not override or implement a method from a supertype [ERROR] /data/hive-ptest/working/apache-svn-trunk-source/ql/src/java/org/apache/hadoop/hive/ql/io/orc/RecordReaderImpl.java:[228,45] cannot find symbol [ERROR] symbol : method getZeroCopyReader(org.apache.hadoop.fs.FSDataInputStream,org.apache.hadoop.hive.ql.io.orc.RecordReaderImpl.ByteBufferAllocatorPool) [ERROR] location: interface org.apache.hadoop.hive.shims.HadoopShims [ERROR] /data/hive-ptest/working/apache-svn-trunk-source/ql/src/java/org/apache/hadoop/hive/ql/io/orc/ZlibCodec.java:[95,11] cannot find symbol [ERROR] symbol : variable DirectCompressionType [ERROR] location: class org.apache.hadoop.hive.ql.io.orc.ZlibCodec [ERROR] /data/hive-ptest/working/apache-svn-trunk-source/ql/src/java/org/apache/hadoop/hive/ql/io/orc/ZlibCodec.java:[107,5] cannot find symbol [ERROR] symbol : class DirectDecompressorShim [ERROR] location: class org.apache.hadoop.hive.ql.io.orc.ZlibCodec [ERROR] /data/hive-ptest/working/apache-svn-trunk-source/ql/src/java/org/apache/hadoop/hive/ql/io/orc/ZlibCodec.java:[108,32] cannot find symbol [ERROR] symbol : variable DirectCompressionType [ERROR] location: class org.apache.hadoop.hive.ql.io.orc.ZlibCodec [ERROR] -> [Help 1] [ERROR] [ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch. [ERROR] Re-run Maven using the -X switch to enable full debug logging. [ERROR] [ERROR] For more information about the errors and possible solutions, please read the following articles: [ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoFailureException [ERROR] [ERROR] After correcting the problems, you can resume the build with the command [ERROR] mvn <goals> -rf :hive-exec + exit 1 ' This message is automatically generated. ATTACHMENT ID: 12626451
          Hide
          Brock Noland added a comment -

          There are a number of cases there the if contains a negation
          but there is an else condition. Thse should be swapped:

          e.g

          if (!bool) else => if (bool) else

          +      if(zcr != null) {
          +        while(len > 0) {
          +          ByteBuffer partial = zcr.readBuffer(len, false);
          +          result.add(new BufferChunk(partial, off));
          +          int read = partial.remaining();
          +          len -= read;
          +          off += read;
          +        }
          +      } else {
          
          +      if (ShimLoader.getHadoopShims().getDirectDecompressor(
          +          DirectCompressionType.SNAPPY) != null) {
          +        direct = Boolean.valueOf(true);
          +      } else {
          +        direct = Boolean.valueOf(false);
          +      }
          
          Show
          Brock Noland added a comment - There are a number of cases there the if contains a negation but there is an else condition. Thse should be swapped: e.g if (!bool) else => if (bool) else + if(zcr != null) { + while(len > 0) { + ByteBuffer partial = zcr.readBuffer(len, false); + result.add(new BufferChunk(partial, off)); + int read = partial.remaining(); + len -= read; + off += read; + } + } else { + if (ShimLoader.getHadoopShims().getDirectDecompressor( + DirectCompressionType.SNAPPY) != null) { + direct = Boolean.valueOf(true); + } else { + direct = Boolean.valueOf(false); + }
          Hide
          Gopal V added a comment -

          Lefty Leverenz: I will add the docs into the hive-site.xml in the next rev - was waiting for HIVE-6346 to get pushed to branch.

          Brock Noland: a != null is actually positive test, because it implies there exists something (i.e it's presence). That is how rest of the ORC code deals with optional feature/args (like SARGs).

          Show
          Gopal V added a comment - Lefty Leverenz : I will add the docs into the hive-site.xml in the next rev - was waiting for HIVE-6346 to get pushed to branch. Brock Noland : a != null is actually positive test, because it implies there exists something (i.e it's presence). That is how rest of the ORC code deals with optional feature/args (like SARGs).
          Hide
          Gopal V added a comment -

          Address comments on review board (& added docs)

          Show
          Gopal V added a comment - Address comments on review board (& added docs)
          Hide
          Gopal V added a comment -

          munmap() is async and delayed action

          Show
          Gopal V added a comment - munmap() is async and delayed action
          Hide
          Gunther Hagleitner added a comment -

          Committed to branch. Thanks Gopal V

          Show
          Gunther Hagleitner added a comment - Committed to branch. Thanks Gopal V

            People

            • Assignee:
              Gopal V
              Reporter:
              Gopal V
            • Votes:
              0 Vote for this issue
              Watchers:
              6 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development