Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 0.11.0
    • Fix Version/s: 0.12.0, 0.11.1
    • Component/s: Storage
    • Labels:
      None

      Description

      If value are null, projection of parquet does not work.

      parquet.io.ParquetDecodingException: Can not read value at 0 in block 0 in file file:/Users/jhkim/tajo/jinossy/tajo4/target/test-data/TestStorages/testProjection.data
      	at org.apache.tajo.storage.thirdparty.parquet.InternalParquetRecordReader.nextKeyValue(InternalParquetRecordReader.java:184)
      	at org.apache.tajo.storage.thirdparty.parquet.ParquetReader.read(ParquetReader.java:115)
      	at org.apache.tajo.storage.thirdparty.parquet.ParquetReader.read(ParquetReader.java:119)
      	at org.apache.tajo.storage.parquet.ParquetScanner.next(ParquetScanner.java:73)
      	at org.apache.tajo.storage.TestStorages.testProjection(TestStorages.java:350)
      	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
      	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
      	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
      	at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
      	at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
      	at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
      	at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
      	at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
      	at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
      	at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
      	at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
      	at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
      	at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
      	at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
      	at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
      	at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
      	at org.junit.runners.Suite.runChild(Suite.java:128)
      	at org.junit.runners.Suite.runChild(Suite.java:27)
      	at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
      	at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
      	at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
      	at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
      	at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
      	at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
      	at org.junit.runner.JUnitCore.run(JUnitCore.java:137)
      	at com.intellij.junit4.JUnit4IdeaTestRunner.startRunnerWithArgs(JUnit4IdeaTestRunner.java:78)
      	at com.intellij.rt.execution.junit.JUnitStarter.prepareStreamsAndStart(JUnitStarter.java:212)
      	at com.intellij.rt.execution.junit.JUnitStarter.main(JUnitStarter.java:68)
      	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
      	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
      	at com.intellij.rt.execution.application.AppMain.main(AppMain.java:140)
      Caused by: java.lang.ArrayIndexOutOfBoundsException: 2
      	at org.apache.tajo.storage.VTuple.put(VTuple.java:87)
      	at org.apache.tajo.storage.parquet.TajoRecordConverter.set(TajoRecordConverter.java:92)
      	at org.apache.tajo.storage.parquet.TajoRecordConverter.end(TajoRecordConverter.java:162)
      	at parquet.io.RecordReaderImplementation.read(RecordReaderImplementation.java:413)
      	at org.apache.tajo.storage.thirdparty.parquet.InternalParquetRecordReader.nextKeyValue(InternalParquetRecordReader.java:180)
      	... 38 more
      

        Activity

        Hide
        githubbot ASF GitHub Bot added a comment -

        GitHub user jinossy opened a pull request:

        https://github.com/apache/tajo/pull/903

        TAJO-2010: Parquet can not read null value

        You can merge this pull request into a Git repository by running:

        $ git pull https://github.com/jinossy/tajo TAJO-2010

        Alternatively you can review and apply these changes as the patch at:

        https://github.com/apache/tajo/pull/903.patch

        To close this pull request, make a commit to your master/trunk branch
        with (at least) the following in the commit message:

        This closes #903


        commit a673cdff2e04ffb35a8eed069e88fa9b93051045
        Author: Jinho Kim <jhkim@apache.org>
        Date: 2015-12-03T13:54:13Z

        TAJO-2010: Parquet can not read null value


        Show
        githubbot ASF GitHub Bot added a comment - GitHub user jinossy opened a pull request: https://github.com/apache/tajo/pull/903 TAJO-2010 : Parquet can not read null value You can merge this pull request into a Git repository by running: $ git pull https://github.com/jinossy/tajo TAJO-2010 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/tajo/pull/903.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #903 commit a673cdff2e04ffb35a8eed069e88fa9b93051045 Author: Jinho Kim <jhkim@apache.org> Date: 2015-12-03T13:54:13Z TAJO-2010 : Parquet can not read null value
        Hide
        githubbot ASF GitHub Bot added a comment -

        Github user blrunner commented on a diff in the pull request:

        https://github.com/apache/tajo/pull/903#discussion_r46641693

        — Diff: tajo-storage/tajo-storage-hdfs/src/main/java/org/apache/tajo/storage/parquet/TajoRecordConverter.java —
        @@ -154,14 +157,6 @@ public void start() {
        */
        @Override
        public void end() {

        • for (int i = 0; i < projectionMap.length; ++i) {
            • End diff –

        Could you explain why you remove existing codes?

        Show
        githubbot ASF GitHub Bot added a comment - Github user blrunner commented on a diff in the pull request: https://github.com/apache/tajo/pull/903#discussion_r46641693 — Diff: tajo-storage/tajo-storage-hdfs/src/main/java/org/apache/tajo/storage/parquet/TajoRecordConverter.java — @@ -154,14 +157,6 @@ public void start() { */ @Override public void end() { for (int i = 0; i < projectionMap.length; ++i) { End diff – Could you explain why you remove existing codes?
        Hide
        githubbot ASF GitHub Bot added a comment -

        Github user jinossy commented on a diff in the pull request:

        https://github.com/apache/tajo/pull/903#discussion_r46641819

        — Diff: tajo-storage/tajo-storage-hdfs/src/main/java/org/apache/tajo/storage/parquet/TajoRecordConverter.java —
        @@ -154,14 +157,6 @@ public void start() {
        */
        @Override
        public void end() {

        • for (int i = 0; i < projectionMap.length; ++i) {
            • End diff –

        You can find it in start()

        Show
        githubbot ASF GitHub Bot added a comment - Github user jinossy commented on a diff in the pull request: https://github.com/apache/tajo/pull/903#discussion_r46641819 — Diff: tajo-storage/tajo-storage-hdfs/src/main/java/org/apache/tajo/storage/parquet/TajoRecordConverter.java — @@ -154,14 +157,6 @@ public void start() { */ @Override public void end() { for (int i = 0; i < projectionMap.length; ++i) { End diff – You can find it in start()
        Hide
        githubbot ASF GitHub Bot added a comment -

        Github user blrunner commented on the pull request:

        https://github.com/apache/tajo/pull/903#issuecomment-161852331

        +1

        Ship it.

        Show
        githubbot ASF GitHub Bot added a comment - Github user blrunner commented on the pull request: https://github.com/apache/tajo/pull/903#issuecomment-161852331 +1 Ship it.
        Hide
        githubbot ASF GitHub Bot added a comment -

        Github user jinossy commented on the pull request:

        https://github.com/apache/tajo/pull/903#issuecomment-161854671

        Thanks for your quick review!

        Show
        githubbot ASF GitHub Bot added a comment - Github user jinossy commented on the pull request: https://github.com/apache/tajo/pull/903#issuecomment-161854671 Thanks for your quick review!
        Hide
        githubbot ASF GitHub Bot added a comment -

        Github user asfgit closed the pull request at:

        https://github.com/apache/tajo/pull/903

        Show
        githubbot ASF GitHub Bot added a comment - Github user asfgit closed the pull request at: https://github.com/apache/tajo/pull/903
        Hide
        jhkim Jinho Kim added a comment -

        committed it.
        Thanks

        Show
        jhkim Jinho Kim added a comment - committed it. Thanks
        Hide
        hudson Hudson added a comment -

        FAILURE: Integrated in Tajo-master-CODEGEN-build #618 (See https://builds.apache.org/job/Tajo-master-CODEGEN-build/618/)
        TAJO-2010: Parquet can not read null value. (jhkim: rev 80218d03dffbb16eaaf8eb8b82010c3e98e617d5)

        • tajo-storage/tajo-storage-hdfs/src/test/java/org/apache/tajo/storage/TestStorages.java
        • CHANGES
        • tajo-storage/tajo-storage-hdfs/src/main/java/org/apache/tajo/storage/parquet/TajoRecordMaterializer.java
        • tajo-storage/tajo-storage-hdfs/src/main/java/org/apache/tajo/storage/parquet/TajoRecordConverter.java
        Show
        hudson Hudson added a comment - FAILURE: Integrated in Tajo-master-CODEGEN-build #618 (See https://builds.apache.org/job/Tajo-master-CODEGEN-build/618/ ) TAJO-2010 : Parquet can not read null value. (jhkim: rev 80218d03dffbb16eaaf8eb8b82010c3e98e617d5) tajo-storage/tajo-storage-hdfs/src/test/java/org/apache/tajo/storage/TestStorages.java CHANGES tajo-storage/tajo-storage-hdfs/src/main/java/org/apache/tajo/storage/parquet/TajoRecordMaterializer.java tajo-storage/tajo-storage-hdfs/src/main/java/org/apache/tajo/storage/parquet/TajoRecordConverter.java
        Hide
        hudson Hudson added a comment -

        FAILURE: Integrated in Tajo-0.11.1-build #127 (See https://builds.apache.org/job/Tajo-0.11.1-build/127/)
        TAJO-2010: Parquet can not read null value. (jhkim: rev f2e4773f3b85beba8e39f76d6cf362f72e3a0ddc)

        • tajo-storage/tajo-storage-hdfs/src/main/java/org/apache/tajo/storage/parquet/TajoRecordConverter.java
        • tajo-storage/tajo-storage-hdfs/src/test/java/org/apache/tajo/storage/TestStorages.java
        • CHANGES
        • tajo-storage/tajo-storage-hdfs/src/main/java/org/apache/tajo/storage/parquet/TajoRecordMaterializer.java
        Show
        hudson Hudson added a comment - FAILURE: Integrated in Tajo-0.11.1-build #127 (See https://builds.apache.org/job/Tajo-0.11.1-build/127/ ) TAJO-2010 : Parquet can not read null value. (jhkim: rev f2e4773f3b85beba8e39f76d6cf362f72e3a0ddc) tajo-storage/tajo-storage-hdfs/src/main/java/org/apache/tajo/storage/parquet/TajoRecordConverter.java tajo-storage/tajo-storage-hdfs/src/test/java/org/apache/tajo/storage/TestStorages.java CHANGES tajo-storage/tajo-storage-hdfs/src/main/java/org/apache/tajo/storage/parquet/TajoRecordMaterializer.java
        Hide
        hudson Hudson added a comment -

        SUCCESS: Integrated in Tajo-master-build #1010 (See https://builds.apache.org/job/Tajo-master-build/1010/)
        TAJO-2010: Parquet can not read null value. (jhkim: rev 80218d03dffbb16eaaf8eb8b82010c3e98e617d5)

        • tajo-storage/tajo-storage-hdfs/src/main/java/org/apache/tajo/storage/parquet/TajoRecordMaterializer.java
        • tajo-storage/tajo-storage-hdfs/src/main/java/org/apache/tajo/storage/parquet/TajoRecordConverter.java
        • tajo-storage/tajo-storage-hdfs/src/test/java/org/apache/tajo/storage/TestStorages.java
        • CHANGES
        Show
        hudson Hudson added a comment - SUCCESS: Integrated in Tajo-master-build #1010 (See https://builds.apache.org/job/Tajo-master-build/1010/ ) TAJO-2010 : Parquet can not read null value. (jhkim: rev 80218d03dffbb16eaaf8eb8b82010c3e98e617d5) tajo-storage/tajo-storage-hdfs/src/main/java/org/apache/tajo/storage/parquet/TajoRecordMaterializer.java tajo-storage/tajo-storage-hdfs/src/main/java/org/apache/tajo/storage/parquet/TajoRecordConverter.java tajo-storage/tajo-storage-hdfs/src/test/java/org/apache/tajo/storage/TestStorages.java CHANGES
        Hide
        hudson Hudson added a comment -

        SUCCESS: Integrated in Tajo-0.11.1-build #128 (See https://builds.apache.org/job/Tajo-0.11.1-build/128/)
        TAJO-2010: Parquet can not read null value. (fix method signature) (jhkim: rev 8cdd21f7bb712a56fbe0e33a24dd4ba13980c701)

        • tajo-storage/tajo-storage-hdfs/src/test/java/org/apache/tajo/storage/TestStorages.java
        Show
        hudson Hudson added a comment - SUCCESS: Integrated in Tajo-0.11.1-build #128 (See https://builds.apache.org/job/Tajo-0.11.1-build/128/ ) TAJO-2010 : Parquet can not read null value. (fix method signature) (jhkim: rev 8cdd21f7bb712a56fbe0e33a24dd4ba13980c701) tajo-storage/tajo-storage-hdfs/src/test/java/org/apache/tajo/storage/TestStorages.java

          People

          • Assignee:
            jhkim Jinho Kim
            Reporter:
            jhkim Jinho Kim
          • Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development