Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 0.8.0, 0.9.0
    • Fix Version/s: 0.8.0, 0.9.0
    • Component/s: None
    • Labels:
      None

      Description

      I ran a bellow query. This query throws IOException in worker

      tajo> create table orders_rc (
      o_orderkey      INT8,
      o_custkey       INT8,
      o_orderstatus   TEXT,
      o_totalprice    FLOAT8,
      o_orderdate     TEXT,
      o_orderpriority TEXT,
      o_clerk TEXT,
      o_shippriority  INT4,
      o_comment       TEXT)
      using rcfile WITH ('compression.codec'='org.apache.hadoop.io.compress.GzipCodec')
      
      tajo> insert overwrite into orders_rc select * from orders;
      
      2014-03-05 10:29:18,763 INFO  zlib.ZlibFactory (ZlibFactory.java:<clinit>(48)) - Successfully loaded & initialized native-zlib library
      2014-03-05 10:29:18,765 INFO  compress.CodecPool (CodecPool.java:getCompressor(108)) - Got brand-new compressor [.gz]
      2014-03-05 10:29:18,936 INFO  compress.CodecPool (CodecPool.java:getCompressor(108)) - Got brand-new compressor [.gz]
      2014-03-05 10:29:18,954 INFO  compress.CodecPool (CodecPool.java:getCompressor(108)) - Got brand-new compressor [.gz]
      2014-03-05 10:29:18,963 INFO  compress.CodecPool (CodecPool.java:getCompressor(108)) - Got brand-new compressor [.gz]
      2014-03-05 10:29:18,975 INFO  compress.CodecPool (CodecPool.java:getCompressor(108)) - Got brand-new compressor [.gz]
      2014-03-05 10:29:18,979 INFO  compress.CodecPool (CodecPool.java:getCompressor(108)) - Got brand-new compressor [.gz]
      2014-03-05 10:29:18,993 INFO  compress.CodecPool (CodecPool.java:getCompressor(108)) - Got brand-new compressor [.gz]
      2014-03-05 10:29:18,998 INFO  compress.CodecPool (CodecPool.java:getCompressor(108)) - Got brand-new compressor [.gz]
      2014-03-05 10:29:19,014 INFO  compress.CodecPool (CodecPool.java:getCompressor(108)) - Got brand-new compressor [.gz]
      2014-03-05 10:29:19,028 INFO  compress.CodecPool (CodecPool.java:getCompressor(108)) - Got brand-new compressor [.gz]
      2014-03-05 10:29:19,039 INFO  compress.CodecPool (CodecPool.java:getCompressor(108)) - Got brand-new compressor [.gz]
      2014-03-05 10:29:19,080 INFO  compress.CodecPool (CodecPool.java:getCompressor(108)) - Got brand-new compressor [.gz]
      2014-03-05 10:29:19,094 INFO  compress.CodecPool (CodecPool.java:getCompressor(108)) - Got brand-new compressor [.gz]
      2014-03-05 10:29:22,484 ERROR worker.Task (Task.java:run(383)) - java.io.IOException: write beyond end of stream
              at org.apache.hadoop.io.compress.CompressorStream.write(CompressorStream.java:66)
              at java.io.BufferedOutputStream.write(BufferedOutputStream.java:122)
              at java.io.DataOutputStream.write(DataOutputStream.java:107)
              at org.apache.tajo.storage.rcfile.NonSyncByteArrayOutputStream.writeTo(NonSyncByteArrayOutputStream.java:142)
              at org.apache.tajo.storage.rcfile.RCFile$RCFileAppender.WriteKeyBuffer(RCFile.java:947)
              at org.apache.tajo.storage.rcfile.RCFile$RCFileAppender.writeKey(RCFile.java:1046)
              at org.apache.tajo.storage.rcfile.RCFile$RCFileAppender.flushRecords(RCFile.java:1005)
              at org.apache.tajo.storage.rcfile.RCFile$RCFileAppender.append(RCFile.java:894)
              at org.apache.tajo.storage.rcfile.RCFile$RCFileAppender.addTuple(RCFile.java:851)
              at org.apache.tajo.engine.planner.physical.StoreTableExec.next(StoreTableExec.java:77)
              at org.apache.tajo.worker.Task.run(Task.java:374)
              at org.apache.tajo.worker.TaskRunner$1.run(TaskRunner.java:391)
              at java.lang.Thread.run(Thread.java:744)
      

        Activity

        Hide
        jhkim Jinho Kim added a comment -

        Created a review request against branch master in reviewboard
        https://reviews.apache.org/r/18761/

        Show
        jhkim Jinho Kim added a comment - Created a review request against branch master in reviewboard https://reviews.apache.org/r/18761/
        Hide
        jhkim Jinho Kim added a comment -

        Thanks Hyoungjun for nice finding.

        Show
        jhkim Jinho Kim added a comment - Thanks Hyoungjun for nice finding.
        Hide
        tajoqa Tajo QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12632769/TAJO-653.patch
        against master revision e0b6f7a.

        +1 @author. The patch does not contain any @author tags.

        -1 tests included. The patch doesn't appear to include any new or modified tests.
        Please justify why no new tests are needed for this patch.
        Also please list what manual steps were performed to verify this patch.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 javadoc. The applied patch does not increase the total number of javadoc warnings.

        +1 checkstyle. The patch generated 0 code style errors.

        +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        +1 core tests. The patch passed unit tests in tajo-storage.

        Test results: https://builds.apache.org/job/PreCommit-TAJO-Build/186//testReport/
        Console output: https://builds.apache.org/job/PreCommit-TAJO-Build/186//console

        This message is automatically generated.

        Show
        tajoqa Tajo QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12632769/TAJO-653.patch against master revision e0b6f7a. +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 javadoc. The applied patch does not increase the total number of javadoc warnings. +1 checkstyle. The patch generated 0 code style errors. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed unit tests in tajo-storage. Test results: https://builds.apache.org/job/PreCommit-TAJO-Build/186//testReport/ Console output: https://builds.apache.org/job/PreCommit-TAJO-Build/186//console This message is automatically generated.
        Hide
        hudson Hudson added a comment -

        SUCCESS: Integrated in Tajo-0.8.0-build #9 (See https://builds.apache.org/job/Tajo-0.8.0-build/9/)
        TAJO-653: RCFileAppender throws IOException. (jinho) (jinossy: https://git-wip-us.apache.org/repos/asf/?p=incubator-tajo.git&a=commit&h=f4aad2bf3e28dbde765bdab562157212a51d71ab)

        • tajo-storage/src/main/java/org/apache/tajo/storage/rcfile/RCFile.java
        • CHANGES.txt
        Show
        hudson Hudson added a comment - SUCCESS: Integrated in Tajo-0.8.0-build #9 (See https://builds.apache.org/job/Tajo-0.8.0-build/9/ ) TAJO-653 : RCFileAppender throws IOException. (jinho) (jinossy: https://git-wip-us.apache.org/repos/asf/?p=incubator-tajo.git&a=commit&h=f4aad2bf3e28dbde765bdab562157212a51d71ab ) tajo-storage/src/main/java/org/apache/tajo/storage/rcfile/RCFile.java CHANGES.txt
        Hide
        jhkim Jinho Kim added a comment -

        This issue got +1 from reviewboard.
        I've just committed it

        Show
        jhkim Jinho Kim added a comment - This issue got +1 from reviewboard. I've just committed it
        Hide
        hudson Hudson added a comment -

        SUCCESS: Integrated in Tajo-master-build #101 (See https://builds.apache.org/job/Tajo-master-build/101/)
        TAJO-653: RCFileAppender throws IOException. (jinho) (jinossy: https://git-wip-us.apache.org/repos/asf/?p=incubator-tajo.git&a=commit&h=7454fd87162af864738e29f0fef83897d9dd35fc)

        • tajo-storage/src/main/java/org/apache/tajo/storage/rcfile/RCFile.java
        • CHANGES.txt
        Show
        hudson Hudson added a comment - SUCCESS: Integrated in Tajo-master-build #101 (See https://builds.apache.org/job/Tajo-master-build/101/ ) TAJO-653 : RCFileAppender throws IOException. (jinho) (jinossy: https://git-wip-us.apache.org/repos/asf/?p=incubator-tajo.git&a=commit&h=7454fd87162af864738e29f0fef83897d9dd35fc ) tajo-storage/src/main/java/org/apache/tajo/storage/rcfile/RCFile.java CHANGES.txt

          People

          • Assignee:
            jhkim Jinho Kim
            Reporter:
            hjkim Hyoungjun Kim
          • Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development