[TAJO-653] RCFileAppender throws IOException

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 0.8.0, 0.9.0
• Fix Version/s: 0.8.0, 0.9.0
• Assignee: Jinho Kim
• Reporter: Hyoungjun Kim
• Created: 05/Mar/14 01:35
• Updated: 05/Mar/14 05:35
• Resolved: 05/Mar/14 05:28

Description

I ran a bellow query. This query throws IOException in worker

tajo> create table orders_rc (
o_orderkey      INT8,
o_custkey       INT8,
o_orderstatus   TEXT,
o_totalprice    FLOAT8,
o_orderdate     TEXT,
o_orderpriority TEXT,
o_clerk TEXT,
o_shippriority  INT4,
o_comment       TEXT)
using rcfile WITH ('compression.codec'='org.apache.hadoop.io.compress.GzipCodec')

tajo> insert overwrite into orders_rc select * from orders;

2014-03-05 10:29:18,763 INFO  zlib.ZlibFactory (ZlibFactory.java:<clinit>(48)) - Successfully loaded & initialized native-zlib library
2014-03-05 10:29:18,765 INFO  compress.CodecPool (CodecPool.java:getCompressor(108)) - Got brand-new compressor [.gz]
2014-03-05 10:29:18,936 INFO  compress.CodecPool (CodecPool.java:getCompressor(108)) - Got brand-new compressor [.gz]
2014-03-05 10:29:18,954 INFO  compress.CodecPool (CodecPool.java:getCompressor(108)) - Got brand-new compressor [.gz]
2014-03-05 10:29:18,963 INFO  compress.CodecPool (CodecPool.java:getCompressor(108)) - Got brand-new compressor [.gz]
2014-03-05 10:29:18,975 INFO  compress.CodecPool (CodecPool.java:getCompressor(108)) - Got brand-new compressor [.gz]
2014-03-05 10:29:18,979 INFO  compress.CodecPool (CodecPool.java:getCompressor(108)) - Got brand-new compressor [.gz]
2014-03-05 10:29:18,993 INFO  compress.CodecPool (CodecPool.java:getCompressor(108)) - Got brand-new compressor [.gz]
2014-03-05 10:29:18,998 INFO  compress.CodecPool (CodecPool.java:getCompressor(108)) - Got brand-new compressor [.gz]
2014-03-05 10:29:19,014 INFO  compress.CodecPool (CodecPool.java:getCompressor(108)) - Got brand-new compressor [.gz]
2014-03-05 10:29:19,028 INFO  compress.CodecPool (CodecPool.java:getCompressor(108)) - Got brand-new compressor [.gz]
2014-03-05 10:29:19,039 INFO  compress.CodecPool (CodecPool.java:getCompressor(108)) - Got brand-new compressor [.gz]
2014-03-05 10:29:19,080 INFO  compress.CodecPool (CodecPool.java:getCompressor(108)) - Got brand-new compressor [.gz]
2014-03-05 10:29:19,094 INFO  compress.CodecPool (CodecPool.java:getCompressor(108)) - Got brand-new compressor [.gz]
2014-03-05 10:29:22,484 ERROR worker.Task (Task.java:run(383)) - java.io.IOException: write beyond end of stream
        at org.apache.hadoop.io.compress.CompressorStream.write(CompressorStream.java:66)
        at java.io.BufferedOutputStream.write(BufferedOutputStream.java:122)
        at java.io.DataOutputStream.write(DataOutputStream.java:107)
        at org.apache.tajo.storage.rcfile.NonSyncByteArrayOutputStream.writeTo(NonSyncByteArrayOutputStream.java:142)
        at org.apache.tajo.storage.rcfile.RCFile$RCFileAppender.WriteKeyBuffer(RCFile.java:947)
        at org.apache.tajo.storage.rcfile.RCFile$RCFileAppender.writeKey(RCFile.java:1046)
        at org.apache.tajo.storage.rcfile.RCFile$RCFileAppender.flushRecords(RCFile.java:1005)
        at org.apache.tajo.storage.rcfile.RCFile$RCFileAppender.append(RCFile.java:894)
        at org.apache.tajo.storage.rcfile.RCFile$RCFileAppender.addTuple(RCFile.java:851)
        at org.apache.tajo.engine.planner.physical.StoreTableExec.next(StoreTableExec.java:77)
        at org.apache.tajo.worker.Task.run(Task.java:374)
        at org.apache.tajo.worker.TaskRunner$1.run(TaskRunner.java:391)
        at java.lang.Thread.run(Thread.java:744)