[ORC-1288] [C++] Invalid memory freeing with ZLIB compression - ASF JIRA

Attach files

Attach Screenshot

Voters

Watch issue

Watchers

Create sub-task

Link

Clone

Update Comment Author

Replace String in Comment

Update Comment Visibility

Delete Comments

XML

Word

Printable

JSON

Details

Type: Bug
Status: Resolved
Priority: Major
Resolution: Fixed
Affects Version/s: 1.8.0
Fix Version/s: 1.8.1
Component/s: None
Labels:
None

Language:
- C++

Description

The simple example code ends with a segfault/munmap_chunk(): invalid pointer:

#include "orc/Common.hh"
#include "orc/OrcFile.hh"

using namespace orc;

int main(void) {
    ORC_UNIQUE_PTR<OutputStream> outStream = writeLocalFile("test_file.orc");
    ORC_UNIQUE_PTR<Type> schema(Type::buildTypeFromString("struct<c0:int>"));
    WriterOptions options;
    options.setCompression(orc::CompressionKind_ZLIB);
    options.setStripeSize(4096);
    options.setCompressionBlockSize(4096);
    ORC_UNIQUE_PTR<Writer> writer = createWriter(*schema, outStream.get(), options);
    uint64_t batchSize = 65535, rowCount = 10000000;
    ORC_UNIQUE_PTR<ColumnVectorBatch> batch = writer->createRowBatch(batchSize);
    StructVectorBatch *root = dynamic_cast<StructVectorBatch *>(batch.get());
    LongVectorBatch *c0 = dynamic_cast<LongVectorBatch *>(root->fields[0]);
    uint64_t rows = 0;
    
    for (uint64_t i = 0; i < rowCount; ++i) {
        c0->data[rows] = i;
        rows++;
        if (rows == batchSize) {
            root->numElements = rows;
            c0->numElements = rows;
            writer->add(*batch);
            rows = 0;
        }
    }
    if (rows != 0) {
        root->numElements = rows;
        c0->numElements = rows;
        writer->add(*batch);
        rows = 0;
    }
    writer->close();
    return 0;
}

The bug depends on the stripe size, compression size, and the record number written to the file as well. I wasn't able to reproduce the error with other compression strategies than ZLIB.

It looks like to me that it's related to ORC-1130 somehow, but I couldn't comprehend how. (Reverting that modification on the main branch solved the issue).