Uploaded image for project: 'Apache Arrow'
  1. Apache Arrow
  2. ARROW-9035

[C++] Writing IPC messages with 64-byte buffer alignment vs. 8-byte default

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Open
    • Priority: Minor
    • Resolution: Unresolved
    • Affects Version/s: 0.17.0
    • Fix Version/s: None
    • Component/s: C++, Documentation
    • Labels:
      None

      Description

      I used the C++ library to create a very small arrow file (1 field of 5 int32) and was surprised that the buffers are not aligned to 64 bytes as per the documentation section "Buffer Alignment and Padding" with examples.. based on the examples there, the 20 bytes of int32 should be padded to 64 bytes, but it is only 24 (see below) .   

      extract message metadata - see BodyLength = 24

      {
        version: "V4",
        header_type: "RecordBatch",
        header: {
          nodes: [
            {
              length: 5,
              null_count: 0
            }
          ],
          buffers: [
            {
              offset: 0,
              length: 0
            },
            {
              offset: 0,
              length: 20
            }
          ]
        },
        bodyLength: 24
      } 

      Reading further down the documentation section "Encapsulated message format" it says serialization should use 8 byte alignment. 

      These both seem at odds with each other and some clarification is needed.

      Is the documentation wrong? 

      Or

      Should 8 byte alignment be used for File and 64 byte for IPC ?

        Attachments

          Activity

            People

            • Assignee:
              Unassigned
              Reporter:
              abbot Anthony Abate
            • Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

              Dates

              • Created:
                Updated: