Uploaded image for project: 'Apache Arrow'
  1. Apache Arrow
  2. ARROW-2790

[C++] Buffers contain uninitialized memory

    Details

    • Type: New Feature
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 0.9.0
    • Fix Version/s: 0.10.0
    • Component/s: C++

      Description

      Hi,

      currently, buffers can contain uninitalized memory when built with buffer builders. This happens in two ways:

      1. The most common implementation of AppendNull(s) adds a null-bit to the null-map and increments the lengths, leaving chunks of data values untouched
      2. The padding is never initialized

      This can lead to a number of problems, some of more or less theoretical nature, but it's worth fixing just to make valgrind happy again and allow it to run in CI  Also, leaking uninitialized memory over a wire is never nice, and currently problem 1. ist never mitigated by the ipc writers (although the problem 2 ist). 

      This PR changes the behaviour of AppendNull(s) and FinishInternal for the affected builders and adds a lot of checks to ensure the correct initialization. Most of the test succeed the current master, but cause valgrind warnings, if the buffer builders aren't patched.

      I decided not to test explicitly if the nulled is really 0, just that it can be read without errors, since the specs don't specify exact values. I check the padded memory values explicitly, but I'm not completely sure, what the best course of action is there: we could also ignore the values in padding just checking if they are initialized, or we could adjust the spec in this point. 

        Attachments

          Activity

            People

            • Assignee:
              alendit Dimitri Vorona
              Reporter:
              alendit Dimitri Vorona
            • Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Time Tracking

                Estimated:
                Original Estimate - Not Specified
                Not Specified
                Remaining:
                Remaining Estimate - 0h
                0h
                Logged:
                Time Spent - 13h 10m
                13h 10m