[SPARK-22534] Add integration test case to explicitly verify optional validity buffer

• Type: Test
• Status: Closed
• Priority: Major
• Resolution: Not A Problem
• Affects Version/s: 0.8.0
• Component/s: Tests
• Assignee: Unassigned
• Reporter: Bryan Cutler
• Created: 15/Nov/17 23:35
• Updated: 15/Nov/17 23:37
• Resolved: 15/Nov/17 23:36

Description

When the null count of a Field is 0 the validity buffer is optional in a RecordBatch message. From ARROW-1476, integration tests were failing because of this (see discussion here https://github.com/apache/arrow/pull/1316#issuecomment-344439955). The failure occurred due to chance of the random seed for generating the validity buffer. We should explicitly test this case for integration testing in the future.