[ARROW-10663] [C++/Doc] The IsIn kernel ignores the skip_nulls option of SetLookupOptions

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 3.0.0
• Component/s: C++
• Labels: pull-request-available
• External issue URL: https://github.com/apache/arrow/issues/26618
• Assignee: Antoine Pitrou
• Reporter: Joris Van den Bossche
• Created: 20/Nov/20 10:44
• Updated: 11/Jan/23 08:14
• Resolved: 12/Jan/21 17:50

Description

The C++ docs of SetLookupOptions has this explanation of the skip_nulls option:

  /// Whether nulls in `value_set` count for lookup.
  ///
  /// If true, any null in `value_set` is ignored and nulls in the input
  /// produce null (IndexIn) or false (IsIn) values in the output.
  /// If false, any null in `value_set` is successfully matched in
  /// the input.
  bool skip_nulls;

(from https://github.com/apache/arrow/blob/8b9f6b9d28b4524724e60fac589fb1a3552a32b4/cpp/src/arrow/compute/api_scalar.h#L78-L84)

However, for IsIn this explanation doesn't seem to hold in practice:

In [16]: arr = pa.array([1, 2, None])

In [17]: pc.is_in(arr, value_set=pa.array([1, None]), skip_null=True)
Out[17]: 
<pyarrow.lib.BooleanArray object at 0x7fcf666f9408>
[
  true,
  false,
  true
]

In [18]: pc.is_in(arr, value_set=pa.array([1, None]), skip_null=False)
Out[18]: 
<pyarrow.lib.BooleanArray object at 0x7fcf666b13a8>
[
  true,
  false,
  true
]

This documentation was added in https://github.com/apache/arrow/pull/7695 (ARROW-8989)/
.

BTW, for "index_in", it works as documented:

In [19]: pc.index_in(arr, value_set=pa.array([1, None]), skip_null=True)
Out[19]: 
<pyarrow.lib.Int32Array object at 0x7fcf666f04c8>
[
  0,
  null,
  null
]

In [20]: pc.index_in(arr, value_set=pa.array([1, None]), skip_null=False)
Out[20]: 
<pyarrow.lib.Int32Array object at 0x7fcf666f0ee8>
[
  0,
  null,
  1
]

Issue Links

• links to: GitHub Pull Request #9164