[ARROW-2136] [Python] Non-nullable schema fields not checked in conversions from pandas

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 0.8.0
• Fix Version/s: 0.14.0
• Component/s: Python
• Labels: pull-request-available
• External issue URL: https://github.com/apache/arrow/issues/18104
• Assignee: Wes McKinney
• Reporter: Matthew Gilbert
• Created: 12/Feb/18 14:04
• Updated: 11/Jan/23 07:19
• Resolved: 25/Jun/19 11:42

Description

If you provide a schema with nullable=False but pass a DataFrame which in fact has nulls it appears the schema is ignored? I would expect an error here.

import pyarrow as pa
import pandas as pd

df = pd.DataFrame({"a":[1.2, 2.1, pd.np.NaN]})
schema = pa.schema([pa.field("a", pa.float64(), nullable=False)])
table = pa.Table.from_pandas(df, schema=schema)
table[0]

<pyarrow.lib.Column object at 0x7f213bf2fb70>
chunk 0: <pyarrow.lib.DoubleArray object at 0x7f213bf20ea8>
[
  1.2,
  2.1,
  NA
]

Issue Links

• links to: GitHub Pull Request #4683