Uploaded image for project: 'Apache Arrow'
  1. Apache Arrow
  2. ARROW-2136

[Python] Non-nullable schema fields not checked in conversions from pandas

    XMLWordPrintableJSON

    Details

      Description

      If you provide a schema with nullable=False but pass a DataFrame which in fact has nulls it appears the schema is ignored? I would expect an error here.

      import pyarrow as pa
      import pandas as pd
      
      df = pd.DataFrame({"a":[1.2, 2.1, pd.np.NaN]})
      schema = pa.schema([pa.field("a", pa.float64(), nullable=False)])
      table = pa.Table.from_pandas(df, schema=schema)
      table[0]
      
      <pyarrow.lib.Column object at 0x7f213bf2fb70>
      chunk 0: <pyarrow.lib.DoubleArray object at 0x7f213bf20ea8>
      [
        1.2,
        2.1,
        NA
      ]
      

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                wesm Wes McKinney
                Reporter:
                matthewgilbert Matthew Gilbert
              • Votes:
                0 Vote for this issue
                Watchers:
                4 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved:

                  Time Tracking

                  Estimated:
                  Original Estimate - Not Specified
                  Not Specified
                  Remaining:
                  Remaining Estimate - 0h
                  0h
                  Logged:
                  Time Spent - 50m
                  50m