Uploaded image for project: 'Apache Arrow'
  1. Apache Arrow
  2. ARROW-7569

[Python] Add API to map Arrow types to pandas ExtensionDtypes for to_pandas conversions

Attach filesAttach ScreenshotVotersWatch issueWatchersCreate sub-taskLinkCloneUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

Details

    Description

      ARROW-2428 was about adding such a mapping, and described three use cases (see this comment for details):

      • Basic roundtrip based on the pandas_metadata (in to_pandas, we check if the pandas_metadata specify pandas extension dtypes, and if so, use this as the target dtype for that column)
      • Conversion for pyarrow extension types that can define their equivalent pandas extension dtype
      • A way to override default conversion (eg for the built-in types, or in absence of pandas_metadata in the schema). This would require the user to be able to specify some mapping of pyarrow type or column name to the pandas extension dtype to use.

      The PR that closed ARROW-2428 (https://github.com/apache/arrow/pull/5512) only covered the first two cases, and not the third case.

      I think it is still interesting to also cover the third case in some way.

      An example use case are the new nullable dtypes that are introduced in pandas (eg the nullable integer dtype). Assume I want to read a parquet file into a pandas DataFrame using this nullable integer dtype. The pyarrow Table has no pandas_metadata indicating to use this dtype (unless it was created from a pandas DataFrame that was already using this dtype, but that will often not be the case), and the pyarrow.int64() type is also not an extension type that can define its equivalent pandas extension dtype.
      Currently, the only solution is first read it into pandas DataFrame (which will use floats for the integers if there are nulls), and then afterwards to convert those floats back to a nullable integer dtype.

      A possible API for this could look like:

      table.to_pandas(types_mapping={pa.int64(): pd.Int64Dtype()})
      

      to indicate that you want to convert all columns of the pyarrow table with int64 type to a pandas column using the nullable Int64 dtype.

      Attachments

        Issue Links

        Activity

          This comment will be Viewable by All Users Viewable by All Users
          Cancel

          People

            jorisvandenbossche Joris Van den Bossche
            jorisvandenbossche Joris Van den Bossche
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Time Tracking

                Estimated:
                Original Estimate - Not Specified
                Not Specified
                Remaining:
                Remaining Estimate - 0h
                0h
                Logged:
                Time Spent - 1h 20m
                1h 20m

                Slack

                  Issue deployment