Uploaded image for project: 'Apache Arrow'
  1. Apache Arrow
  2. ARROW-1309

[Python] Error inferring List type in Array.from_pandas when inner values are all None

Attach filesAttach ScreenshotVotersWatch issueWatchersCreate sub-taskLinkCloneUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

Details

    • Bug
    • Status: Resolved
    • Minor
    • Resolution: Fixed
    • None
    • 0.6.0
    • Python
    • None
    • centos 7.3

    Description

      I have an avro file in hdfs that I am reading in using fastavro, converting to a pandas dataframe and then trying to create an arrow table and get as error:

      >>> table=pyarrow.Table.from_pandas(my_dataframe)
      Traceback (most recent call last):
      File "<stdin>", line 1, in <module>
      File "pyarrow/table.pxi", line 746, in pyarrow.lib.Table.from_pandas (/arrow/python/build/temp.linux-x86_64-3.6/lib.cxx:34089)
      File "pyarrow/table.pxi", line 346, in pyarrow.lib._dataframe_to_arrays (/arrow/python/build/temp.linux-x86_64-3.6/lib.cxx:30476)
      File "pyarrow/array.pxi", line 182, in pyarrow.lib.Array.from_pandas (/arrow/python/build/temp.linux-x86_64-3.6/lib.cxx:22110)
      File "pyarrow/error.pxi", line 66, in pyarrow.lib.check_status (/arrow/python/build/temp.linux-x86_64-3.6/lib.cxx:7702)
      pyarrow.lib.ArrowNotImplementedError: NotImplemented: null

      The avro schema indeed has null fields possible. Is this not implemented? I am using pyarrow 0.5.0. Also, for what I am doing I am not using pandas at all, I just read in the avro and I have a list of dicts and really want to write them to disk in parquet format and am utilizing these steps (which isn't optimal but may be necessary without writing more code of my own).

      thanks,
      Luke

      Attachments

        Activity

          This comment will be Viewable by All Users Viewable by All Users
          Cancel

          People

            wesm Wes McKinney
            virtualluke Luke Higgins
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Slack

                Issue deployment