Uploaded image for project: 'Apache Arrow'
  1. Apache Arrow
  2. ARROW-3065

[Python] concat_tables() failing from bad Pandas Metadata

    Details

      Description

      Looks like the major bug from https://issues.apache.org/jira/browse/ARROW-1941 is back...

      After I downgraded from 0.10.0 to 0.9.0, the error disappeared..

      new_arrow_table = pa.concat_tables(my_arrow_tables)
      
       File "pyarrow/table.pxi", line 1562, in pyarrow.lib.concat_tables
        File "pyarrow/error.pxi", line 81, in pyarrow.lib.check_status
      pyarrow.lib.ArrowInvalid: Schema at index 2 was different:
      

      In order to debug this I saved the first 4 arrow tables to 4 parquet files and inspected the parquet files. The parquet schema is identical, but the Pandas Metadata is different.

      for i in range(5):
           pq.write_table(my_arrow_tables[i], "test" + str(i) + ".parquet")
      

      It looks like a column which contains empty strings is getting typed as float64.

      >>> test1.schema
      HoldingDetail_Id: string
      metadata
      --------
      {b'pandas': b'{"index_columns": [], "column_indexes": [], "columns": [
      {"name": "HoldingDetail_Id", "field_name": "HoldingDetail_Id", "pandas_type": "unicode", "numpy_type": "object", "metadata": null},
      
      >>> test1[0]
      <Column name='HoldingDetail_Id' type=DataType(string)>
      [
        [
          "Z4",
          "SF",
          "J7",
          "W6",
          "L7",
          "Q9",
          "NE",
          "F7",
      
      
      >>> test2.schema
      HoldingDetail_Id: string
      metadata
      --------
      {b'pandas': b'{"index_columns": [], "column_indexes": [], "columns": [
      {"name": "HoldingDetail_Id", "field_name": "HoldingDetail_Id", "pandas_type": "unicode", "numpy_type": "float64", "metadata": null},
      
      >>> test2[0]
      <Column name='HoldingDetail_Id' type=DataType(string)>
      [
        [
          "",
          "",
          "",
          "",
          "",
          "",
          "",
          "",
      

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                kszucs Krisztian Szucs
                Reporter:
                davlee1972@yahoo.com David Lee
              • Votes:
                0 Vote for this issue
                Watchers:
                3 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved:

                  Time Tracking

                  Estimated:
                  Original Estimate - Not Specified
                  Not Specified
                  Remaining:
                  Remaining Estimate - 0h
                  0h
                  Logged:
                  Time Spent - 0.5h
                  0.5h