[ARROW-11257] [C++][Parquet] PyArrow Table contains different data after writing and reloading from Parquet

• Type: Bug
• Status: Closed
• Priority: Critical
• Resolution: Duplicate
• Affects Version/s: 2.0.0
• Fix Version/s: 3.0.0
• Component/s: Python
• External issue URL: https://github.com/apache/arrow/issues/27158
• Assignee: Unassigned
• Reporter: Kari Schoonbee
• Created: 15/Jan/21 07:42
• Updated: 11/Jan/23 08:18
• Resolved: 19/Jan/21 20:39

Description

• I'm loading a JSONlines object into a table using 

  pa.json.readjson

  It contains one column that is a nested dictionary.

• I select a row by key and inspect its nested dictionary.
• I write the table to parquet 
• I load the table again from the parquet file 
• I check the same key and the nested dictionary is not the same.

 

To reproduce:

 

Find the attached JSONLines file and Jupyter Notebook.

The json file contains entries per customer with a generated `msisdn`, `scoring_request_id` and `scorecard_result` object. Each `scorecard result consists of a list of feature objects, all with the value the same as the msidn` and a score.

The notebook reads the file and demonstrates the issue.

 

Issue Links

• is duplicated by: ARROW-10493 [C++][Parquet] Writing nullable nested strings results in wrong data in file