[ARROW-9020] [Python] read_json won't respect explicit_schema in parse_options

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 0.17.1
• Fix Version/s: 1.0.0
• Component/s: Python
• Labels: pull-request-available
• Environment: CPython 3.8.2, MacOS Mojave 10.14.6
• External issue URL: https://github.com/apache/arrow/issues/25137
• Assignee: Krisztian Szucs
• Reporter: Felipe Santos
• Created: 02/Jun/20 22:46
• Updated: 11/Jan/23 08:04
• Resolved: 09/Jul/20 06:41

Description

I am trying to read a json file using an explicit schema but it looks like the schema is ignored. Moreover, if the my schema contains a field not present in the json file, then the output table contains all the fields in the json file plus the fields of my schema not found in the file.

A minimal example:

import pyarrow as pa
from pyarrow import json

# allowing for type inference
print(json.read_json('tmp.json'))
# prints:
# pyarrow.Table
# foo: string
# baz: string

# using an explicit schema that would read only "foo"
schema = pa.schema([('foo', pa.string())])
print(json.read_json('tmp.json', parse_options=json.ParseOptions(explicit_schema=schema)))
# prints:
# pyarrow.Table
# foo: string
# baz: string

# using an explicit schema that would read only "not_a_field",
# which is not present in the json file
schema = pa.schema([('not_a_field', pa.string())])
print(json.read_json('tmp.json', parse_options=json.ParseOptions(explicit_schema=schema)))
# prints:
# pyarrow.Table
# not_a_field: string
# foo: string
# baz: string

And the tmp.json file looks like:

{"foo": "bar", "baz": "1"}

Issue Links

• links to: GitHub Pull Request #7661