[SPARK-21019] read orc when some of the columns are missing in some files

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Duplicate
• Affects Version/s: 2.1.1
• Component/s: Input/Output
• Assignee: Unassigned
• Reporter: Modi Tamam
• Created: 08/Jun/17 10:27
• Updated: 12/Dec/22 18:10
• Resolved: 28/Jun/17 04:21

Description

I'm using Spark-2.1.1.

I'm experiencing an issue when I'm reading a bunch of ORC files when some of the fields are missing from some of the files (file-1 has fields 'a' and 'b', file-2 has fields 'a' and 'c'). When I'm running the same flow with JSON files format, every thing is just fine (you can see it at the code snippets , if you'll run it...) My question is whether it's a bug or an expected behavior?

I'v pushed a maven project, ready for run, you can find it here https://github.com/MordechaiTamam/spark-orc-issue.

Issue Links

• duplicates: SPARK-11412 Support merge schema for ORC