[PARQUET-1317] ParquetMetadataConverter throw NPE

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 1.11.0
• Fix Version/s: 1.11.0
• Component/s: parquet-mr
• Assignee: Yuming Wang
• Reporter: Yuming Wang
• Created: 04/Jun/18 06:46
• Updated: 04/Jun/18 16:58
• Resolved: 04/Jun/18 16:58

Description

How to reproduce:

$ bin/spark-shell 
scala> spark.range(10).selectExpr("cast(id as string) as id").coalesce(1).write.parquet("/tmp/parquet-1317")
scala> 

java -jar ./parquet-tools/target/parquet-tools-1.10.1-SNAPSHOT.jar head --debug file:///tmp/parquet-1317/part-00000-6cfafbdd-fdeb-4861-8499-8583852ba437-c000.snappy.parquet

java.io.IOException: Could not read footer: java.lang.NullPointerException

at org.apache.parquet.hadoop.ParquetFileReader.readAllFootersInParallel(ParquetFileReader.java:271)

at org.apache.parquet.hadoop.ParquetFileReader.readAllFootersInParallelUsingSummaryFiles(ParquetFileReader.java:202)

at org.apache.parquet.hadoop.ParquetFileReader.readFooters(ParquetFileReader.java:354)

at org.apache.parquet.tools.command.RowCountCommand.execute(RowCountCommand.java:88)

at org.apache.parquet.tools.Main.main(Main.java:223)

Caused by: java.lang.NullPointerException

at org.apache.parquet.format.converter.ParquetMetadataConverter.getOriginalType(ParquetMetadataConverter.java:828)

at org.apache.parquet.format.converter.ParquetMetadataConverter.buildChildren(ParquetMetadataConverter.java:1173)

at org.apache.parquet.format.converter.ParquetMetadataConverter.fromParquetSchema(ParquetMetadataConverter.java:1124)

at org.apache.parquet.format.converter.ParquetMetadataConverter.fromParquetMetadata(ParquetMetadataConverter.java:1058)

at org.apache.parquet.format.converter.ParquetMetadataConverter.readParquetMetadata(ParquetMetadataConverter.java:1052)

at org.apache.parquet.hadoop.ParquetFileReader.readFooter(ParquetFileReader.java:532)

at org.apache.parquet.hadoop.ParquetFileReader.readFooter(ParquetFileReader.java:505)

at org.apache.parquet.hadoop.ParquetFileReader.readFooter(ParquetFileReader.java:499)

at org.apache.parquet.hadoop.ParquetFileReader.readFooter(ParquetFileReader.java:476)

at org.apache.parquet.hadoop.ParquetFileReader$2.call(ParquetFileReader.java:261)

at org.apache.parquet.hadoop.ParquetFileReader$2.call(ParquetFileReader.java:257)

at java.util.concurrent.FutureTask.run(FutureTask.java:266)

at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)

at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)

at java.lang.Thread.run(Thread.java:748)

java.io.IOException: Could not read footer: java.lang.NullPointerException

Issue Links

• links to: GitHub Pull Request #489
• links to: GitHub Pull Request #491