[PARQUET-317] writeMetaDataFile crashes when a relative root Path is used

• Type: Bug
• Status: Resolved
• Priority: Minor
• Resolution: Fixed
• Affects Version/s: 1.8.0
• Fix Version/s: 1.8.0
• Component/s: parquet-mr
• Assignee: Steven She
• Reporter: Steven She
• Created: 24/Jun/15 18:21
• Updated: 23/Jun/24 03:27
• Resolved: 26/Jun/15 04:50

Description

In Spark, I can save an RDD to the local file system using a relative path, e.g.:

rdd.saveAsNewAPIHadoopFile(
        "relativeRoot",
        classOf[Void],
        tag.runtimeClass.asInstanceOf[Class[T]],
        classOf[ParquetOutputFormat[T]],
        job.getConfiguration)

This leads to a crash in the ParquetFileWriter.mergeFooters(..) method since the footer paths are read as fully qualified paths, but the root path is provided as a relative path:

org.apache.parquet.io.ParquetEncodingException: /Users/stevenshe/schema/relativeRoot/part-r-00000.snappy.parquet invalid: all the files must be contained in the root relativeRoot

Issue Links

• links to: PR #228