Uploaded image for project: 'Parquet'
  1. Parquet
  2. PARQUET-317

writeMetaDataFile crashes when a relative root Path is used

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Minor
    • Resolution: Fixed
    • Affects Version/s: 1.8.0
    • Fix Version/s: 1.8.0
    • Component/s: parquet-mr
    • Labels:
      None

      Description

      In Spark, I can save an RDD to the local file system using a relative path, e.g.:

      rdd.saveAsNewAPIHadoopFile(
              "relativeRoot",
              classOf[Void],
              tag.runtimeClass.asInstanceOf[Class[T]],
              classOf[ParquetOutputFormat[T]],
              job.getConfiguration)
      

      This leads to a crash in the ParquetFileWriter.mergeFooters(..) method since the footer paths are read as fully qualified paths, but the root path is provided as a relative path:

      org.apache.parquet.io.ParquetEncodingException: /Users/stevenshe/schema/relativeRoot/part-r-00000.snappy.parquet invalid: all the files must be contained in the root relativeRoot
      

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                stevenshe Steven She
                Reporter:
                stevenshe Steven She
              • Votes:
                0 Vote for this issue
                Watchers:
                2 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: