[SPARK-6581] Metadata is missing when saving parquet file using hadoop 1.0.4

• Type: Bug
• Status: Resolved
• Priority: Critical
• Resolution: Won't Fix
• Affects Version/s: 1.3.0
• Component/s: SQL

Environment

hadoop 1.0.4

• Assignee: Unassigned
• Reporter: Pei-Lun Lee
• Created: 28/Mar/15 02:51
• Updated: 28/Jan/16 13:22
• Resolved: 28/Jan/16 13:22

Description

When saving parquet file with

df.save("foo", "parquet")

It generates only _common_data while _metadata is missing:

-rwxrwxrwx  1 peilunlee  staff    0 Mar 27 11:29 _SUCCESS*
-rwxrwxrwx  1 peilunlee  staff  250 Mar 27 11:29 _common_metadata*
-rwxrwxrwx  1 peilunlee  staff  272 Mar 27 11:29 part-r-00001.parquet*
-rwxrwxrwx  1 peilunlee  staff  272 Mar 27 11:29 part-r-00002.parquet*
-rwxrwxrwx  1 peilunlee  staff  272 Mar 27 11:29 part-r-00003.parquet*
-rwxrwxrwx  1 peilunlee  staff  488 Mar 27 11:29 part-r-00004.parquet*

If saving with

df.save("foo", "parquet", SaveMode.Overwrite)

Both _metadata and _common_metadata are missing:

-rwxrwxrwx  1 peilunlee  staff    0 Mar 27 11:29 _SUCCESS*
-rwxrwxrwx  1 peilunlee  staff  272 Mar 27 11:29 part-r-00001.parquet*
-rwxrwxrwx  1 peilunlee  staff  272 Mar 27 11:29 part-r-00002.parquet*
-rwxrwxrwx  1 peilunlee  staff  272 Mar 27 11:29 part-r-00003.parquet*
-rwxrwxrwx  1 peilunlee  staff  488 Mar 27 11:29 part-r-00004.parquet*

Issue Links

• is related to: SPARK-6579 save as parquet with overwrite failed when linking with Hadoop 1.0.4