Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-2195

Parquet extraMetadata can contain key information

Attach filesAttach ScreenshotVotersWatch issueWatchersCreate sub-taskLinkCloneUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Blocker
    • Resolution: Fixed
    • Affects Version/s: 1.0.0
    • Fix Version/s: 1.0.1, 1.1.0
    • Component/s: SQL
    • Labels:
      None
    • Target Version/s:

      Description

      14/06/19 01:52:05 INFO NewHadoopRDD: Input split: ParquetInputSplit{part: file:/Users/pat/Projects/spark-summit-training-2014/usb/data/wiki-parquet/part-r-1.parquet start: 0 length: 24971040 hosts: [localhost] blocks: 1 requestedSchema: same as file fileSchema: message root {
      
        optional int32 id;
      
        optional binary title;
      
        optional int64 modified;
      
        optional binary text;
      
        optional binary username;
      
      }
      
       extraMetadata: {org.apache.spark.sql.parquet.row.metadata=StructType(List(StructField(id,IntegerType,true), StructField(title,StringType,true), StructField(modified,LongType,true), StructField(text,StringType,true), StructField(username,StringType,true))), path=************************ MY AWS KEYS!!! ************************} readSupportMetadata: {org.apache.spark.sql.parquet.row.metadata=StructType(List(StructField(id,IntegerType,true), StructField(title,StringType,true), StructField(modified,LongType,true), StructField(text,StringType,true), StructField(username,StringType,true))), path=**************************** MY AWS KEYS!!!! ***********************************}}
      

        Attachments

          Activity

            People

            • Assignee:
              Unassigned
              Reporter:
              marmbrus Michael Armbrust

              Dates

              • Created:
                Updated:
                Resolved:

                Issue deployment