[CRUNCH-310] There should be a way to specify projection schema for Parquet files

• Type: Improvement
• Status: Closed
• Priority: Critical
• Resolution: Fixed
• Fix Version/s: 0.9.0, 0.8.2
• Component/s: IO
• Assignee: Unassigned
• Reporter: Alex Kozlov
• Created: 09/Dec/13 23:51
• Updated: 19/Dec/13 20:14
• Resolved: 11/Dec/13 20:13

Description

Currently the projection schema is set based on the ptype:

 private static <S> FormatBundle<AvroParquetInputFormat> getBundle(AvroType<S> ptype) {
    return FormatBundle.forInput(AvroParquetInputFormat.class)
        .set(AvroReadSupport.AVRO_REQUESTED_PROJECTION, ptype.getSchema().toString())
        // ParquetRecordReader expects ParquetInputSplits, not FileSplits, so it
        // doesn't work with CombineFileInputFormat
        .set(RuntimeParameters.DISABLE_COMBINE_FILE, "true");
  }

Sometimes a user wants a subset of columns as a projection. Need a mechanism to supply desired projection schema.