[SPARK-4365] Remove unnecessary filter call on records returned from parquet library

• Type: Improvement
• Status: Resolved
• Priority: Minor
• Resolution: Fixed
• Affects Version/s: 1.1.0
• Fix Version/s: 1.2.0
• Component/s: SQL
• Assignee: Unassigned
• Reporter: Yash Datta
• Created: 12/Nov/14 14:11
• Updated: 14/Nov/14 23:17
• Resolved: 14/Nov/14 23:17

Description

Since parquet library has been updated , we no longer need to filter the records returned from parquet library for null records , as now the library skips those :

from parquet-hadoop/src/main/java/parquet/hadoop/InternalParquetRecordReader.java

public boolean nextKeyValue() throws IOException, InterruptedException {
boolean recordFound = false;

while (!recordFound) {
// no more records left
if (current >= total)

{ return false; }

try {
checkRead();
currentValue = recordReader.read();
current ++;
if (recordReader.shouldSkipCurrentRecord())

{ // this record is being filtered via the filter2 package if (DEBUG) LOG.debug("skipping record"); continue; }

if (currentValue == null)

{ // only happens with FilteredRecordReader at end of block current = totalCountLoadedSoFar; if (DEBUG) LOG.debug("filtered record reader reached end of block"); continue; }

recordFound = true;

if (DEBUG) LOG.debug("read value: " + currentValue);
} catch (RuntimeException e)

{ throw new ParquetDecodingException(format("Can not read value at %d in block %d in file %s", current, currentBlock, file), e); }

}
return true;
}

Issue Links

• links to: [Github] Pull Request #3229 (saucam)