[HUDI-528] Incremental Pull fails when latest commit is empty

• Type: Bug
• Status: Closed
• Priority: Minor
• Resolution: Fixed
• Fix Version/s: 0.6.0, 0.5.3
• Component/s: incremental-query
• Labels: bug-bash-0.6.0 help-requested pull-request-available
• Assignee: Yanjia Gary Li
• Reporter: Luis Vega
• Created: 13/Jan/20 21:47
• Updated: 17/May/20 21:59
• Resolved: 15/May/20 18:10

Description

When trying to create an incremental view of a dataset, an exception is thrown when the latest commit in the time range is empty. In order to determine the schema of the dataset, Hudi will grab the latest commit file, parse it, and grab the first metadata file path. If the latest commit was empty though, the field which is used to determine file paths (partitionToWriteStats) will be empty causing the following exception:

 

 

java.util.NoSuchElementException
  at java.util.HashMap$HashIterator.nextNode(HashMap.java:1447)
  at java.util.HashMap$ValueIterator.next(HashMap.java:1474)
  at org.apache.hudi.IncrementalRelation.<init>(IncrementalRelation.scala:80)
  at org.apache.hudi.DefaultSource.createRelation(DefaultSource.scala:65)
  at org.apache.hudi.DefaultSource.createRelation(DefaultSource.scala:46)
  at org.apache.spark.sql.execution.datasources.DataSource.resolveRelation(DataSource.scala:318)
  at org.apache.spark.sql.DataFrameReader.loadV1Source(DataFrameReader.scala:223)
  at org.apache.spark.sql.DataFrameReader.load(DataFrameReader.scala:211)
  at org.apache.spark.sql.DataFrameReader.load(DataFrameReader.scala:178)

Issue Links

• is depended upon by: HUDI-901 Bug Bash 0.6.0 Tracking Ticket
• links to: GitHub Pull Request #1612