[HIVE-13217] Replication for HoS mapjoin small file needs to respect dfs.replication.max

• Type: Bug
• Status: Closed
• Priority: Minor
• Resolution: Fixed
• Affects Version/s: 1.2.1, 2.0.0
• Fix Version/s: 2.1.0
• Component/s: Spark
• Assignee: Chinna Rao Lalam
• Reporter: Szehon Ho
• Created: 07/Mar/16 18:50
• Updated: 21/Jun/16 15:43
• Resolved: 24/Mar/16 08:39

Description

Currently Hive on Spark Mapjoin replicates small table file to a hard-coded value of 10. See SparkHashTableSinkOperator.MIN_REPLICATION.

When dfs.replication.max is less than 10, HoS query fails. This constant should cap at dfs.replication.max.

Normally dfs.replication.max seems set at 512.