[HDFS-2492] BlockManager cross-rack replication checks only work for ScriptBasedMapping

• Type: Bug
• Status: Open
• Priority: Minor
• Resolution: Unresolved
• Affects Version/s: 1.0.1, 1.0.2, 2.0.0-alpha, 3.0.0-alpha1
• Assignee: Steve Loughran
• Reporter: Steve Loughran
• Created: 22/Oct/11 15:48
• Updated: 12/May/16 18:13

Description

The BlockManager cross-rack replication checks only works if script files are used for replication, not if alternate plugins provide the topology information.

This is because the BlockManager sets its rack checking flag if there is a filename key

shouldCheckForEnoughRacks = conf.get(DFSConfigKeys.NET_TOPOLOGY_SCRIPT_FILE_NAME_KEY) != null;

yet this filename key is only used if the topology mapper defined by

DFSConfigKeys.NET_TOPOLOGY_NODE_SWITCH_MAPPING_IMPL_KEY

is an instance of ScriptBasedMapping

If any other mapper is used, the system may be multi rack, but the Block Manager will not be aware of this fact unless the filename key is set to something non-null

Issue Links

• depends upon: HADOOP-8046 Revert StaticMapping semantics to the existing ones, add DNS mapping diagnostics in progress
• depends upon: HADOOP-7878 Regression HADOOP-7777 switch changes break HDFS tests when the isSingleSwitch() predicate is used
• depends upon: HADOOP-8630 rename isSingleSwitch() methods in new topo base class to isFlatTopology()
• depends upon: HADOOP-7777 Implement a base class for DNSToSwitchMapping implementations that can offer extra topology information
• depends upon: HADOOP-7772 javadoc the topology classes
• is depended upon by: HADOOP-8091 Add a topology mapper that reads hostname to rack mappings from a Java properties file
• is depended upon by: HDFS-2472 Extend UnderReplicatedBlocks queue to give blocks whose existing copies are all on a single rack priority over multi-rack blocks
• is related to: HADOOP-8231 Make topologies easier to set up and debug