[CASSANDRA-5544] Hadoop jobs assigns only one mapper in task

• Type: Bug
• Status: Resolved
• Priority: Normal
• Resolution: Fixed
• Fix Version/s: 1.2.6

Environment

Red hat linux 5.4, Hadoop 1.0.3, pig 0.11.1

• Severity: Normal
• Assignee: Alex Liu
• Reporter: Shamim Ahmed

Authors

Alex Liu

Reviewers

Brandon Williams

• Created: 07/May/13 07:00
• Updated: 16/Apr/19 09:32
• Resolved: 29/May/13 17:55

Description

We have got very strange beheviour of hadoop cluster after upgrading
Cassandra from 1.1.5 to Cassandra 1.2.1. We have 5 nodes cluster of Cassandra, where three of them are hodoop slaves. Now when we are submitting job through Pig script, only one map assigns in task running on one of the hadoop slaves regardless of
volume of data (already tried with more than million rows).
Configure of pig as follows:
export PIG_HOME=/oracle/pig-0.10.0
export PIG_CONF_DIR=${HADOOP_HOME}/conf
export PIG_INITIAL_ADDRESS=192.168.157.103
export PIG_RPC_PORT=9160
export PIG_PARTITIONER=org.apache.cassandra.dht.Murmur3Partitioner

Also we have these following properties in hadoop:
<property>
<name>mapred.tasktracker.map.tasks.maximum</name>
<value>10</value>
</property>
<property>
<name>mapred.map.tasks</name>
<value>4</value>
</property>

Issue Links

• relates to: CASSANDRA-5604 Vnodes decrease Hadoop performances cause it creates too many small splits