[BIGTOP-764] Bigtop mapreduce job fails in ubuntu precise - Invalid shuffle port number

• Type: Bug
• Status: Resolved
• Priority: Minor
• Resolution: Cannot Reproduce
• Affects Version/s: 0.4.0, 0.6.0
• Component/s: deployment, vm
• Labels: newbie patch

Environment

Ubuntu Precise 12.04 LTS 64 bit running on VMWare Fusion

• Assignee: Sean Mackrory
• Reporter: P Mohan
• Created: 22/Oct/12 03:18
• Updated: 28/May/14 23:55
• Resolved: 28/May/14 14:59

Description

i installed bigtop 0.4.0 following the instructions in the wiki. Installation was fine. i ran into "Invalid shuffle port number" when i tried to run the pi calculation example. I did a initial check on the internet and found - https://issues.apache.org/jira/browse/MAPREDUCE-2983 . but this did not help.

did anyone face this error before ?

hadoop jar /usr/lib/hadoop-mapreduce/hadoop-mapreduce-examples*.jar pi 10 1000

error message:
12/10/21 19:26:03 INFO mapreduce.Job: map 0% reduce 0%
12/10/21 19:26:04 INFO mapreduce.Job: Task Id : attempt_1350872346733_0001_m_000004_0, Status : FAILED
Container launch failed for container_1350872346733_0001_01_000006 : java.lang.IllegalStateException: Invalid shuffle port number -1 returned for attempt_1350872346733_0001_m_000004_0
at org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl$Container.launch(ContainerLauncherImpl.java:162)
at org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl$EventProcessor.run(ContainerLauncherImpl.java:373)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
at java.lang.Thread.run(Thread.java:722)

12/10/21 19:26:04 WARN mapreduce.Job: Error reading task output Connection refused

I am attaching the installation instructions i followed below. Any help would be highly appreciated.

installation instructions

#Environment : Ubuntu 12.04 LTS 64 bit running on VMWare fusion

#Install JAVA
sudo add-apt-repository ppa:webupd8team/java
sudo apt-get update
sudo apt-get install oracle-java7-installer

#Set Java Home
JAVA_HOME=/usr/lib/jvm/java-7-oracle/
export JAVA_HOME

1. add big top repos
   wget O http://www.apache.org/dist/incubator/bigtop/bigtop-0.4.0-incubating/repos/GPG-KEY-bigtop | sudo apt-key add -

sudo wget -O /etc/apt/sources.list.d/bigtop.list http://bigtop01.cloudera.org:8080/view/Releases/job/Bigtop-0.4.0/label=precise/lastSuccessfulBuild/artifact/output/bigtop.list

sudo apt-get update
apt-cache search hadoop

#install hadoop
sudo apt-get install hadoop* flume-* mahout* oozie* whirr-*

#initialize namenode - format
sudo /etc/init.d/hadoop-hdfs-namenode init
for i in hadoop-hdfs-namenode hadoop-hdfs-datanode ; do sudo service $i start ; done

#setup folders necessary for hadoop operarions
sudo -u hdfs hadoop fs -mkdir /user/$USER
sudo -u hdfs hadoop fs -chown $USER:$USER /user/$USER

sudo -u hdfs hadoop fs -mkdir /tmp
sudo -u hdfs hadoop fs -chmod -R 1777 /tmp

sudo -u hdfs hadoop fs -mkdir /var/log/hadoop-yarn
sudo -u hdfs hadoop fs -chown yarn:mapred /var/log/hadoop-yarn

sudo -u hdfs hadoop fs -mkdir /user/history
sudo -u hdfs hadoop fs -chown mapred:mapred /user/history
sudo -u hdfs hadoop fs -chmod 770 /user/history

sudo -u hdfs hadoop fs -mkdir /tmp/hadoop-yarn/staging
sudo -u hdfs hadoop fs -chmod -R 1777 /tmp/hadoop-yarn/staging

sudo -u hdfs hadoop fs -mkdir /tmp/hadoop-yarn/staging/history/done_intermediate
sudo -u hdfs hadoop fs -chmod -R 1777 /tmp/hadoop-yarn/staging/history/done_intermediate
sudo -u hdfs hadoop fs -chown -R mapred:mapred /tmp/hadoop-yarn/staging

sudo service hadoop-yarn-resourcemanager start
sudo service hadoop-yarn-nodemanager start

#test
hadoop fs -lsr /