[HADOOP-895] Reduce hang on small set of SequenceFiles (reproducible)

• Type: Bug
• Status: Closed
• Priority: Blocker
• Resolution: Cannot Reproduce
• Affects Version/s: 0.10.1

Environment

This happens both on a mac and on a Linux x86_64 machine.

• Assignee: Unassigned
• Reporter: Doug Judd
• Created: 16/Jan/07 23:51
• Updated: 08/Jul/09 16:51
• Resolved: 23/Jul/07 23:01

Description

I'm running into a problem where a Map/Reduce job hangs in the reduce phase. It happens on a simple program to dump the keys of a small set of SequenceFiles. It can be reproduced as follows:

1. Pull a vanilla hadoop-0.10.1 release and untar it
2. Apply the attached patch
3. Modify conf/hadoop-env.sh to have the correct JAVA_HOME
4. Run the following script:

#!/bin/sh

ant zvents
./bin/hadoop namenode -format
mkdir fs-hadoop/dfs/tmp
mkdir fs-hadoop/dfs/data
mkdir -p fs-hadoop/mapred/local
./bin/start-all.sh
sleep 5
./bin/hadoop dfs -mkdir /heritrix
./bin/hadoop dfs -mkdir /heritrix/crawls
./bin/hadoop dfs -mkdir /heritrix/crawls/test-4-20070116061502956
./bin/hadoop dfs -copyFromLocal data/IAH-20070116061532-00000-doug-judds-computer.local /heritrix/crawls/test-4-20070116061502956
./bin/hadoop dfs -copyFromLocal data/IAH-20070116061539-00001-doug-judds-computer.local /heritrix/crawls/test-4-20070116061502956
./bin/hadoop dfs -copyFromLocal data/IAH-20070116061539-00002-doug-judds-computer.local /heritrix/crawls/test-4-20070116061502956
./bin/hadoop com.zvents.hadoop.DumpUrls /heritrix/crawls/test-4-20070116061502956 /foo

• Doug