contrib/data_join doesn't work

• Key: HADOOP-3526
• Type: Bug
• Status: Resolved
• Resolution: Fixed
• Priority: Blocker
• Assignee: Spyros Blanas
• Reporter: Spyros Blanas
• Affects Version/s: 0.17.0
• Fix Version/s: 0.17.1

Description

The example in the README.txt in contrig/data_join/src/example doesn't work in 0.17.0. It works perfectly in 0.16.4.

Comments

[Spyros Blanas]

The issue is related to HADOOP-3522. The patch just clones the objects returned from the ValueIterator and forces the user-provided classes to implement a clone() method as well.

[Runping Qi]

Since applications using this package break with 0.17,
I think this is a blocker for 0.17.1

[Chris Douglas]

Spyros-

This looks like the right idea; a couple suggestions:

• It would break less existing code if this used WritableUtils::clone instead of adding an abstract method
• The tag at DataJoinReducerBase::107 doesn't need to be cloned

[Nigel Daley]

-1 as the patch should have a unit test.

[Spyros Blanas]

Chris,
The tag needs cloning, because it's stored as a key in the SortedMap. Otherwise, the Text.equals() method always returns true (see HADOOP-3522).
As for the WritableUtils.clone() suggestion, I upload a new version of the patch which incorporates it.

Nigel,
Unfortunately there is no existing JUnit test covering the contributed data_join code and I don't know how to change the build.xml file to run the new tests. Also, building a JUnit testing framework for generic map-reduce jobs like data_join is much more difficult than targeting specific functions, as done in the core. I would be happy to create a test if you can point me to some code that tests a generic map/reduce job that I can tweak.

[Nigel Daley]

Spyros, Chris has agreed to write some unit tests for Hadoop 0.18.

[Chris Douglas]

The tag needs cloning, because it's stored as a key in the SortedMap. Otherwise, the Text.equals() method always returns true

Sorry, I mean it doesn't need to be cloned there, for the lookup. Since the record needs to be cloned anyway and it includes the tag, it should be sufficient to clone the record and use the tag for the map key.

[Chris Douglas]

I just committed this. Thanks, Spyros

[Hudson]

Integrated in Hadoop-trunk #523 (See http://hudson.zones.apache.org/hudson/job/Hadoop-trunk/523/)