Details

    • Type: Bug Bug
    • Status: Resolved
    • Priority: Blocker Blocker
    • Resolution: Fixed
    • Affects Version/s: 0.17.0
    • Fix Version/s: 0.17.1
    • Component/s: None
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      The example in the README.txt in contrig/data_join/src/example doesn't work in 0.17.0. It works perfectly in 0.16.4.

      1. patch
        1 kB
        Spyros Blanas

        Issue Links

          Activity

          Hide
          Spyros Blanas added a comment -

          The issue is related to HADOOP-3522. The patch just clones the objects returned from the ValueIterator and forces the user-provided classes to implement a clone() method as well.

          Show
          Spyros Blanas added a comment - The issue is related to HADOOP-3522 . The patch just clones the objects returned from the ValueIterator and forces the user-provided classes to implement a clone() method as well.
          Hide
          Runping Qi added a comment -

          Since applications using this package break with 0.17,
          I think this is a blocker for 0.17.1

          Show
          Runping Qi added a comment - Since applications using this package break with 0.17, I think this is a blocker for 0.17.1
          Hide
          Chris Douglas added a comment -

          Spyros-

          This looks like the right idea; a couple suggestions:

          • It would break less existing code if this used WritableUtils::clone instead of adding an abstract method
          • The tag at DataJoinReducerBase::107 doesn't need to be cloned
          Show
          Chris Douglas added a comment - Spyros- This looks like the right idea; a couple suggestions: It would break less existing code if this used WritableUtils::clone instead of adding an abstract method The tag at DataJoinReducerBase::107 doesn't need to be cloned
          Hide
          Nigel Daley added a comment -

          -1 as the patch should have a unit test.

          Show
          Nigel Daley added a comment - -1 as the patch should have a unit test.
          Hide
          Spyros Blanas added a comment -

          Chris,
          The tag needs cloning, because it's stored as a key in the SortedMap. Otherwise, the Text.equals() method always returns true (see HADOOP-3522).
          As for the WritableUtils.clone() suggestion, I upload a new version of the patch which incorporates it.

          Nigel,
          Unfortunately there is no existing JUnit test covering the contributed data_join code and I don't know how to change the build.xml file to run the new tests. Also, building a JUnit testing framework for generic map-reduce jobs like data_join is much more difficult than targeting specific functions, as done in the core. I would be happy to create a test if you can point me to some code that tests a generic map/reduce job that I can tweak.

          Show
          Spyros Blanas added a comment - Chris, The tag needs cloning, because it's stored as a key in the SortedMap. Otherwise, the Text.equals() method always returns true (see HADOOP-3522 ). As for the WritableUtils.clone() suggestion, I upload a new version of the patch which incorporates it. Nigel, Unfortunately there is no existing JUnit test covering the contributed data_join code and I don't know how to change the build.xml file to run the new tests. Also, building a JUnit testing framework for generic map-reduce jobs like data_join is much more difficult than targeting specific functions, as done in the core. I would be happy to create a test if you can point me to some code that tests a generic map/reduce job that I can tweak.
          Hide
          Nigel Daley added a comment -

          Spyros, Chris has agreed to write some unit tests for Hadoop 0.18.

          Show
          Nigel Daley added a comment - Spyros, Chris has agreed to write some unit tests for Hadoop 0.18.
          Hide
          Chris Douglas added a comment -

          The tag needs cloning, because it's stored as a key in the SortedMap. Otherwise, the Text.equals() method always returns true

          Sorry, I mean it doesn't need to be cloned there, for the lookup. Since the record needs to be cloned anyway and it includes the tag, it should be sufficient to clone the record and use the tag for the map key.

          Show
          Chris Douglas added a comment - The tag needs cloning, because it's stored as a key in the SortedMap. Otherwise, the Text.equals() method always returns true Sorry, I mean it doesn't need to be cloned there, for the lookup. Since the record needs to be cloned anyway and it includes the tag, it should be sufficient to clone the record and use the tag for the map key.
          Hide
          Chris Douglas added a comment -

          I just committed this. Thanks, Spyros

          Show
          Chris Douglas added a comment - I just committed this. Thanks, Spyros
          Hide
          Hudson added a comment -
          Show
          Hudson added a comment - Integrated in Hadoop-trunk #523 (See http://hudson.zones.apache.org/hudson/job/Hadoop-trunk/523/ )

            People

            • Assignee:
              Spyros Blanas
              Reporter:
              Spyros Blanas
            • Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development