Details

    • Type: Bug Bug
    • Status: Resolved
    • Priority: Blocker Blocker
    • Resolution: Fixed
    • Affects Version/s: 0.17.0
    • Fix Version/s: 0.17.1
    • Component/s: None
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      The example in the README.txt in contrig/data_join/src/example doesn't work in 0.17.0. It works perfectly in 0.16.4.

      1. patch
        1 kB
        Spyros Blanas

        Issue Links

          Activity

          Spyros Blanas created issue -
          Hide
          Spyros Blanas added a comment -

          The issue is related to HADOOP-3522. The patch just clones the objects returned from the ValueIterator and forces the user-provided classes to implement a clone() method as well.

          Show
          Spyros Blanas added a comment - The issue is related to HADOOP-3522 . The patch just clones the objects returned from the ValueIterator and forces the user-provided classes to implement a clone() method as well.
          Spyros Blanas made changes -
          Field Original Value New Value
          Attachment datajoin-0.17.patch [ 12383926 ]
          Milind Bhandarkar made changes -
          Fix Version/s 0.17.1 [ 12313190 ]
          Priority Major [ 3 ] Blocker [ 1 ]
          Hide
          Runping Qi added a comment -

          Since applications using this package break with 0.17,
          I think this is a blocker for 0.17.1

          Show
          Runping Qi added a comment - Since applications using this package break with 0.17, I think this is a blocker for 0.17.1
          Hide
          Chris Douglas added a comment -

          Spyros-

          This looks like the right idea; a couple suggestions:

          • It would break less existing code if this used WritableUtils::clone instead of adding an abstract method
          • The tag at DataJoinReducerBase::107 doesn't need to be cloned
          Show
          Chris Douglas added a comment - Spyros- This looks like the right idea; a couple suggestions: It would break less existing code if this used WritableUtils::clone instead of adding an abstract method The tag at DataJoinReducerBase::107 doesn't need to be cloned
          Hide
          Nigel Daley added a comment -

          -1 as the patch should have a unit test.

          Show
          Nigel Daley added a comment - -1 as the patch should have a unit test.
          Devaraj Das made changes -
          Assignee Spyros Blanas [ sblanas ]
          Spyros Blanas made changes -
          Attachment datajoin-0.17.patch [ 12383926 ]
          Hide
          Spyros Blanas added a comment -

          Chris,
          The tag needs cloning, because it's stored as a key in the SortedMap. Otherwise, the Text.equals() method always returns true (see HADOOP-3522).
          As for the WritableUtils.clone() suggestion, I upload a new version of the patch which incorporates it.

          Nigel,
          Unfortunately there is no existing JUnit test covering the contributed data_join code and I don't know how to change the build.xml file to run the new tests. Also, building a JUnit testing framework for generic map-reduce jobs like data_join is much more difficult than targeting specific functions, as done in the core. I would be happy to create a test if you can point me to some code that tests a generic map/reduce job that I can tweak.

          Show
          Spyros Blanas added a comment - Chris, The tag needs cloning, because it's stored as a key in the SortedMap. Otherwise, the Text.equals() method always returns true (see HADOOP-3522 ). As for the WritableUtils.clone() suggestion, I upload a new version of the patch which incorporates it. Nigel, Unfortunately there is no existing JUnit test covering the contributed data_join code and I don't know how to change the build.xml file to run the new tests. Also, building a JUnit testing framework for generic map-reduce jobs like data_join is much more difficult than targeting specific functions, as done in the core. I would be happy to create a test if you can point me to some code that tests a generic map/reduce job that I can tweak.
          Spyros Blanas made changes -
          Attachment patch [ 12384086 ]
          Hide
          Nigel Daley added a comment -

          Spyros, Chris has agreed to write some unit tests for Hadoop 0.18.

          Show
          Nigel Daley added a comment - Spyros, Chris has agreed to write some unit tests for Hadoop 0.18.
          Hide
          Chris Douglas added a comment -

          The tag needs cloning, because it's stored as a key in the SortedMap. Otherwise, the Text.equals() method always returns true

          Sorry, I mean it doesn't need to be cloned there, for the lookup. Since the record needs to be cloned anyway and it includes the tag, it should be sufficient to clone the record and use the tag for the map key.

          Show
          Chris Douglas added a comment - The tag needs cloning, because it's stored as a key in the SortedMap. Otherwise, the Text.equals() method always returns true Sorry, I mean it doesn't need to be cloned there, for the lookup. Since the record needs to be cloned anyway and it includes the tag, it should be sufficient to clone the record and use the tag for the map key.
          Chris Douglas made changes -
          Link This issue is related to HADOOP-3587 [ HADOOP-3587 ]
          Hide
          Chris Douglas added a comment -

          I just committed this. Thanks, Spyros

          Show
          Chris Douglas added a comment - I just committed this. Thanks, Spyros
          Chris Douglas made changes -
          Hadoop Flags [Reviewed]
          Resolution Fixed [ 1 ]
          Status Open [ 1 ] Resolved [ 5 ]
          Fix Version/s 0.18.0 [ 12312972 ]
          Hide
          Hudson added a comment -
          Show
          Hudson added a comment - Integrated in Hadoop-trunk #523 (See http://hudson.zones.apache.org/hudson/job/Hadoop-trunk/523/ )
          Nigel Daley made changes -
          Fix Version/s 0.18.0 [ 12312972 ]

            People

            • Assignee:
              Spyros Blanas
              Reporter:
              Spyros Blanas
            • Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development