Uploaded image for project: 'Hadoop Map/Reduce'
  1. Hadoop Map/Reduce
  2. MAPREDUCE-1820

InputSampler does not create a deep copy of the key object when creating a sample, which causes problems with some formats like SequenceFile<Text,Text>

VotersWatch issueWatchersCreate sub-taskLinkCloneUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

Details

    • Bug
    • Status: Closed
    • Major
    • Resolution: Fixed
    • None
    • 0.22.0
    • None
    • None
    • Reviewed

    Description

      I tried to use the InputSampler on a SequenceFile<Text,Text> and found that it comes up with duplicate keys in the sample. The problem was tracked down to the fact that the Text object returned from the reader is essentially a wrapper pointing to a byte array, which changes as the sequence file reader progresses. There was also a bug in that the reader should be initialized before the use. The am attaching a patch that fixes both of the issues. --Alex K

      Attachments

        1. M1820-4.patch
          11 kB
          Christopher Douglas
        2. M1820-5.patch
          11 kB
          Christopher Douglas
        3. MAPREDUCE-1820.patch
          6 kB
          Alex Kozlov
        4. MAPREDUCE-1820-2.patch
          6 kB
          Alex Kozlov
        5. MAPREDUCE-1820-3.patch
          6 kB
          Alex Kozlov

        Issue Links

        Activity

          This comment will be Viewable by All Users Viewable by All Users
          Cancel

          People

            alexvk Alex Kozlov
            alexvk Alex Kozlov
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Time Tracking

                Estimated:
                Original Estimate - 2h
                2h
                Remaining:
                Remaining Estimate - 2h
                2h
                Logged:
                Time Spent - Not Specified
                Not Specified

                Issue deployment