Hadoop Map/Reduce
  1. Hadoop Map/Reduce
  2. MAPREDUCE-1820

InputSampler does not create a deep copy of the key object when creating a sample, which causes problems with some formats like SequenceFile<Text,Text>

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.22.0
    • Component/s: None
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      I tried to use the InputSampler on a SequenceFile<Text,Text> and found that it comes up with duplicate keys in the sample. The problem was tracked down to the fact that the Text object returned from the reader is essentially a wrapper pointing to a byte array, which changes as the sequence file reader progresses. There was also a bug in that the reader should be initialized before the use. The am attaching a patch that fixes both of the issues. --Alex K

      1. MAPREDUCE-1820-3.patch
        6 kB
        Alex Kozlov
      2. MAPREDUCE-1820-2.patch
        6 kB
        Alex Kozlov
      3. MAPREDUCE-1820.patch
        6 kB
        Alex Kozlov
      4. M1820-5.patch
        11 kB
        Chris Douglas
      5. M1820-4.patch
        11 kB
        Chris Douglas

        Issue Links

          Activity

          Zhijie Shen made changes -
          Link This issue is related to MAPREDUCE-5225 [ MAPREDUCE-5225 ]
          Konstantin Shvachko made changes -
          Status Resolved [ 5 ] Closed [ 6 ]
          Chris Douglas made changes -
          Status Patch Available [ 10002 ] Resolved [ 5 ]
          Hadoop Flags [Reviewed]
          Fix Version/s 0.22.0 [ 12314184 ]
          Resolution Fixed [ 1 ]
          Chris Douglas made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          Chris Douglas made changes -
          Status Patch Available [ 10002 ] Open [ 1 ]
          Chris Douglas made changes -
          Attachment M1820-5.patch [ 12448863 ]
          Chris Douglas made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          Chris Douglas made changes -
          Attachment M1820-4.patch [ 12448832 ]
          Jeff Hammerbacher made changes -
          Link This issue is related to MAPREDUCE-366 [ MAPREDUCE-366 ]
          Chris Douglas made changes -
          Status Patch Available [ 10002 ] Open [ 1 ]
          Alex Kozlov made changes -
          Attachment MAPREDUCE-1820-3.patch [ 12445629 ]
          Alex Kozlov made changes -
          Summary InputSampler does not create a deep copy of the key class when creating a sample, which causes problems with some formats like SequenceFile<Text,Text> InputSampler does not create a deep copy of the key object when creating a sample, which causes problems with some formats like SequenceFile<Text,Text>
          Alex Kozlov made changes -
          Attachment MAPREDUCE-1820-2.patch [ 12445604 ]
          Alex Kozlov made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          Alex Kozlov made changes -
          Attachment MAPREDUCE-1820.patch [ 12445594 ]
          Alex Kozlov made changes -
          Status Patch Available [ 10002 ] Open [ 1 ]
          Assignee Alex Kozlov [ alexvk ]
          Alex Kozlov made changes -
          Field Original Value New Value
          Status Open [ 1 ] Patch Available [ 10002 ]
          Alex Kozlov created issue -

            People

            • Assignee:
              Alex Kozlov
              Reporter:
              Alex Kozlov
            • Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Time Tracking

                Estimated:
                Original Estimate - 2h
                2h
                Remaining:
                Remaining Estimate - 2h
                2h
                Logged:
                Time Spent - Not Specified
                Not Specified

                  Development