Hadoop Common
  1. Hadoop Common
  2. HADOOP-7183

WritableComparator.get should not cache comparator objects

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Blocker Blocker
    • Resolution: Fixed
    • Affects Version/s: 0.22.0
    • Fix Version/s: 0.21.1, 0.22.0
    • Component/s: None
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      HADOOP-6881 modified WritableComparator.get such that the constructed WritableComparator gets saved back into the static map. This is fine for stateless comparators, but some comparators have per-instance state, and thus this becomes thread-unsafe and causes errors in the shuffle where multiple threads are doing comparisons. An example of a Comparator with per-instance state is WritableComparator itself.

      1. HADOOP-7183.patch
        2 kB
        Tom White
      2. HADOOP-7183.patch
        0.9 kB
        Tom White

        Issue Links

          Activity

          Todd Lipcon created issue -
          Doug Cutting made changes -
          Field Original Value New Value
          Fix Version/s 0.20.3 [ 12314812 ]
          Fix Version/s 0.21.1 [ 12315270 ]
          Tom White made changes -
          Attachment HADOOP-7183.patch [ 12473422 ]
          Tom White made changes -
          Assignee Tom White [ tomwhite ]
          Tom White made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          Tom White made changes -
          Link This issue is related to HADOOP-7219 [ HADOOP-7219 ]
          Tom White made changes -
          Attachment HADOOP-7183.patch [ 12477363 ]
          Eli Collins made changes -
          Status Patch Available [ 10002 ] Resolved [ 5 ]
          Hadoop Flags [Reviewed]
          Fix Version/s 0.20.3 [ 12314812 ]
          Resolution Fixed [ 1 ]
          Konstantin Shvachko made changes -
          Status Resolved [ 5 ] Closed [ 6 ]

            People

            • Assignee:
              Tom White
              Reporter:
              Todd Lipcon
            • Votes:
              0 Vote for this issue
              Watchers:
              8 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development