Uploaded image for project: 'Hadoop Common'
  1. Hadoop Common
  2. HADOOP-7183

WritableComparator.get should not cache comparator objects

VotersWatch issueWatchersCreate sub-taskLinkCloneUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

Details

    • Bug
    • Status: Closed
    • Blocker
    • Resolution: Fixed
    • 0.22.0
    • 0.21.1, 0.22.0
    • None
    • None
    • Reviewed

    Description

      HADOOP-6881 modified WritableComparator.get such that the constructed WritableComparator gets saved back into the static map. This is fine for stateless comparators, but some comparators have per-instance state, and thus this becomes thread-unsafe and causes errors in the shuffle where multiple threads are doing comparisons. An example of a Comparator with per-instance state is WritableComparator itself.

      Attachments

        1. HADOOP-7183.patch
          2 kB
          Thomas White
        2. HADOOP-7183.patch
          0.9 kB
          Thomas White

        Issue Links

        Activity

          This comment will be Viewable by All Users Viewable by All Users
          Cancel

          People

            tomwhite Thomas White
            tlipcon Todd Lipcon
            Votes:
            0 Vote for this issue
            Watchers:
            8 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Issue deployment