Uploaded image for project: 'Hadoop Map/Reduce'
  1. Hadoop Map/Reduce
  2. MAPREDUCE-1347

Missing synchronization in MultipleOutputFormat

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Open
    • Major
    • Resolution: Unresolved
    • 1.0.0
    • None
    • client
    • Fix missing synchronization in MultiOutputFormat.

    Description

      MultipleOutputFormat's RecordWriter implementation doesn't use synchronization when accessing the recordWriters member. When using multithreaded mappers or reducers, this can result in problems where two threads will both try to create the same file, causing AlreadyBeingCreatedException. Doing this more fine-grained than just synchronizing the whole method is probably a good idea, so that multithreaded mappers can actually achieve parallelism writing into separate output streams.

      From what I can tell, the new API's MultipleOutputs seems not to have this issue.

      Attachments

        1. MAPREDUCE-1347.r10.diff
          9 kB
          Harsh J
        2. MAPREDUCE-1347.r9.diff
          8 kB
          Harsh J
        3. MAPREDUCE-1347.r8.diff
          8 kB
          Harsh J
        4. MAPREDUCE-1347.r7.diff
          8 kB
          Harsh J
        5. MAPREDUCE-1347.r6.diff
          8 kB
          Harsh J
        6. MAPREDUCE-1347.r5.diff
          8 kB
          Harsh J
        7. MAPREDUCE-1347.r4.diff
          8 kB
          Harsh J
        8. MAPREDUCE-1347.r3.diff
          7 kB
          Harsh J
        9. MAPREDUCE-1347.r2.diff
          7 kB
          Harsh J
        10. mapreduce.1347.r1.diff
          2 kB
          Harsh J

        Issue Links

          is blocked by

          Test - A new unit, integration or system test. HADOOP-7298 Add test utility for writing multi-threaded tests

          • Major - Major loss of function.
          • Closed

          Activity

            People

              Unassigned Unassigned
              tlipcon Todd Lipcon
              Votes:
              0 Vote for this issue
              Watchers:
              10 Start watching this issue

              Dates

                Created:
                Updated: