Hadoop Map/Reduce
  1. Hadoop Map/Reduce
  2. MAPREDUCE-2506

Create a compatible interface for frameworks that need to clone MapReduce context objects.

    Details

    • Type: Improvement Improvement
    • Status: Open
    • Priority: Major Major
    • Resolution: Unresolved
    • Affects Version/s: None
    • Fix Version/s: None
    • Component/s: None
    • Labels:
      None

      Description

      In 0.21 we moved the org.apache.hadoop.mapreduce context objects to interfaces.

      That made the APIs much better, but broke backwards compatibility for frameworks that need to clone them.

      1. ContextFactory.java
        9 kB
        Owen O'Malley
      2. mr-2506.patch
        19 kB
        Owen O'Malley

        Activity

        Hide
        Owen O'Malley added a comment -

        This patch I wrote for Pig that compiles and runs against unmodified 0.20.2, 0.20.203.0, and 0.21.0.

        Show
        Owen O'Malley added a comment - This patch I wrote for Pig that compiles and runs against unmodified 0.20.2, 0.20.203.0, and 0.21.0.
        Hide
        Owen O'Malley added a comment -

        This patch creates a ContextFactory class that provides two functions:

          public static JobContext cloneContext(JobContext original,
                                                Configuration conf
                                                ) throws IOException, 
                                                         InterruptedException;
        
          public static <K1,V1,K2,V2> Mapper<K1,V1,K2,V2>.Context 
               cloneMapContext(MapContext<K1,V1,K2,V2> context,
                               Configuration conf,
                               RecordReader<K1,V1> reader,
                               RecordWriter<K2,V2> writer
                              ) throws IOException, InterruptedException;
        

        This first clones JobContexts, TaskAttemptContexts, and MapContexts. The contexts are the same, except they have a new configuration.

        The second method allows additional hooks to replace the reader and writer for the context.

        Show
        Owen O'Malley added a comment - This patch creates a ContextFactory class that provides two functions: public static JobContext cloneContext(JobContext original, Configuration conf ) throws IOException, InterruptedException; public static <K1,V1,K2,V2> Mapper<K1,V1,K2,V2>.Context cloneMapContext(MapContext<K1,V1,K2,V2> context, Configuration conf, RecordReader<K1,V1> reader, RecordWriter<K2,V2> writer ) throws IOException, InterruptedException; This first clones JobContexts, TaskAttemptContexts, and MapContexts. The contexts are the same, except they have a new configuration. The second method allows additional hooks to replace the reader and writer for the context.

          People

          • Assignee:
            Unassigned
            Reporter:
            Owen O'Malley
          • Votes:
            0 Vote for this issue
            Watchers:
            9 Start watching this issue

            Dates

            • Created:
              Updated:

              Development