Details

    • Type: Improvement Improvement
    • Status: Open
    • Priority: Major Major
    • Resolution: Unresolved
    • Affects Version/s: None
    • Fix Version/s: None
    • Component/s: client
    • Labels:
      None

      Description

      Currently users must specify the output types of the map and reduce if they are different to the default (LongWritable keys, Text values). In many cases this information is available to the system from the user-supplied mapper and reducer, so it would be nice if the user didn't have to specify them in two places.

        Issue Links

          Activity

          Hide
          Tom White added a comment -

          Here's a patch for discussion which introspects the Reducer for its output key and value types. It's only a demonstration of how we might do this - it needs more testing and cleaning up, but it show the general approach.

          It works using generics-introspection code from http://www.artima.com/weblogs/viewpost.jsp?thread=208860, although it might be worth using the (Apache 2.0 licensed) http://code.google.com/p/gentyref/ library which has the same functionality (and handles extra cases).

          With this change the following lines are removed from word count:

          -    job.setOutputKeyClass(Text.class);
          -    job.setOutputValueClass(IntWritable.class);
          
          Show
          Tom White added a comment - Here's a patch for discussion which introspects the Reducer for its output key and value types. It's only a demonstration of how we might do this - it needs more testing and cleaning up, but it show the general approach. It works using generics-introspection code from http://www.artima.com/weblogs/viewpost.jsp?thread=208860 , although it might be worth using the (Apache 2.0 licensed) http://code.google.com/p/gentyref/ library which has the same functionality (and handles extra cases). With this change the following lines are removed from word count: - job.setOutputKeyClass(Text.class); - job.setOutputValueClass(IntWritable.class);

            People

            • Assignee:
              Tom White
              Reporter:
              Tom White
            • Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

              • Created:
                Updated:

                Development