Hadoop Common
  1. Hadoop Common
  2. HADOOP-5858

Eliminate UTF8 and fix warnings in test/hdfs-with-mr package

    Details

    • Type: Improvement Improvement
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 0.19.0
    • Fix Version/s: 0.21.0
    • Component/s: test
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      Replace UTF8 with Text and fix java warnings not related to deprecated mapred api.
      Warnings related to the deprecated map reduce api should probably be targeted in a separate unified approach.

      1. hdfsWithMR.patch
        27 kB
        Konstantin Shvachko
      2. hdfsWithMR.patch
        25 kB
        Konstantin Shvachko

        Activity

        Hide
        Konstantin Shvachko added a comment -

        I just committed this.

        Show
        Konstantin Shvachko added a comment - I just committed this.
        Hide
        Konstantin Shvachko added a comment -

        As I said in the beginning converting to the new "mapreduce" api from "mapred" is not the goal of this issue.

        Show
        Konstantin Shvachko added a comment - As I said in the beginning converting to the new "mapreduce" api from "mapred" is not the goal of this issue.
        Hide
        Konstantin Boudnik added a comment -

        It seems like AccumulatingReducer shouldn't extend MapReducerBase not implement org.apache.hadoop.mapred.Reducer
        Instead it needs to extend org.apache.hadoop.mapreduce.Reducer

        public class AccumulatingReducer extends Reducer<Text, Text, Text, Text>

        Show
        Konstantin Boudnik added a comment - It seems like AccumulatingReducer shouldn't extend MapReducerBase not implement org.apache.hadoop.mapred.Reducer Instead it needs to extend org.apache.hadoop.mapreduce.Reducer public class AccumulatingReducer extends Reducer<Text, Text, Text, Text>
        Hide
        Tsz Wo Nicholas Sze added a comment -

        Changes look good to me. Thanks, Konstantin.

        Want to mention that the following codes in the AccumulatingReducer is inefficient: The sSum string should be constructed by StringBuilder for reducing unnecessary object creations. Of course, this is an existing problem and has nothing to do with the patch or this issue.

               String sSum = "";
               while (values.hasNext())
                 sSum += values.next().toString() + ";";
        
        Show
        Tsz Wo Nicholas Sze added a comment - Changes look good to me. Thanks, Konstantin. Want to mention that the following codes in the AccumulatingReducer is inefficient: The sSum string should be constructed by StringBuilder for reducing unnecessary object creations. Of course, this is an existing problem and has nothing to do with the patch or this issue. String sSum = ""; while (values.hasNext()) sSum += values.next().toString() + ";" ;
        Hide
        Konstantin Shvachko added a comment -

        Fixed all three suggestions.

        1. Using VALUE_TYPE_* constants instead of "l:" etc in all related tests.
        2. VALUE_TYPE_LONG is not public.
        3. The return values of read() are verified.
        Show
        Konstantin Shvachko added a comment - Fixed all three suggestions. Using VALUE_TYPE_* constants instead of "l:" etc in all related tests. VALUE_TYPE_LONG is not public. The return values of read() are verified.
        Hide
        Suresh Srinivas added a comment -
        1. TestHarFileSystem.java - Not using fin.read(b) return value might result in findbugs warnings
        2. nit - AccumulatingReducer.VALUE_TYPE_LONG is public but other VALUE_TYPE_* are not
        Show
        Suresh Srinivas added a comment - TestHarFileSystem.java - Not using fin.read(b) return value might result in findbugs warnings nit - AccumulatingReducer.VALUE_TYPE_LONG is public but other VALUE_TYPE_* are not
        Hide
        Tsz Wo Nicholas Sze added a comment -

        Should we use the constants VALUE_TYPE_LONG in TestDFSIO?

        Show
        Tsz Wo Nicholas Sze added a comment - Should we use the constants VALUE_TYPE_LONG in TestDFSIO?

          People

          • Assignee:
            Konstantin Shvachko
            Reporter:
            Konstantin Shvachko
          • Votes:
            0 Vote for this issue
            Watchers:
            0 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development