Hadoop Common
  1. Hadoop Common
  2. HADOOP-6156 Move map/reduce specific classes out of common
  3. HADOOP-6230

Move process tree, and memory calculator classes out of Common into Map/Reduce.

    Details

    • Type: Sub-task Sub-task
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.21.0
    • Component/s: util
    • Labels:
      None
    • Hadoop Flags:
      Incompatible change, Reviewed
    • Release Note:
      Moved process tree, and memory calculator classes out of Common project into the Map/Reduce project.
    1. HADOOP-6230-20090903-mapred.txt
      64 kB
      Vinod Kumar Vavilapalli
    2. HADOOP-6230-20090903-common.txt
      52 kB
      Vinod Kumar Vavilapalli
    3. HADOOP-6230-20090902-mapred.txt
      52 kB
      Vinod Kumar Vavilapalli
    4. HADOOP-6230-20090902-common.txt
      52 kB
      Vinod Kumar Vavilapalli

      Issue Links

        Activity

        Hide
        Vinod Kumar Vavilapalli added a comment -

        I've verified that the following are referenced only by mapred and so am going to move them into mapreduce project.

        • ProcessTree.java
        • ProcfsBasedProcessTree.java
        • MemoryCalculatorPlugin.java
        • DummyMemoryCalculatorPlugin.java
        • LinuxMemoryCalculatorPlugin.java
        • TestProcfsBasedProcessTree.java
        Show
        Vinod Kumar Vavilapalli added a comment - I've verified that the following are referenced only by mapred and so am going to move them into mapreduce project. ProcessTree.java ProcfsBasedProcessTree.java MemoryCalculatorPlugin.java DummyMemoryCalculatorPlugin.java LinuxMemoryCalculatorPlugin.java TestProcfsBasedProcessTree.java
        Hide
        Vinod Kumar Vavilapalli added a comment -

        Attaching two patches for this issue. The common patch removes the files and mapreduce patch adds them to the mapreduce project. DummyMemoryPlugin is already in mapreduce, so not doing anything with it.

        Show
        Vinod Kumar Vavilapalli added a comment - Attaching two patches for this issue. The common patch removes the files and mapreduce patch adds them to the mapreduce project. DummyMemoryPlugin is already in mapreduce, so not doing anything with it.
        Hide
        Vinod Kumar Vavilapalli added a comment -

        The common tests passes with the patch for common project.

        I generated common jar and common test jar after applying the common patch and put them in mapreduce lib dir so as to run mapred and contrib tests. These tests also succeeded.

        These patches can be committed.

        Show
        Vinod Kumar Vavilapalli added a comment - The common tests passes with the patch for common project. I generated common jar and common test jar after applying the common patch and put them in mapreduce lib dir so as to run mapred and contrib tests. These tests also succeeded. These patches can be committed.
        Hide
        Vinod Kumar Vavilapalli added a comment -

        I am outlining steps for committing this (Copied from MAPREDUCE-711)

        • First, commit the changes to common using the common patch.
        • Trigger a Hudson build of common and get the common jar and common test jar files.
        • Then, commit the following to mapreduce project.
          • common jar and common test jar files picked up from Hudson to mapreduce/lib folder and
          • changes to mapreduce sources using the mapreduce patch

        No jars are needed to be committed to HDFS as of now.

        Show
        Vinod Kumar Vavilapalli added a comment - I am outlining steps for committing this (Copied from MAPREDUCE-711 ) First, commit the changes to common using the common patch. Trigger a Hudson build of common and get the common jar and common test jar files. Then, commit the following to mapreduce project. common jar and common test jar files picked up from Hudson to mapreduce/lib folder and changes to mapreduce sources using the mapreduce patch No jars are needed to be committed to HDFS as of now.
        Hide
        Vinod Kumar Vavilapalli added a comment -

        ant `test-patch` results for both mapred and common return all +1s.

        Show
        Vinod Kumar Vavilapalli added a comment - ant `test-patch` results for both mapred and common return all +1s.
        Hide
        Hemanth Yamijala added a comment -

        I think the classes in mapreduce should be in the package o.a.h.mapreduce.util instead of o.a.h.util as the latter is currently being used to refer to classes that are common utilities across all sub-projects of Hadoop. Thoughts ?

        Show
        Hemanth Yamijala added a comment - I think the classes in mapreduce should be in the package o.a.h.mapreduce.util instead of o.a.h.util as the latter is currently being used to refer to classes that are common utilities across all sub-projects of Hadoop. Thoughts ?
        Hide
        Vinod Kumar Vavilapalli added a comment -

        Attaching updated patches moving the new files in mapred to o.a.h.mapreduce.util and deprecating the relevant files in o.a.h.util.

        There are still two tests TestRunJar.java and TestReflectionUtils.java in o.a.h.mapreduce.util. The later doesn't clearly belong to mapreduce. Will create a new issue to move them to more appropriate locations.

        Show
        Vinod Kumar Vavilapalli added a comment - Attaching updated patches moving the new files in mapred to o.a.h.mapreduce.util and deprecating the relevant files in o.a.h.util. There are still two tests TestRunJar.java and TestReflectionUtils.java in o.a.h.mapreduce.util. The later doesn't clearly belong to mapreduce. Will create a new issue to move them to more appropriate locations.
        Hide
        Tom White added a comment -

        TestReflectionUtils depends on JobConf, so it can't be moved in its entirety. Probably best to split it into core and mapreduce pieces.

        Show
        Tom White added a comment - TestReflectionUtils depends on JobConf, so it can't be moved in its entirety. Probably best to split it into core and mapreduce pieces.
        Hide
        Vinod Kumar Vavilapalli added a comment -

        TestReflectionUtils depends on JobConf, so it can't be moved in its entirety. Probably best to split it into core and mapreduce pieces.

        Agreed Tom. Created HADOOP-6237 for the same.

        Show
        Vinod Kumar Vavilapalli added a comment - TestReflectionUtils depends on JobConf, so it can't be moved in its entirety. Probably best to split it into core and mapreduce pieces. Agreed Tom. Created HADOOP-6237 for the same.
        Hide
        Hemanth Yamijala added a comment -

        This seems fine to me. +1.

        Show
        Hemanth Yamijala added a comment - This seems fine to me. +1.
        Hide
        Vinod Kumar Vavilapalli added a comment -

        `ant test' and `ant test-patch' passed for the common project's patch. Common patch can first be committed now.

        Show
        Vinod Kumar Vavilapalli added a comment - `ant test' and `ant test-patch' passed for the common project's patch. Common patch can first be committed now.
        Hide
        Hemanth Yamijala added a comment -

        I just committed to trunk. We should fire a hudson build to get jar files for testing the Map/Reduce changes.

        Show
        Hemanth Yamijala added a comment - I just committed to trunk. We should fire a hudson build to get jar files for testing the Map/Reduce changes.
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Common-trunk-Commit #14 (See http://hudson.zones.apache.org/hudson/job/Hadoop-Common-trunk-Commit/14/)
        . Moved process tree and memory calculator related classes from Common to Map/Reduce. Contributed by Vinod Kumar Vavilapalli.

        Show
        Hudson added a comment - Integrated in Hadoop-Common-trunk-Commit #14 (See http://hudson.zones.apache.org/hudson/job/Hadoop-Common-trunk-Commit/14/ ) . Moved process tree and memory calculator related classes from Common to Map/Reduce. Contributed by Vinod Kumar Vavilapalli.
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Common-trunk #87 (See http://hudson.zones.apache.org/hudson/job/Hadoop-Common-trunk/87/)
        . Moved process tree and memory calculator related classes from Common to Map/Reduce. Contributed by Vinod Kumar Vavilapalli.

        Show
        Hudson added a comment - Integrated in Hadoop-Common-trunk #87 (See http://hudson.zones.apache.org/hudson/job/Hadoop-Common-trunk/87/ ) . Moved process tree and memory calculator related classes from Common to Map/Reduce. Contributed by Vinod Kumar Vavilapalli.
        Hide
        Vinod Kumar Vavilapalli added a comment -

        I ran mapred, and contrib tests locally which have all passed.

        Running `ant test-patch' got a bit trickier. I had to generate a special common jars with only the changes to Shell.java from the common patch and copy them over to mapred. I got all +1's except the following which are unavoidable:

                The class name org.apache.hadoop.util.LinuxMemoryCalculatorPlugin     
           Nm   shadows the simple name of the superclass                             
                org.apache.hadoop.mapreduce.util.LinuxMemoryCalculatorPlugin          
                [5]Bug type NM_SAME_SIMPLE_NAME_AS_SUPERCLASS (click for details)     
                In class org.apache.hadoop.util.LinuxMemoryCalculatorPlugin           
                In class org.apache.hadoop.mapreduce.util.LinuxMemoryCalculatorPlugin 
                At LinuxMemoryCalculatorPlugin.java:[line 28]                         
                The class name org.apache.hadoop.util.MemoryCalculatorPlugin shadows  
           Nm   the simple name of the superclass                                     
                org.apache.hadoop.mapreduce.util.MemoryCalculatorPlugin               
                [6]Bug type NM_SAME_SIMPLE_NAME_AS_SUPERCLASS (click for details)     
                In class org.apache.hadoop.util.MemoryCalculatorPlugin                
                In class org.apache.hadoop.mapreduce.util.MemoryCalculatorPlugin      
                At MemoryCalculatorPlugin.java:[lines 32-37]                          
           Nm   The class name org.apache.hadoop.util.ProcessTree shadows the simple  
                name of the superclass org.apache.hadoop.mapreduce.util.ProcessTree   
                [7]Bug type NM_SAME_SIMPLE_NAME_AS_SUPERCLASS (click for details)     
                In class org.apache.hadoop.util.ProcessTree                           
                In class org.apache.hadoop.mapreduce.util.ProcessTree                 
                At ProcessTree.java:[line 27]                                         
                The class name org.apache.hadoop.util.ProcfsBasedProcessTree shadows  
           Nm   the simple name of the superclass                                     
                org.apache.hadoop.mapreduce.util.ProcfsBasedProcessTree               
                [8]Bug type NM_SAME_SIMPLE_NAME_AS_SUPERCLASS (click for details)     
                In class org.apache.hadoop.util.ProcfsBasedProcessTree                
                In class org.apache.hadoop.mapreduce.util.ProcfsBasedProcessTree      
                At ProcfsBasedProcessTree.java:[lines 28-47]                          
        

        The mapreduce patch is committable.

        Show
        Vinod Kumar Vavilapalli added a comment - I ran mapred, and contrib tests locally which have all passed. Running `ant test-patch' got a bit trickier. I had to generate a special common jars with only the changes to Shell.java from the common patch and copy them over to mapred. I got all +1's except the following which are unavoidable: The class name org.apache.hadoop.util.LinuxMemoryCalculatorPlugin Nm shadows the simple name of the superclass org.apache.hadoop.mapreduce.util.LinuxMemoryCalculatorPlugin [5]Bug type NM_SAME_SIMPLE_NAME_AS_SUPERCLASS (click for details) In class org.apache.hadoop.util.LinuxMemoryCalculatorPlugin In class org.apache.hadoop.mapreduce.util.LinuxMemoryCalculatorPlugin At LinuxMemoryCalculatorPlugin.java:[line 28] The class name org.apache.hadoop.util.MemoryCalculatorPlugin shadows Nm the simple name of the superclass org.apache.hadoop.mapreduce.util.MemoryCalculatorPlugin [6]Bug type NM_SAME_SIMPLE_NAME_AS_SUPERCLASS (click for details) In class org.apache.hadoop.util.MemoryCalculatorPlugin In class org.apache.hadoop.mapreduce.util.MemoryCalculatorPlugin At MemoryCalculatorPlugin.java:[lines 32-37] Nm The class name org.apache.hadoop.util.ProcessTree shadows the simple name of the superclass org.apache.hadoop.mapreduce.util.ProcessTree [7]Bug type NM_SAME_SIMPLE_NAME_AS_SUPERCLASS (click for details) In class org.apache.hadoop.util.ProcessTree In class org.apache.hadoop.mapreduce.util.ProcessTree At ProcessTree.java:[line 27] The class name org.apache.hadoop.util.ProcfsBasedProcessTree shadows Nm the simple name of the superclass org.apache.hadoop.mapreduce.util.ProcfsBasedProcessTree [8]Bug type NM_SAME_SIMPLE_NAME_AS_SUPERCLASS (click for details) In class org.apache.hadoop.util.ProcfsBasedProcessTree In class org.apache.hadoop.mapreduce.util.ProcfsBasedProcessTree At ProcfsBasedProcessTree.java:[lines 28-47] The mapreduce patch is committable.
        Hide
        Hemanth Yamijala added a comment -

        I just committed the mapreduce changes and the common project jars from Hudson build to the map/reduce subproject thus completing the move.

        Thanks, Vinod !

        Show
        Hemanth Yamijala added a comment - I just committed the mapreduce changes and the common project jars from Hudson build to the map/reduce subproject thus completing the move. Thanks, Vinod !
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Mapreduce-trunk-Commit #23 (See http://hudson.zones.apache.org/hudson/job/Hadoop-Mapreduce-trunk-Commit/23/)

        Show
        Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk-Commit #23 (See http://hudson.zones.apache.org/hudson/job/Hadoop-Mapreduce-trunk-Commit/23/ )

          People

          • Assignee:
            Vinod Kumar Vavilapalli
            Reporter:
            Vinod Kumar Vavilapalli
          • Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development