Details

    • Type: Sub-task Sub-task
    • Status: Closed
    • Priority: Blocker Blocker
    • Resolution: Fixed
    • Affects Version/s: 0.21.0
    • Fix Version/s: 0.21.0
    • Component/s: None
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      At present, all the sub-projects - common, hdfs and mapreduce - have copies of all the configuration files. Common configuration files should be left in common, mapreduce specific files should be moved to mapreduce project, same with hdfs related files.

      1. MAPREDUCE-1033.patch
        28 kB
        Tom White
      2. MAPREDUCE-1033.patch
        28 kB
        Tom White
      3. MAPREDUCE-1033.patch
        28 kB
        Tom White
      4. MAPREDUCE-1033.patch
        28 kB
        Tom White

        Issue Links

          Activity

          Transition Time In Source Status Execution Times Last Executer Last Execution Date
          Patch Available Patch Available Open Open
          1d 6h 26m 1 Tom White 02/Jun/10 06:12
          Open Open Patch Available Patch Available
          249d 14h 42m 2 Tom White 02/Jun/10 06:12
          Patch Available Patch Available Resolved Resolved
          8d 18h 38m 1 Tom White 11/Jun/10 00:50
          Resolved Resolved Closed Closed
          74d 21h 27m 1 Tom White 24/Aug/10 22:18
          Tom White made changes -
          Status Resolved [ 5 ] Closed [ 6 ]
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Hdfs-trunk-Commit #310 (See http://hudson.zones.apache.org/hudson/job/Hadoop-Hdfs-trunk-Commit/310/)

          Show
          Hudson added a comment - Integrated in Hadoop-Hdfs-trunk-Commit #310 (See http://hudson.zones.apache.org/hudson/job/Hadoop-Hdfs-trunk-Commit/310/ )
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Common-trunk #363 (See http://hudson.zones.apache.org/hudson/job/Hadoop-Common-trunk/363/)
          HADOOP-6794. Move configuration and script files post split. Includes HDFS-1181, MAPREDUCE-1033.

          Show
          Hudson added a comment - Integrated in Hadoop-Common-trunk #363 (See http://hudson.zones.apache.org/hudson/job/Hadoop-Common-trunk/363/ ) HADOOP-6794 . Move configuration and script files post split. Includes HDFS-1181 , MAPREDUCE-1033 .
          Tom White made changes -
          Status Patch Available [ 10002 ] Resolved [ 5 ]
          Resolution Fixed [ 1 ]
          Hide
          Tom White added a comment -

          I've just committed this.

          Show
          Tom White added a comment - I've just committed this.
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Common-trunk-Commit #293 (See http://hudson.zones.apache.org/hudson/job/Hadoop-Common-trunk-Commit/293/)
          HADOOP-6794. Move configuration and script files post split. Includes HDFS-1181, MAPREDUCE-1033.

          Show
          Hudson added a comment - Integrated in Hadoop-Common-trunk-Commit #293 (See http://hudson.zones.apache.org/hudson/job/Hadoop-Common-trunk-Commit/293/ ) HADOOP-6794 . Move configuration and script files post split. Includes HDFS-1181 , MAPREDUCE-1033 .
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12446106/MAPREDUCE-1033.patch
          against trunk revision 950286.

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 2 new or modified tests.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 findbugs. The patch does not introduce any new Findbugs warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed core unit tests.

          -1 contrib tests. The patch failed contrib unit tests.

          Test results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h4.grid.sp2.yahoo.net/220/testReport/
          Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h4.grid.sp2.yahoo.net/220/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
          Checkstyle results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h4.grid.sp2.yahoo.net/220/artifact/trunk/build/test/checkstyle-errors.html
          Console output: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h4.grid.sp2.yahoo.net/220/console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12446106/MAPREDUCE-1033.patch against trunk revision 950286. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 2 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed core unit tests. -1 contrib tests. The patch failed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h4.grid.sp2.yahoo.net/220/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h4.grid.sp2.yahoo.net/220/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Checkstyle results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h4.grid.sp2.yahoo.net/220/artifact/trunk/build/test/checkstyle-errors.html Console output: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h4.grid.sp2.yahoo.net/220/console This message is automatically generated.
          Tom White made changes -
          Attachment MAPREDUCE-1033.patch [ 12446305 ]
          Vinod Kumar Vavilapalli made changes -
          Summary Resolve location of configuration files after project split Resolve location of scripts and configuration files after project split
          Hadoop Flags [Reviewed]
          Hide
          Vinod Kumar Vavilapalli added a comment -

          +1 for the latest patch for mapreduce. I could successfully bring up a single node cluster using the patches for all the projects and HADOOP-6461.

          Show
          Vinod Kumar Vavilapalli added a comment - +1 for the latest patch for mapreduce. I could successfully bring up a single node cluster using the patches for all the projects and HADOOP-6461 .
          Tom White made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          Tom White made changes -
          Status Patch Available [ 10002 ] Open [ 1 ]
          Tom White made changes -
          Attachment MAPREDUCE-1033.patch [ 12446106 ]
          Tom White made changes -
          Attachment MAPREDUCE-1033.patch [ 12446074 ]
          Hide
          Hadoop QA added a comment -

          +1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12445967/MAPREDUCE-1033.patch
          against trunk revision 949815.

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 2 new or modified tests.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 findbugs. The patch does not introduce any new Findbugs warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed core unit tests.

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h4.grid.sp2.yahoo.net/213/testReport/
          Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h4.grid.sp2.yahoo.net/213/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
          Checkstyle results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h4.grid.sp2.yahoo.net/213/artifact/trunk/build/test/checkstyle-errors.html
          Console output: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h4.grid.sp2.yahoo.net/213/console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - +1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12445967/MAPREDUCE-1033.patch against trunk revision 949815. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 2 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed core unit tests. +1 contrib tests. The patch passed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h4.grid.sp2.yahoo.net/213/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h4.grid.sp2.yahoo.net/213/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Checkstyle results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h4.grid.sp2.yahoo.net/213/artifact/trunk/build/test/checkstyle-errors.html Console output: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h4.grid.sp2.yahoo.net/213/console This message is automatically generated.
          Hide
          Tom White added a comment -

          Sorry, third part patch is in HDFS-1181.

          Show
          Tom White added a comment - Sorry, third part patch is in HDFS-1181 .
          Tom White made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          Assignee Tom White [ tomwhite ]
          Tom White made changes -
          Attachment MAPREDUCE-1033.patch [ 12445967 ]
          Hide
          Tom White added a comment -

          Set of three patches (see also HADOOP-6794, MAPREDUCE-1033) to move config and script files per previous comment.

          Also:

          • Fixes script permissions in HDFS and MapReduce
          • Fixes a name collision in hadoop-config.sh
          • Fixed hdfs-config.sh, mapred-config.sh to define HADOOP_HDFS_HOME, HADOOP_MAPRED_HOME (respectively), which is needed to run in separate install locations. (See also HADOOP-6342, which will make it easy to install in one location.)
          • Changes HADOOP_CORE_HOME to HADOOP_COMMON_HOME. There has been no release with HADOOP_CORE_HOME, so it's good to make this change now.

          To test, I built tarballs for each project, unpacked them in separate locations, and successfully ran a job in pseudo-distributed mode.

          Show
          Tom White added a comment - Set of three patches (see also HADOOP-6794 , MAPREDUCE-1033 ) to move config and script files per previous comment. Also: Fixes script permissions in HDFS and MapReduce Fixes a name collision in hadoop-config.sh Fixed hdfs-config.sh, mapred-config.sh to define HADOOP_HDFS_HOME, HADOOP_MAPRED_HOME (respectively), which is needed to run in separate install locations. (See also HADOOP-6342 , which will make it easy to install in one location.) Changes HADOOP_CORE_HOME to HADOOP_COMMON_HOME. There has been no release with HADOOP_CORE_HOME, so it's good to make this change now. To test, I built tarballs for each project, unpacked them in separate locations, and successfully ran a job in pseudo-distributed mode.
          Tom White made changes -
          Link This issue incorporates HADOOP-6794 [ HADOOP-6794 ]
          Hide
          Tom White added a comment -

          Files that should be in mapreduce/conf

          capacity-scheduler.xml
          capacity-scheduler.xml.template
          fair-scheduler.xml
          fair-scheduler.xml.template
          mapred-queue-acls.xml
          mapred-queue-acls.xml.template
          mapred-site.xml
          mapred-site.xml.template
          taskcontroller.cfg
          

          Files that should be in hdfs/conf

          hdfs-site.xml
          hdfs-site.xml.template
          

          We could add configuration.xsl too (i.e. have it in all conf directories). Does this look right?

          Also, the bin scripts are all in common - we should move these too.

          hdfs/bin:

          hdfs
          hdfs-config.sh
          start-balancer.sh
          start-dfs.sh
          stop-dfs.sh
          

          mapreduce/bin:

          mapred
          mapred-config.sh
          start-mapred.sh
          stop-mapred.sh
          
          Show
          Tom White added a comment - Files that should be in mapreduce/conf capacity-scheduler.xml capacity-scheduler.xml.template fair-scheduler.xml fair-scheduler.xml.template mapred-queue-acls.xml mapred-queue-acls.xml.template mapred-site.xml mapred-site.xml.template taskcontroller.cfg Files that should be in hdfs/conf hdfs-site.xml hdfs-site.xml.template We could add configuration.xsl too (i.e. have it in all conf directories). Does this look right? Also, the bin scripts are all in common - we should move these too. hdfs/bin: hdfs hdfs-config.sh start-balancer.sh start-dfs.sh stop-dfs.sh mapreduce/bin: mapred mapred-config.sh start-mapred.sh stop-mapred.sh
          Hide
          Vinod Kumar Vavilapalli added a comment -

          Tom and Owen, I took the liberty of adding you guys to the watch list so that this issue can move forward..

          Show
          Vinod Kumar Vavilapalli added a comment - Tom and Owen, I took the liberty of adding you guys to the watch list so that this issue can move forward..
          Vinod Kumar Vavilapalli made changes -
          Field Original Value New Value
          Affects Version/s 0.21.0 [ 12314045 ]
          Hide
          Vinod Kumar Vavilapalli added a comment -

          Irrespective of what is going to be decided here, one thing is clear. Some of the conf files have changed in the individual projects - I know of issues in mapreduce which changed 3-4 conf files and conf templates, by the time of this comment. We need to make sure these changes are propagated to the right place when 0.21 is released.

          Show
          Vinod Kumar Vavilapalli added a comment - Irrespective of what is going to be decided here, one thing is clear. Some of the conf files have changed in the individual projects - I know of issues in mapreduce which changed 3-4 conf files and conf templates, by the time of this comment. We need to make sure these changes are propagated to the right place when 0.21 is released.
          Vinod Kumar Vavilapalli created issue -

            People

            • Assignee:
              Tom White
              Reporter:
              Vinod Kumar Vavilapalli
            • Votes:
              0 Vote for this issue
              Watchers:
              7 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development