Details

    • Type: Sub-task Sub-task
    • Status: Closed
    • Priority: Blocker Blocker
    • Resolution: Fixed
    • Affects Version/s: 0.21.0
    • Fix Version/s: 0.21.0
    • Component/s: None
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      At present, all the sub-projects - common, hdfs and mapreduce - have copies of all the configuration files. Common configuration files should be left in common, mapreduce specific files should be moved to mapreduce project, same with hdfs related files.

      1. MAPREDUCE-1033.patch
        28 kB
        Tom White
      2. MAPREDUCE-1033.patch
        28 kB
        Tom White
      3. MAPREDUCE-1033.patch
        28 kB
        Tom White
      4. MAPREDUCE-1033.patch
        28 kB
        Tom White

        Issue Links

          Activity

          Hide
          Hudson added a comment -

          Integrated in Hadoop-Hdfs-trunk-Commit #310 (See http://hudson.zones.apache.org/hudson/job/Hadoop-Hdfs-trunk-Commit/310/)

          Show
          Hudson added a comment - Integrated in Hadoop-Hdfs-trunk-Commit #310 (See http://hudson.zones.apache.org/hudson/job/Hadoop-Hdfs-trunk-Commit/310/ )
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Common-trunk #363 (See http://hudson.zones.apache.org/hudson/job/Hadoop-Common-trunk/363/)
          HADOOP-6794. Move configuration and script files post split. Includes HDFS-1181, MAPREDUCE-1033.

          Show
          Hudson added a comment - Integrated in Hadoop-Common-trunk #363 (See http://hudson.zones.apache.org/hudson/job/Hadoop-Common-trunk/363/ ) HADOOP-6794 . Move configuration and script files post split. Includes HDFS-1181 , MAPREDUCE-1033 .
          Hide
          Tom White added a comment -

          I've just committed this.

          Show
          Tom White added a comment - I've just committed this.
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Common-trunk-Commit #293 (See http://hudson.zones.apache.org/hudson/job/Hadoop-Common-trunk-Commit/293/)
          HADOOP-6794. Move configuration and script files post split. Includes HDFS-1181, MAPREDUCE-1033.

          Show
          Hudson added a comment - Integrated in Hadoop-Common-trunk-Commit #293 (See http://hudson.zones.apache.org/hudson/job/Hadoop-Common-trunk-Commit/293/ ) HADOOP-6794 . Move configuration and script files post split. Includes HDFS-1181 , MAPREDUCE-1033 .
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12446106/MAPREDUCE-1033.patch
          against trunk revision 950286.

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 2 new or modified tests.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 findbugs. The patch does not introduce any new Findbugs warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed core unit tests.

          -1 contrib tests. The patch failed contrib unit tests.

          Test results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h4.grid.sp2.yahoo.net/220/testReport/
          Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h4.grid.sp2.yahoo.net/220/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
          Checkstyle results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h4.grid.sp2.yahoo.net/220/artifact/trunk/build/test/checkstyle-errors.html
          Console output: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h4.grid.sp2.yahoo.net/220/console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12446106/MAPREDUCE-1033.patch against trunk revision 950286. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 2 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed core unit tests. -1 contrib tests. The patch failed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h4.grid.sp2.yahoo.net/220/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h4.grid.sp2.yahoo.net/220/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Checkstyle results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h4.grid.sp2.yahoo.net/220/artifact/trunk/build/test/checkstyle-errors.html Console output: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h4.grid.sp2.yahoo.net/220/console This message is automatically generated.
          Hide
          Vinod Kumar Vavilapalli added a comment -

          +1 for the latest patch for mapreduce. I could successfully bring up a single node cluster using the patches for all the projects and HADOOP-6461.

          Show
          Vinod Kumar Vavilapalli added a comment - +1 for the latest patch for mapreduce. I could successfully bring up a single node cluster using the patches for all the projects and HADOOP-6461 .
          Hide
          Hadoop QA added a comment -

          +1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12445967/MAPREDUCE-1033.patch
          against trunk revision 949815.

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 2 new or modified tests.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 findbugs. The patch does not introduce any new Findbugs warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed core unit tests.

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h4.grid.sp2.yahoo.net/213/testReport/
          Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h4.grid.sp2.yahoo.net/213/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
          Checkstyle results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h4.grid.sp2.yahoo.net/213/artifact/trunk/build/test/checkstyle-errors.html
          Console output: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h4.grid.sp2.yahoo.net/213/console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - +1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12445967/MAPREDUCE-1033.patch against trunk revision 949815. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 2 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed core unit tests. +1 contrib tests. The patch passed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h4.grid.sp2.yahoo.net/213/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h4.grid.sp2.yahoo.net/213/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Checkstyle results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h4.grid.sp2.yahoo.net/213/artifact/trunk/build/test/checkstyle-errors.html Console output: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h4.grid.sp2.yahoo.net/213/console This message is automatically generated.
          Hide
          Tom White added a comment -

          Sorry, third part patch is in HDFS-1181.

          Show
          Tom White added a comment - Sorry, third part patch is in HDFS-1181 .
          Hide
          Tom White added a comment -

          Set of three patches (see also HADOOP-6794, MAPREDUCE-1033) to move config and script files per previous comment.

          Also:

          • Fixes script permissions in HDFS and MapReduce
          • Fixes a name collision in hadoop-config.sh
          • Fixed hdfs-config.sh, mapred-config.sh to define HADOOP_HDFS_HOME, HADOOP_MAPRED_HOME (respectively), which is needed to run in separate install locations. (See also HADOOP-6342, which will make it easy to install in one location.)
          • Changes HADOOP_CORE_HOME to HADOOP_COMMON_HOME. There has been no release with HADOOP_CORE_HOME, so it's good to make this change now.

          To test, I built tarballs for each project, unpacked them in separate locations, and successfully ran a job in pseudo-distributed mode.

          Show
          Tom White added a comment - Set of three patches (see also HADOOP-6794 , MAPREDUCE-1033 ) to move config and script files per previous comment. Also: Fixes script permissions in HDFS and MapReduce Fixes a name collision in hadoop-config.sh Fixed hdfs-config.sh, mapred-config.sh to define HADOOP_HDFS_HOME, HADOOP_MAPRED_HOME (respectively), which is needed to run in separate install locations. (See also HADOOP-6342 , which will make it easy to install in one location.) Changes HADOOP_CORE_HOME to HADOOP_COMMON_HOME. There has been no release with HADOOP_CORE_HOME, so it's good to make this change now. To test, I built tarballs for each project, unpacked them in separate locations, and successfully ran a job in pseudo-distributed mode.
          Hide
          Tom White added a comment -

          Files that should be in mapreduce/conf

          capacity-scheduler.xml
          capacity-scheduler.xml.template
          fair-scheduler.xml
          fair-scheduler.xml.template
          mapred-queue-acls.xml
          mapred-queue-acls.xml.template
          mapred-site.xml
          mapred-site.xml.template
          taskcontroller.cfg
          

          Files that should be in hdfs/conf

          hdfs-site.xml
          hdfs-site.xml.template
          

          We could add configuration.xsl too (i.e. have it in all conf directories). Does this look right?

          Also, the bin scripts are all in common - we should move these too.

          hdfs/bin:

          hdfs
          hdfs-config.sh
          start-balancer.sh
          start-dfs.sh
          stop-dfs.sh
          

          mapreduce/bin:

          mapred
          mapred-config.sh
          start-mapred.sh
          stop-mapred.sh
          
          Show
          Tom White added a comment - Files that should be in mapreduce/conf capacity-scheduler.xml capacity-scheduler.xml.template fair-scheduler.xml fair-scheduler.xml.template mapred-queue-acls.xml mapred-queue-acls.xml.template mapred-site.xml mapred-site.xml.template taskcontroller.cfg Files that should be in hdfs/conf hdfs-site.xml hdfs-site.xml.template We could add configuration.xsl too (i.e. have it in all conf directories). Does this look right? Also, the bin scripts are all in common - we should move these too. hdfs/bin: hdfs hdfs-config.sh start-balancer.sh start-dfs.sh stop-dfs.sh mapreduce/bin: mapred mapred-config.sh start-mapred.sh stop-mapred.sh
          Hide
          Vinod Kumar Vavilapalli added a comment -

          Tom and Owen, I took the liberty of adding you guys to the watch list so that this issue can move forward..

          Show
          Vinod Kumar Vavilapalli added a comment - Tom and Owen, I took the liberty of adding you guys to the watch list so that this issue can move forward..
          Hide
          Vinod Kumar Vavilapalli added a comment -

          Irrespective of what is going to be decided here, one thing is clear. Some of the conf files have changed in the individual projects - I know of issues in mapreduce which changed 3-4 conf files and conf templates, by the time of this comment. We need to make sure these changes are propagated to the right place when 0.21 is released.

          Show
          Vinod Kumar Vavilapalli added a comment - Irrespective of what is going to be decided here, one thing is clear. Some of the conf files have changed in the individual projects - I know of issues in mapreduce which changed 3-4 conf files and conf templates, by the time of this comment. We need to make sure these changes are propagated to the right place when 0.21 is released.

            People

            • Assignee:
              Tom White
              Reporter:
              Vinod Kumar Vavilapalli
            • Votes:
              0 Vote for this issue
              Watchers:
              7 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development