Hadoop Common
  1. Hadoop Common
  2. HADOOP-7563

hadoop-config.sh setup CLASSPATH, HADOOP_HDFS_HOME and HADOOP_MAPRED_HOME incorrectly

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 0.23.0
    • Fix Version/s: 0.23.0
    • Component/s: scripts
    • Labels:
      None

      Description

      HADOOP_HDFS_HOME and HADOOP_MAPRED_HOME was set to HADOOP_PREFIX/share/hadoop/hdfs and HADOOP_PREFIX/share/hadoop/mapreduce. This setup confuses the location of hdfs and mapred scripts. Instead the script should look for hdfs and mapred script in HADOOP_PREFIX/bin.

      1. HADOOP-7563-5.patch
        19 kB
        Eric Yang
      2. HADOOP-7563-4.patch
        20 kB
        Eric Yang
      3. HADOOP-7563-3.patch
        19 kB
        Eric Yang
      4. HADOOP-7563-2.patch
        19 kB
        Eric Yang
      5. HADOOP-7563-1.patch
        8 kB
        Eric Yang
      6. HADOOP-7563.patch
        0.8 kB
        Eric Yang

        Issue Links

          Activity

          Hide
          Eric Yang added a comment -

          Aligned HADOOP_HDFS_HOME and HADOOP_MAPRED_HOME with HADOOP_PREFIX.

          Show
          Eric Yang added a comment - Aligned HADOOP_HDFS_HOME and HADOOP_MAPRED_HOME with HADOOP_PREFIX.
          Hide
          Eli Collins added a comment -

          +1 lgtm. Testing?

          Show
          Eli Collins added a comment - +1 lgtm. Testing?
          Hide
          Eric Yang added a comment -

          There is some issue in the script for resolving CLASSPATH for HDFS and MAPRED.

          Show
          Eric Yang added a comment - There is some issue in the script for resolving CLASSPATH for HDFS and MAPRED.
          Hide
          Eric Yang added a comment -

          Updated script to locate HDFS and MAPRED in their respective location. Removed legacy layout of running from source code.

          Show
          Eric Yang added a comment - Updated script to locate HDFS and MAPRED in their respective location. Removed legacy layout of running from source code.
          Hide
          Aaron T. Myers added a comment -

          I tested this out by building a dist layout and setting HADOOP_COMMON_HOME and HADOOP_HDFS_HOME to the resulting directories. When applied with the patch from HDFS-2277, this patch seems to have fixed the issues with finding the hadoop-config.sh file, and HDFS-2277 seems to have fixed the HDFS classpath to find the HDFS jars. However, I now get this error when I try to start at NN:

          java.lang.NoClassDefFoundError: org/apache/hadoop/alfredo/util/KerberosName
          	at java.lang.ClassLoader.defineClass1(Native Method)
          	at java.lang.ClassLoader.defineClassCond(ClassLoader.java:631)
          	at java.lang.ClassLoader.defineClass(ClassLoader.java:615)
          	at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:141)
          	at java.net.URLClassLoader.defineClass(URLClassLoader.java:283)
          	at java.net.URLClassLoader.access$000(URLClassLoader.java:58)
          	at java.net.URLClassLoader$1.run(URLClassLoader.java:197)
          	at java.security.AccessController.doPrivileged(Native Method)
          	at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
          	at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
          	at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
          	at java.lang.ClassLoader.loadClass(ClassLoader.java:247)
          	at org.apache.hadoop.security.UserGroupInformation.initialize(UserGroupInformation.java:197)
          	at org.apache.hadoop.security.UserGroupInformation.setConfiguration(UserGroupInformation.java:254)
          	at org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:411)
          	at org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:566)
          	at org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:558)
          	at org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1546)
          	at org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1586)
          Caused by: java.lang.ClassNotFoundException: org.apache.hadoop.alfredo.util.KerberosName
          	at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
          	at java.security.AccessController.doPrivileged(Native Method)
          	at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
          	at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
          	at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
          	at java.lang.ClassLoader.loadClass(ClassLoader.java:247)
          

          Once this is fixed, I think these patches are ready to go.

          However, at this point the scopes of both this JIRA and HDFS-2277 have creeped somewhat. This JIRA is now addressing more than just setting HADOOP_HDFS_HOME and HADOOP_MAPRED_HOME appropriately, and HDFS-2277 is doing more than just fixing the classpath to include the HDFS jars. Since these two JIRAs seem intimately related, I suggest we make all of the needed changes (to both Common and HDFS) in this JIRA, change the description of this JIRA to something like "fix hadoop scripts to reference each other and set up classpaths correctly," and resolve HDFS-2277 as a duplicate.

          Show
          Aaron T. Myers added a comment - I tested this out by building a dist layout and setting HADOOP_COMMON_HOME and HADOOP_HDFS_HOME to the resulting directories. When applied with the patch from HDFS-2277 , this patch seems to have fixed the issues with finding the hadoop-config.sh file, and HDFS-2277 seems to have fixed the HDFS classpath to find the HDFS jars. However, I now get this error when I try to start at NN: java.lang.NoClassDefFoundError: org/apache/hadoop/alfredo/util/KerberosName at java.lang.ClassLoader.defineClass1(Native Method) at java.lang.ClassLoader.defineClassCond(ClassLoader.java:631) at java.lang.ClassLoader.defineClass(ClassLoader.java:615) at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:141) at java.net.URLClassLoader.defineClass(URLClassLoader.java:283) at java.net.URLClassLoader.access$000(URLClassLoader.java:58) at java.net.URLClassLoader$1.run(URLClassLoader.java:197) at java.security.AccessController.doPrivileged(Native Method) at java.net.URLClassLoader.findClass(URLClassLoader.java:190) at java.lang.ClassLoader.loadClass(ClassLoader.java:306) at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301) at java.lang.ClassLoader.loadClass(ClassLoader.java:247) at org.apache.hadoop.security.UserGroupInformation.initialize(UserGroupInformation.java:197) at org.apache.hadoop.security.UserGroupInformation.setConfiguration(UserGroupInformation.java:254) at org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:411) at org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:566) at org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:558) at org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1546) at org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1586) Caused by: java.lang.ClassNotFoundException: org.apache.hadoop.alfredo.util.KerberosName at java.net.URLClassLoader$1.run(URLClassLoader.java:202) at java.security.AccessController.doPrivileged(Native Method) at java.net.URLClassLoader.findClass(URLClassLoader.java:190) at java.lang.ClassLoader.loadClass(ClassLoader.java:306) at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301) at java.lang.ClassLoader.loadClass(ClassLoader.java:247) Once this is fixed, I think these patches are ready to go. However, at this point the scopes of both this JIRA and HDFS-2277 have creeped somewhat. This JIRA is now addressing more than just setting HADOOP_HDFS_HOME and HADOOP_MAPRED_HOME appropriately, and HDFS-2277 is doing more than just fixing the classpath to include the HDFS jars. Since these two JIRAs seem intimately related, I suggest we make all of the needed changes (to both Common and HDFS) in this JIRA, change the description of this JIRA to something like "fix hadoop scripts to reference each other and set up classpaths correctly," and resolve HDFS-2277 as a duplicate.
          Hide
          Eric Yang added a comment -

          The missing class in from hadoop-alfredo module. Alejandro has made hadoop-common depends on hadoop-alfredo, but there is no script to package hadoop-alfredo-0.23.0-SNAPSHOT.jar file into hadoop-common-0.23.0-SNAPSHOT/share/hadoop/common. If you manually copy it over there, then it is possible to use the scripts to start HDFS in both merged layout as well as separated layout using HADOOP_COMMON_HOME and HADOOP_HDFS_HOME environment variables. Fixing alfredo is in HADOOP-7560. I think this jira and HDFS-2277 are ready to be committed to contain the scope of the changes and keep things clean.

          Show
          Eric Yang added a comment - The missing class in from hadoop-alfredo module. Alejandro has made hadoop-common depends on hadoop-alfredo, but there is no script to package hadoop-alfredo-0.23.0-SNAPSHOT.jar file into hadoop-common-0.23.0-SNAPSHOT/share/hadoop/common. If you manually copy it over there, then it is possible to use the scripts to start HDFS in both merged layout as well as separated layout using HADOOP_COMMON_HOME and HADOOP_HDFS_HOME environment variables. Fixing alfredo is in HADOOP-7560 . I think this jira and HDFS-2277 are ready to be committed to contain the scope of the changes and keep things clean.
          Hide
          Alejandro Abdelnur added a comment -

          I think I know the problem, the assembly is excluding hadoop-* JARs from going into lib.

          Currently Hadoop JARs are special and they don't go under lib/, there was a discussion going on in the alias a while back on this.

          Given that the dist is created in a complete different way now with Maven, and running from source is a bit different, I think the cleanest way (for assembly and from scripts) would be to put all JARs (Hadoop included) under lib/.

          The release-notes would clearly state this change.

          I'll verify this later this morning and I'll follow up with a JIRA.

          Show
          Alejandro Abdelnur added a comment - I think I know the problem, the assembly is excluding hadoop-* JARs from going into lib. Currently Hadoop JARs are special and they don't go under lib/, there was a discussion going on in the alias a while back on this. Given that the dist is created in a complete different way now with Maven, and running from source is a bit different, I think the cleanest way (for assembly and from scripts) would be to put all JARs (Hadoop included) under lib/. The release-notes would clearly state this change. I'll verify this later this morning and I'll follow up with a JIRA.
          Hide
          Eric Yang added a comment -

          Alejandro, it is not maven prevents jar to go in lib. The problem is in hadoop-common pom.xml, the dependency is expressed as <scope>compiled</scope>. Since hadoop-alfredo is required at run time, by removing <scope>compiled</scope> should solves hadoop-alfredo CLASSPATH problem. In the assemblies/hadoop-dist.xml, it should not <exclude>org.apache.hadoop:hadoop-*:jar</exclude>, if the dependencies are expressed properly in hadoop-[hdfs|mapreduce]/pom.xml.

          Show
          Eric Yang added a comment - Alejandro, it is not maven prevents jar to go in lib. The problem is in hadoop-common pom.xml, the dependency is expressed as <scope>compiled</scope>. Since hadoop-alfredo is required at run time, by removing <scope>compiled</scope> should solves hadoop-alfredo CLASSPATH problem. In the assemblies/hadoop-dist.xml, it should not <exclude>org.apache.hadoop:hadoop-*:jar</exclude>, if the dependencies are expressed properly in hadoop- [hdfs|mapreduce] /pom.xml.
          Hide
          Alejandro Abdelnur added a comment -

          Eric, the compiled scope in Maven actually means that the JAR will be bundled with the application, it is a misleading name but I guess it stuck for historical reasons.

          The assemblies are doing a funny logic to exclude/include JARs in lib, the purpose of using wildcard instead the concrete artifacts to exclude/include is that the same assembly is used by common & hdfs (as per your feedback), and eventually it should be leveraged by mapreduce.

          Currently the assembly does:

            <fileSets>
              ...
              <fileSet>
                <directory>${project.build.directory}</directory>
                <outputDirectory>/share/hadoop/${hadoop.component}</outputDirectory>
                <includes>
                  <include>${project.artifactId}-${project.version}.jar</include>
                  <include>${project.artifactId}-${project.version}-tests.jar</include>
                  <include>${project.artifactId}-${project.version}-sources.jar</include>
                  <include>${project.artifactId}-${project.version}-test-sources.jar</include>
                </includes>
              </fileSet>
              ...
            </fileSets>
            <dependencySets>
              <dependencySet>
                <outputDirectory>/share/hadoop/${hadoop.component}/lib</outputDirectory>
                <unpack>false</unpack>
                <scope>runtime</scope>
                <useProjectArtifact>false</useProjectArtifact>
                <excludes>
                  <exclude>org.apache.ant:*:jar</exclude>
                  <exclude>org.apache.hadoop:hadoop-*:jar</exclude>
                  <exclude>jdiff:jdiff:jar</exclude>
                </excludes>
              </dependencySet>
            </dependencySets>
          

          IMO, the assembly should do:

            <fileSets>
              ...
            </fileSets>
            <dependencySets>
              <dependencySet>
                <outputDirectory>/share/hadoop/${hadoop.component}/lib</outputDirectory>
                <unpack>false</unpack>
                <scope>runtime</scope>
                <useProjectArtifact>false</useProjectArtifact>
                <excludes>
                  <exclude>org.apache.ant:*:jar</exclude>
                  <exclude>jdiff:jdiff:jar</exclude>
                </excludes>
              </dependencySet>
            </dependencySets>
          

          What's different?

          • Hadoop JARs don't have an special handling, they end up in lib/
          • Hadoop 'tests' JAR is not bundled with the distribution
          • Hadoop 'sources' JARs are not bundled with the distribution

          (*: they are all published to Apache Maven repository, and IDEs download them from there automatically, the IDE will not look at the distribution location)

          Show
          Alejandro Abdelnur added a comment - Eric, the compiled scope in Maven actually means that the JAR will be bundled with the application, it is a misleading name but I guess it stuck for historical reasons. The assemblies are doing a funny logic to exclude/include JARs in lib, the purpose of using wildcard instead the concrete artifacts to exclude/include is that the same assembly is used by common & hdfs (as per your feedback), and eventually it should be leveraged by mapreduce. Currently the assembly does: <fileSets> ... <fileSet> <directory>${project.build.directory}</directory> <outputDirectory>/share/hadoop/${hadoop.component}</outputDirectory> <includes> <include>${project.artifactId}-${project.version}.jar</include> <include>${project.artifactId}-${project.version}-tests.jar</include> <include>${project.artifactId}-${project.version}-sources.jar</include> <include>${project.artifactId}-${project.version}-test-sources.jar</include> </includes> </fileSet> ... </fileSets> <dependencySets> <dependencySet> <outputDirectory>/share/hadoop/${hadoop.component}/lib</outputDirectory> <unpack> false </unpack> <scope>runtime</scope> <useProjectArtifact> false </useProjectArtifact> <excludes> <exclude>org.apache.ant:*:jar</exclude> <exclude>org.apache.hadoop:hadoop-*:jar</exclude> <exclude>jdiff:jdiff:jar</exclude> </excludes> </dependencySet> </dependencySets> IMO, the assembly should do: <fileSets> ... </fileSets> <dependencySets> <dependencySet> <outputDirectory>/share/hadoop/${hadoop.component}/lib</outputDirectory> <unpack> false </unpack> <scope>runtime</scope> <useProjectArtifact> false </useProjectArtifact> <excludes> <exclude>org.apache.ant:*:jar</exclude> <exclude>jdiff:jdiff:jar</exclude> </excludes> </dependencySet> </dependencySets> What's different? Hadoop JARs don't have an special handling, they end up in lib/ Hadoop 'tests' JAR is not bundled with the distribution Hadoop 'sources' JARs are not bundled with the distribution (*: they are all published to Apache Maven repository, and IDEs download them from there automatically, the IDE will not look at the distribution location)
          Hide
          Alejandro Abdelnur added a comment -

          (enter too soon)

          In addition, the hadoop-config.sh script would just add to the classpath generation all JARs under lib/*, the logic will be easier.

          Show
          Alejandro Abdelnur added a comment - (enter too soon) In addition, the hadoop-config.sh script would just add to the classpath generation all JARs under lib/*, the logic will be easier.
          Hide
          Ravi Prakash added a comment -

          Maybe this is too late and perhaps it needs a new JIRA but while we are at it, can we please get rid of the ugly for loops for populating jar files into CLASSPATH? http://download.oracle.com/javase/6/docs/technotes/tools/windows/classpath.html . If we only included <directory>/* , all the jars under that directory will be picked up. Why did we have to write these for loops?

          Show
          Ravi Prakash added a comment - Maybe this is too late and perhaps it needs a new JIRA but while we are at it, can we please get rid of the ugly for loops for populating jar files into CLASSPATH? http://download.oracle.com/javase/6/docs/technotes/tools/windows/classpath.html . If we only included <directory>/* , all the jars under that directory will be picked up. Why did we have to write these for loops?
          Hide
          Alejandro Abdelnur added a comment -

          @Ravi, I believe the loops are a leftover from older version of Java (prior to 1.6) which did not handle all JARs in a dir.

          As you suggest, what could be done to simplify things is:

          CLASSPATH="${HADOOP_CONF_DIR}"
          CLASSPATH=${CLASSPATH}:$HADOOP_PREFIX
          CLASSPATH=${CLASSPATH}:"$HADOOP_PREFIX/share/hadoop/common/lib"'/*'
          CLASSPATH=${CLASSPATH}:"$HADOOP_PREFIX/share/hadoop/hdfs/lib"'/*'
          CLASSPATH=${CLASSPATH}:"$HADOOP_PREFIX/share/hadoop/mapreduce/lib"'/*'
          
          Show
          Alejandro Abdelnur added a comment - @Ravi, I believe the loops are a leftover from older version of Java (prior to 1.6) which did not handle all JARs in a dir. As you suggest, what could be done to simplify things is: CLASSPATH= "${HADOOP_CONF_DIR}" CLASSPATH=${CLASSPATH}:$HADOOP_PREFIX CLASSPATH=${CLASSPATH}: "$HADOOP_PREFIX/share/hadoop/common/lib" '/*' CLASSPATH=${CLASSPATH}: "$HADOOP_PREFIX/share/hadoop/hdfs/lib" '/*' CLASSPATH=${CLASSPATH}: "$HADOOP_PREFIX/share/hadoop/mapreduce/lib" '/*'
          Hide
          Eric Yang added a comment -

          I will update the patch with wild card. For hadoop-common not to be in hadoop-hdfs, or hadoop-mapreduce. Shouldn't it be marked as <scope>provided</scope>?

          Show
          Eric Yang added a comment - I will update the patch with wild card. For hadoop-common not to be in hadoop-hdfs, or hadoop-mapreduce. Shouldn't it be marked as <scope>provided</scope>?
          Hide
          Alejandro Abdelnur added a comment -

          @Eric, are you also modifying the dist assembly as per my previous comment?

          Regarding the provided scope, it all depends how we finally package Hadoop, as 3 different system layouts (common, hdfs & mapreduce) or as a single system layouts (hadoop). If the former, yes use provided, if the later, we should use compile and an assembly would stich common/hdfs/mapreduce into as a single system layout (that will be much easier and correct than using a script ot stich tars).

          But the second paragraph should be a separate JIRA, no?

          Show
          Alejandro Abdelnur added a comment - @Eric, are you also modifying the dist assembly as per my previous comment? Regarding the provided scope, it all depends how we finally package Hadoop, as 3 different system layouts (common, hdfs & mapreduce) or as a single system layouts (hadoop). If the former, yes use provided , if the later, we should use compile and an assembly would stich common/hdfs/mapreduce into as a single system layout (that will be much easier and correct than using a script ot stich tars). But the second paragraph should be a separate JIRA, no?
          Hide
          Eric Yang added a comment -

          But the second paragraph should be a separate JIRA, no?

          Agree, my patch will not contain changes for second paragraph in your comment. HADOOP-7560 is the correct place to fix the scope.

          Show
          Eric Yang added a comment - But the second paragraph should be a separate JIRA, no? Agree, my patch will not contain changes for second paragraph in your comment. HADOOP-7560 is the correct place to fix the scope.
          Hide
          Eric Yang added a comment -

          Per Aaron's suggestion, mark HDFS-2277 as part of this JIRA.

          Show
          Eric Yang added a comment - Per Aaron's suggestion, mark HDFS-2277 as part of this JIRA.
          Hide
          Eric Yang added a comment -

          Removed for loop to construct CLASSPATH, and integrate HDFS-2277 as part of this patch.

          Show
          Eric Yang added a comment - Removed for loop to construct CLASSPATH, and integrate HDFS-2277 as part of this patch.
          Hide
          Alejandro Abdelnur added a comment -

          Comments on '-2' patch:

          Where you are doing:

          
          if [ -d "$HADOOP_PREFIX/share/hadoop/common/lib" ]; then
            CLASSPATH=${CLASSPATH}:$HADOOP_PREFIX/share/hadoop/common/lib/*;
          fi
          
          CLASSPATH=${CLASSPATH}:$HADOOP_PREFIX/share/hadoop/common/*.jar;
          

          For the first CLASSPATH=, you have to do '/*' to prevent the shell from resolving the wildcard. 'java' does special handing.

          For the second CLASSPATH=, I don't that will be correctly resolved. There are multiple JARs at common/ level, and some of the them are test classes and sources. You don't want them in the classpath.

          IMO, the correct fix is to change the assembly as I've pointed out before (to have the hadoop-* artifact JAR under lib) and then do:

            CLASSPATH=${CLASSPATH}:$HADOOP_PREFIX/share/hadoop/common/lib'/*';
          
          Show
          Alejandro Abdelnur added a comment - Comments on '-2' patch: Where you are doing: if [ -d "$HADOOP_PREFIX/share/hadoop/common/lib" ]; then CLASSPATH=${CLASSPATH}:$HADOOP_PREFIX/share/hadoop/common/lib/*; fi CLASSPATH=${CLASSPATH}:$HADOOP_PREFIX/share/hadoop/common/*.jar; For the first CLASSPATH=, you have to do '/*' to prevent the shell from resolving the wildcard. 'java' does special handing. For the second CLASSPATH=, I don't that will be correctly resolved. There are multiple JARs at common/ level, and some of the them are test classes and sources. You don't want them in the classpath. IMO, the correct fix is to change the assembly as I've pointed out before (to have the hadoop-* artifact JAR under lib) and then do: CLASSPATH=${CLASSPATH}:$HADOOP_PREFIX/share/hadoop/common/lib'/*';
          Hide
          Aaron T. Myers added a comment -

          Thanks a lot for addressing my comments, Eric. Would you mind also changing the description of this JIRA to more accurately reflect the increased scope?

          Show
          Aaron T. Myers added a comment - Thanks a lot for addressing my comments, Eric. Would you mind also changing the description of this JIRA to more accurately reflect the increased scope?
          Hide
          Eric Yang added a comment -

          Updated to reflect increased scope.

          Show
          Eric Yang added a comment - Updated to reflect increased scope.
          Hide
          Alejandro Abdelnur added a comment -

          Eric, the latest patch does not have the changes to the assembly I'm suggesting, nor I see a comment against them. Are you planning to integrate them?

          Show
          Alejandro Abdelnur added a comment - Eric, the latest patch does not have the changes to the assembly I'm suggesting, nor I see a comment against them. Are you planning to integrate them?
          Hide
          Eric Yang added a comment -

          Alejandro, that was my suggestion to remove the exclude line on 23/Aug/11 16:15. At 23/Aug/11 19:57, I agreed with you for your comment on 23/Aug/11 18:29, the fix for assembly does not belong to this jira. Am I missing something?

          Show
          Eric Yang added a comment - Alejandro, that was my suggestion to remove the exclude line on 23/Aug/11 16:15. At 23/Aug/11 19:57, I agreed with you for your comment on 23/Aug/11 18:29, the fix for assembly does not belong to this jira. Am I missing something?
          Hide
          Eric Yang added a comment -

          Quoted wildcard to prevent shell expansion.

          Show
          Eric Yang added a comment - Quoted wildcard to prevent shell expansion.
          Hide
          Eric Yang added a comment -

          For the second CLASSPATH=, I don't that will be correctly resolved. There are multiple JARs at common/ level, and some of the them are test classes and sources. You don't want them in the classpath.

          IMO, the correct fix is to change the assembly as I've pointed out before (to have the hadoop-* artifact JAR under lib) and then do:

          CLASSPATH=${CLASSPATH}:$HADOOP_PREFIX/share/hadoop/common/lib'/*';
          

          It makes no difference because all Hadoop jars would be in common/lib'/' instead of having hadoop-common-[version].jar in common/lib'/'. It may make sense to move test jar to common/test, and javadoc to common/javadoc instead. However, that should be a separated JIRA to fix that issue.

          Show
          Eric Yang added a comment - For the second CLASSPATH=, I don't that will be correctly resolved. There are multiple JARs at common/ level, and some of the them are test classes and sources. You don't want them in the classpath. IMO, the correct fix is to change the assembly as I've pointed out before (to have the hadoop-* artifact JAR under lib) and then do: CLASSPATH=${CLASSPATH}:$HADOOP_PREFIX/share/hadoop/common/lib'/*'; It makes no difference because all Hadoop jars would be in common/lib'/ ' instead of having hadoop-common- [version] .jar in common/lib'/ '. It may make sense to move test jar to common/test, and javadoc to common/javadoc instead. However, that should be a separated JIRA to fix that issue.
          Hide
          Alejandro Abdelnur added a comment -

          why do we need the sources and tests JARs in the distribution? They are published to Maven central, Maven/IDEs download it from there.

          And for the full source, we could do as most projects do, the publish a SRC TAR with an image of the the buildable source.

          Show
          Alejandro Abdelnur added a comment - why do we need the sources and tests JARs in the distribution? They are published to Maven central, Maven/IDEs download it from there. And for the full source, we could do as most projects do, the publish a SRC TAR with an image of the the buildable source.
          Hide
          Eric Yang added a comment -

          source jar is for remote debugging. It is important to keep running jar and source jar in the same place to ensure the debugging session is showing the right source code, and having test jar would help to validate the cluster like teragen, and terasort. Hence, having them loaded to the class path are nice to have.

          Show
          Eric Yang added a comment - source jar is for remote debugging. It is important to keep running jar and source jar in the same place to ensure the debugging session is showing the right source code, and having test jar would help to validate the cluster like teragen, and terasort. Hence, having them loaded to the class path are nice to have.
          Hide
          Alejandro Abdelnur added a comment -

          OK, but having them at common/ level is a bit confusing, wouldn't make sense to have a common/developer/ directory for those JARs, so it is clear they don't have anything to do with the runtime?

          Show
          Alejandro Abdelnur added a comment - OK, but having them at common/ level is a bit confusing, wouldn't make sense to have a common/developer/ directory for those JARs, so it is clear they don't have anything to do with the runtime?
          Hide
          Eric Caspole added a comment -

          Will this patch restore the HADOOP_COMMON_HOME/HADOOP_HDFS_HOME etc shell variables so I can untar the distro files I built with "mvn clean install -DskipTests=true -Dtar -Pdist" into separate directories and everything will work like it used to? Or is there a necessary "munge all into one directory step" to make a working system? I can't figure out how to run the trunk build and these scripts seem to be the issue. Thanks for any advice.

          Show
          Eric Caspole added a comment - Will this patch restore the HADOOP_COMMON_HOME/HADOOP_HDFS_HOME etc shell variables so I can untar the distro files I built with "mvn clean install -DskipTests=true -Dtar -Pdist" into separate directories and everything will work like it used to? Or is there a necessary "munge all into one directory step" to make a working system? I can't figure out how to run the trunk build and these scripts seem to be the issue. Thanks for any advice.
          Hide
          Aaron T. Myers added a comment -

          Will this patch restore the HADOOP_COMMON_HOME/HADOOP_HDFS_HOME etc shell variables so I can untar the distro files I built with "mvn clean install -DskipTests=true -Dtar -Pdist" into separate directories and everything will work like it used to?

          Yes it will. With this patch, plus something to address the Alfredo JAR issue, I've been successfully able to run HDFS trunk just by doing `mvn clean package -DskipTests -Pdist' and then pointing HADOOP_COMMON_HOME and HADOOP_HDFS_HOME at the resulting directories, with no intermediate step of merging the contents of those directories.

          Show
          Aaron T. Myers added a comment - Will this patch restore the HADOOP_COMMON_HOME/HADOOP_HDFS_HOME etc shell variables so I can untar the distro files I built with "mvn clean install -DskipTests=true -Dtar -Pdist" into separate directories and everything will work like it used to? Yes it will. With this patch, plus something to address the Alfredo JAR issue, I've been successfully able to run HDFS trunk just by doing ` mvn clean package -DskipTests -Pdist ' and then pointing HADOOP_COMMON_HOME and HADOOP_HDFS_HOME at the resulting directories, with no intermediate step of merging the contents of those directories.
          Hide
          Arun C Murthy added a comment -

          Aaron, are you happy with this patch? Should I commit?

          Show
          Arun C Murthy added a comment - Aaron, are you happy with this patch? Should I commit?
          Hide
          Alejandro Abdelnur added a comment -

          Patch looks good.

          Still does not address the problem of hadoop-alfredo JAR nto being picked up (cuase's it is being excluded) and that the hadoop-*.jar should move to lib/. But it was agreed we'll do that a as follow up JIRA.

          Show
          Alejandro Abdelnur added a comment - Patch looks good. Still does not address the problem of hadoop-alfredo JAR nto being picked up (cuase's it is being excluded) and that the hadoop-*.jar should move to lib/. But it was agreed we'll do that a as follow up JIRA.
          Hide
          Aaron T. Myers added a comment -

          +1, the latest patch looks good to me.

          Though this patch is definitely a big step in the right direction, I'd like to see the Alfredo JAR issue fixed promptly, since even with this patch a user still needs to do the manual copy/paste of that JAR in order to run from the dist layout.

          Has a JIRA been filed to address that yet? Eric originally said HADOOP-7560 would do so as a side effect, but Alejandro then said that it would not.

          Show
          Aaron T. Myers added a comment - +1, the latest patch looks good to me. Though this patch is definitely a big step in the right direction, I'd like to see the Alfredo JAR issue fixed promptly, since even with this patch a user still needs to do the manual copy/paste of that JAR in order to run from the dist layout. Has a JIRA been filed to address that yet? Eric originally said HADOOP-7560 would do so as a side effect, but Alejandro then said that it would not.
          Hide
          Arun C Murthy added a comment -

          Thanks Aaron.

          Eric - if it isn't a biggie, can you please fix the excludes so that the alfredo jar issue is fixed too? Thanks!

          Show
          Arun C Murthy added a comment - Thanks Aaron. Eric - if it isn't a biggie, can you please fix the excludes so that the alfredo jar issue is fixed too? Thanks!
          Hide
          Eric Yang added a comment -

          I will fix alfredo jar issue. Mark HADOOP-7560 as dependent on this issue.

          Show
          Eric Yang added a comment - I will fix alfredo jar issue. Mark HADOOP-7560 as dependent on this issue.
          Hide
          Eric Yang added a comment -

          Updated patch to include assembly packaging fix.

          Show
          Eric Yang added a comment - Updated patch to include assembly packaging fix.
          Hide
          Eric Yang added a comment -

          Instruction for running in independent layout as well as merged layout.

          For independent layout:

          mv clean package -Pdist -Dmaven.test.skip.exec=true
          export HADOOP_COMMON_HOME=`trunk/hadoop-common/target/hadoop-common-0.23.0-SNAPSHOT`
          export HADOOP_HDFS_HOME=`trunk/hadoop-hdfs/target/hadoop-hdfs-0.23.0-SNAPSHOT`
          # edit $HADOOP_COMMON_HOME/etc/hadoop/core-site.xml
          # edit $HADOOP_HDFS_HOME/etc/hadoop/hdfs-site.xml
          cd $HADOOP_HDFS_HOME
          $HADOOP_HDFS_HOME/bin/hdfs namenode -format
          $HADOOP_COMMON_HOME/sbin/hadoop-daemon.sh start namenode
          $HADOOP_COMMON_HOME/sbin/hadoop-daemon.sh start datanode
          

          For merged layout:

          mv clean package -Pdist -Dtar -Dmaven.test.skip.exec=true
          mkdir hadoop
          cd hadoop
          tar xv -skip-compoents=1 -f trunk/hadoop-common/target/hadoop-common*.tar.gz trunk/hadoop-hdfs/target/hadoop-hdfs*.tar.gz
          # edit etc/hadoop/core-site.xml and hdfs-site.xml
          ./bin/hdfs namenode -format
          ./sbin/hadoop-daemon.sh start namenode
          ./sbin/hadoop-daemon.sh start datanode
          
          Show
          Eric Yang added a comment - Instruction for running in independent layout as well as merged layout. For independent layout: mv clean package -Pdist -Dmaven.test.skip.exec=true export HADOOP_COMMON_HOME=`trunk/hadoop-common/target/hadoop-common-0.23.0-SNAPSHOT` export HADOOP_HDFS_HOME=`trunk/hadoop-hdfs/target/hadoop-hdfs-0.23.0-SNAPSHOT` # edit $HADOOP_COMMON_HOME/etc/hadoop/core-site.xml # edit $HADOOP_HDFS_HOME/etc/hadoop/hdfs-site.xml cd $HADOOP_HDFS_HOME $HADOOP_HDFS_HOME/bin/hdfs namenode -format $HADOOP_COMMON_HOME/sbin/hadoop-daemon.sh start namenode $HADOOP_COMMON_HOME/sbin/hadoop-daemon.sh start datanode For merged layout: mv clean package -Pdist -Dtar -Dmaven.test.skip.exec=true mkdir hadoop cd hadoop tar xv -skip-compoents=1 -f trunk/hadoop-common/target/hadoop-common*.tar.gz trunk/hadoop-hdfs/target/hadoop-hdfs*.tar.gz # edit etc/hadoop/core-site.xml and hdfs-site.xml ./bin/hdfs namenode -format ./sbin/hadoop-daemon.sh start namenode ./sbin/hadoop-daemon.sh start datanode
          Hide
          Aaron T. Myers added a comment -

          I applied the patch and tested it and it worked flawlessly - no Alfredo issues, and the NN/DN came right up from an un-merged dist layout.

          One thing I should have noticed earlier: I think some changes to the empty hdfs-site.xml file crept in to this patch by accident. Eric, could you take care of that?

          Alejandro, do the latest maven changes look good to you?

          Show
          Aaron T. Myers added a comment - I applied the patch and tested it and it worked flawlessly - no Alfredo issues, and the NN/DN came right up from an un-merged dist layout. One thing I should have noticed earlier: I think some changes to the empty hdfs-site.xml file crept in to this patch by accident. Eric, could you take care of that? Alejandro, do the latest maven changes look good to you?
          Hide
          Alejandro Abdelnur added a comment -

          Looks good, only NIT is that in hadoop-common/pom.xml the <scope>compile</scope> is being removed. Even if compile is the default, it should stay (else you can have surprises because of a different scope set in the dependenciesManagement section)

          Show
          Alejandro Abdelnur added a comment - Looks good, only NIT is that in hadoop-common/pom.xml the <scope>compile</scope> is being removed. Even if compile is the default, it should stay (else you can have surprises because of a different scope set in the dependenciesManagement section)
          Hide
          Eric Yang added a comment -

          Correction to hadoop-common/pom.xml and accidental include of hdfs-site.xml.

          Show
          Eric Yang added a comment - Correction to hadoop-common/pom.xml and accidental include of hdfs-site.xml.
          Hide
          Aaron T. Myers added a comment -

          +1, v5 looks great to me.

          Thanks a lot for addressing all of mine/Alejandro's comments, Eric.

          Show
          Aaron T. Myers added a comment - +1, v5 looks great to me. Thanks a lot for addressing all of mine/Alejandro's comments, Eric.
          Hide
          Arun C Murthy added a comment -

          I just committed this. Thanks Eric!

          Show
          Arun C Murthy added a comment - I just committed this. Thanks Eric!
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Mapreduce-trunk-Commit #781 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/781/)
          HADOOP-7563. Setup HADOOP_HDFS_HOME, HADOOP_MAPRED_HOME and classpath correction. Contributed by Eric Yang.

          acmurthy : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1161329
          Files :

          • /hadoop/common/trunk/hadoop-hdfs/src/main/bin/refresh-namenodes.sh
          • /hadoop/common/trunk/hadoop-common/src/main/bin/start-all.sh
          • /hadoop/common/trunk/hadoop-hdfs/src/main/bin/distribute-exclude.sh
          • /hadoop/common/trunk/hadoop-common/CHANGES.txt
          • /hadoop/common/trunk/hadoop-mapreduce/pom.xml
          • /hadoop/common/trunk/hadoop-common/src/main/bin/hadoop-daemons.sh
          • /hadoop/common/trunk/hadoop-common/src/main/bin/hadoop-daemon.sh
          • /hadoop/common/trunk/hadoop-common/src/main/bin/hadoop-config.sh
          • /hadoop/common/trunk/hadoop-hdfs/src/main/bin/stop-dfs.sh
          • /hadoop/common/trunk/hadoop-hdfs/src/main/bin/start-dfs.sh
          • /hadoop/common/trunk/hadoop-hdfs/src/main/bin/hdfs-config.sh
          • /hadoop/common/trunk/hadoop-hdfs/src/main/bin/start-balancer.sh
          • /hadoop/common/trunk/hadoop-hdfs/src/main/bin/stop-balancer.sh
          • /hadoop/common/trunk/hadoop-hdfs/src/main/bin/start-secure-dns.sh
          • /hadoop/common/trunk/hadoop-common/src/main/bin/slaves.sh
          • /hadoop/common/trunk/hadoop-hdfs/src/main/bin/stop-secure-dns.sh
          • /hadoop/common/trunk/hadoop-common/src/main/bin/hadoop
          • /hadoop/common/trunk/hadoop-hdfs/src/main/bin/hdfs
          • /hadoop/common/trunk/hadoop-assemblies/src/main/resources/assemblies/hadoop-dist.xml
          • /hadoop/common/trunk/hadoop-common/src/main/bin/stop-all.sh
          • /hadoop/common/trunk/hadoop-common/src/main/bin/rcc
          Show
          Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk-Commit #781 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/781/ ) HADOOP-7563 . Setup HADOOP_HDFS_HOME, HADOOP_MAPRED_HOME and classpath correction. Contributed by Eric Yang. acmurthy : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1161329 Files : /hadoop/common/trunk/hadoop-hdfs/src/main/bin/refresh-namenodes.sh /hadoop/common/trunk/hadoop-common/src/main/bin/start-all.sh /hadoop/common/trunk/hadoop-hdfs/src/main/bin/distribute-exclude.sh /hadoop/common/trunk/hadoop-common/CHANGES.txt /hadoop/common/trunk/hadoop-mapreduce/pom.xml /hadoop/common/trunk/hadoop-common/src/main/bin/hadoop-daemons.sh /hadoop/common/trunk/hadoop-common/src/main/bin/hadoop-daemon.sh /hadoop/common/trunk/hadoop-common/src/main/bin/hadoop-config.sh /hadoop/common/trunk/hadoop-hdfs/src/main/bin/stop-dfs.sh /hadoop/common/trunk/hadoop-hdfs/src/main/bin/start-dfs.sh /hadoop/common/trunk/hadoop-hdfs/src/main/bin/hdfs-config.sh /hadoop/common/trunk/hadoop-hdfs/src/main/bin/start-balancer.sh /hadoop/common/trunk/hadoop-hdfs/src/main/bin/stop-balancer.sh /hadoop/common/trunk/hadoop-hdfs/src/main/bin/start-secure-dns.sh /hadoop/common/trunk/hadoop-common/src/main/bin/slaves.sh /hadoop/common/trunk/hadoop-hdfs/src/main/bin/stop-secure-dns.sh /hadoop/common/trunk/hadoop-common/src/main/bin/hadoop /hadoop/common/trunk/hadoop-hdfs/src/main/bin/hdfs /hadoop/common/trunk/hadoop-assemblies/src/main/resources/assemblies/hadoop-dist.xml /hadoop/common/trunk/hadoop-common/src/main/bin/stop-all.sh /hadoop/common/trunk/hadoop-common/src/main/bin/rcc
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Hdfs-trunk-Commit #856 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/856/)
          HADOOP-7563. Setup HADOOP_HDFS_HOME, HADOOP_MAPRED_HOME and classpath correction. Contributed by Eric Yang.

          acmurthy : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1161329
          Files :

          • /hadoop/common/trunk/hadoop-hdfs/src/main/bin/refresh-namenodes.sh
          • /hadoop/common/trunk/hadoop-common/src/main/bin/start-all.sh
          • /hadoop/common/trunk/hadoop-hdfs/src/main/bin/distribute-exclude.sh
          • /hadoop/common/trunk/hadoop-common/CHANGES.txt
          • /hadoop/common/trunk/hadoop-mapreduce/pom.xml
          • /hadoop/common/trunk/hadoop-common/src/main/bin/hadoop-daemons.sh
          • /hadoop/common/trunk/hadoop-common/src/main/bin/hadoop-daemon.sh
          • /hadoop/common/trunk/hadoop-common/src/main/bin/hadoop-config.sh
          • /hadoop/common/trunk/hadoop-hdfs/src/main/bin/stop-dfs.sh
          • /hadoop/common/trunk/hadoop-hdfs/src/main/bin/start-dfs.sh
          • /hadoop/common/trunk/hadoop-hdfs/src/main/bin/hdfs-config.sh
          • /hadoop/common/trunk/hadoop-hdfs/src/main/bin/start-balancer.sh
          • /hadoop/common/trunk/hadoop-hdfs/src/main/bin/stop-balancer.sh
          • /hadoop/common/trunk/hadoop-hdfs/src/main/bin/start-secure-dns.sh
          • /hadoop/common/trunk/hadoop-common/src/main/bin/slaves.sh
          • /hadoop/common/trunk/hadoop-hdfs/src/main/bin/stop-secure-dns.sh
          • /hadoop/common/trunk/hadoop-common/src/main/bin/hadoop
          • /hadoop/common/trunk/hadoop-hdfs/src/main/bin/hdfs
          • /hadoop/common/trunk/hadoop-assemblies/src/main/resources/assemblies/hadoop-dist.xml
          • /hadoop/common/trunk/hadoop-common/src/main/bin/stop-all.sh
          • /hadoop/common/trunk/hadoop-common/src/main/bin/rcc
          Show
          Hudson added a comment - Integrated in Hadoop-Hdfs-trunk-Commit #856 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/856/ ) HADOOP-7563 . Setup HADOOP_HDFS_HOME, HADOOP_MAPRED_HOME and classpath correction. Contributed by Eric Yang. acmurthy : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1161329 Files : /hadoop/common/trunk/hadoop-hdfs/src/main/bin/refresh-namenodes.sh /hadoop/common/trunk/hadoop-common/src/main/bin/start-all.sh /hadoop/common/trunk/hadoop-hdfs/src/main/bin/distribute-exclude.sh /hadoop/common/trunk/hadoop-common/CHANGES.txt /hadoop/common/trunk/hadoop-mapreduce/pom.xml /hadoop/common/trunk/hadoop-common/src/main/bin/hadoop-daemons.sh /hadoop/common/trunk/hadoop-common/src/main/bin/hadoop-daemon.sh /hadoop/common/trunk/hadoop-common/src/main/bin/hadoop-config.sh /hadoop/common/trunk/hadoop-hdfs/src/main/bin/stop-dfs.sh /hadoop/common/trunk/hadoop-hdfs/src/main/bin/start-dfs.sh /hadoop/common/trunk/hadoop-hdfs/src/main/bin/hdfs-config.sh /hadoop/common/trunk/hadoop-hdfs/src/main/bin/start-balancer.sh /hadoop/common/trunk/hadoop-hdfs/src/main/bin/stop-balancer.sh /hadoop/common/trunk/hadoop-hdfs/src/main/bin/start-secure-dns.sh /hadoop/common/trunk/hadoop-common/src/main/bin/slaves.sh /hadoop/common/trunk/hadoop-hdfs/src/main/bin/stop-secure-dns.sh /hadoop/common/trunk/hadoop-common/src/main/bin/hadoop /hadoop/common/trunk/hadoop-hdfs/src/main/bin/hdfs /hadoop/common/trunk/hadoop-assemblies/src/main/resources/assemblies/hadoop-dist.xml /hadoop/common/trunk/hadoop-common/src/main/bin/stop-all.sh /hadoop/common/trunk/hadoop-common/src/main/bin/rcc
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Common-trunk-Commit #778 (See https://builds.apache.org/job/Hadoop-Common-trunk-Commit/778/)
          HADOOP-7563. Setup HADOOP_HDFS_HOME, HADOOP_MAPRED_HOME and classpath correction. Contributed by Eric Yang.

          acmurthy : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1161329
          Files :

          • /hadoop/common/trunk/hadoop-hdfs/src/main/bin/refresh-namenodes.sh
          • /hadoop/common/trunk/hadoop-common/src/main/bin/start-all.sh
          • /hadoop/common/trunk/hadoop-hdfs/src/main/bin/distribute-exclude.sh
          • /hadoop/common/trunk/hadoop-common/CHANGES.txt
          • /hadoop/common/trunk/hadoop-mapreduce/pom.xml
          • /hadoop/common/trunk/hadoop-common/src/main/bin/hadoop-daemons.sh
          • /hadoop/common/trunk/hadoop-common/src/main/bin/hadoop-daemon.sh
          • /hadoop/common/trunk/hadoop-common/src/main/bin/hadoop-config.sh
          • /hadoop/common/trunk/hadoop-hdfs/src/main/bin/stop-dfs.sh
          • /hadoop/common/trunk/hadoop-hdfs/src/main/bin/start-dfs.sh
          • /hadoop/common/trunk/hadoop-hdfs/src/main/bin/hdfs-config.sh
          • /hadoop/common/trunk/hadoop-hdfs/src/main/bin/start-balancer.sh
          • /hadoop/common/trunk/hadoop-hdfs/src/main/bin/stop-balancer.sh
          • /hadoop/common/trunk/hadoop-hdfs/src/main/bin/start-secure-dns.sh
          • /hadoop/common/trunk/hadoop-common/src/main/bin/slaves.sh
          • /hadoop/common/trunk/hadoop-hdfs/src/main/bin/stop-secure-dns.sh
          • /hadoop/common/trunk/hadoop-common/src/main/bin/hadoop
          • /hadoop/common/trunk/hadoop-hdfs/src/main/bin/hdfs
          • /hadoop/common/trunk/hadoop-assemblies/src/main/resources/assemblies/hadoop-dist.xml
          • /hadoop/common/trunk/hadoop-common/src/main/bin/stop-all.sh
          • /hadoop/common/trunk/hadoop-common/src/main/bin/rcc
          Show
          Hudson added a comment - Integrated in Hadoop-Common-trunk-Commit #778 (See https://builds.apache.org/job/Hadoop-Common-trunk-Commit/778/ ) HADOOP-7563 . Setup HADOOP_HDFS_HOME, HADOOP_MAPRED_HOME and classpath correction. Contributed by Eric Yang. acmurthy : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1161329 Files : /hadoop/common/trunk/hadoop-hdfs/src/main/bin/refresh-namenodes.sh /hadoop/common/trunk/hadoop-common/src/main/bin/start-all.sh /hadoop/common/trunk/hadoop-hdfs/src/main/bin/distribute-exclude.sh /hadoop/common/trunk/hadoop-common/CHANGES.txt /hadoop/common/trunk/hadoop-mapreduce/pom.xml /hadoop/common/trunk/hadoop-common/src/main/bin/hadoop-daemons.sh /hadoop/common/trunk/hadoop-common/src/main/bin/hadoop-daemon.sh /hadoop/common/trunk/hadoop-common/src/main/bin/hadoop-config.sh /hadoop/common/trunk/hadoop-hdfs/src/main/bin/stop-dfs.sh /hadoop/common/trunk/hadoop-hdfs/src/main/bin/start-dfs.sh /hadoop/common/trunk/hadoop-hdfs/src/main/bin/hdfs-config.sh /hadoop/common/trunk/hadoop-hdfs/src/main/bin/start-balancer.sh /hadoop/common/trunk/hadoop-hdfs/src/main/bin/stop-balancer.sh /hadoop/common/trunk/hadoop-hdfs/src/main/bin/start-secure-dns.sh /hadoop/common/trunk/hadoop-common/src/main/bin/slaves.sh /hadoop/common/trunk/hadoop-hdfs/src/main/bin/stop-secure-dns.sh /hadoop/common/trunk/hadoop-common/src/main/bin/hadoop /hadoop/common/trunk/hadoop-hdfs/src/main/bin/hdfs /hadoop/common/trunk/hadoop-assemblies/src/main/resources/assemblies/hadoop-dist.xml /hadoop/common/trunk/hadoop-common/src/main/bin/stop-all.sh /hadoop/common/trunk/hadoop-common/src/main/bin/rcc
          Hide
          Eric Yang added a comment -

          Thank you all.

          Show
          Eric Yang added a comment - Thank you all.
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Mapreduce-trunk #787 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/787/)
          HADOOP-7563. Setup HADOOP_HDFS_HOME, HADOOP_MAPRED_HOME and classpath correction. Contributed by Eric Yang.

          acmurthy : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1161329
          Files :

          • /hadoop/common/trunk/hadoop-hdfs/src/main/bin/refresh-namenodes.sh
          • /hadoop/common/trunk/hadoop-common/src/main/bin/start-all.sh
          • /hadoop/common/trunk/hadoop-hdfs/src/main/bin/distribute-exclude.sh
          • /hadoop/common/trunk/hadoop-common/CHANGES.txt
          • /hadoop/common/trunk/hadoop-mapreduce/pom.xml
          • /hadoop/common/trunk/hadoop-common/src/main/bin/hadoop-daemons.sh
          • /hadoop/common/trunk/hadoop-common/src/main/bin/hadoop-daemon.sh
          • /hadoop/common/trunk/hadoop-common/src/main/bin/hadoop-config.sh
          • /hadoop/common/trunk/hadoop-hdfs/src/main/bin/stop-dfs.sh
          • /hadoop/common/trunk/hadoop-hdfs/src/main/bin/start-dfs.sh
          • /hadoop/common/trunk/hadoop-hdfs/src/main/bin/hdfs-config.sh
          • /hadoop/common/trunk/hadoop-hdfs/src/main/bin/start-balancer.sh
          • /hadoop/common/trunk/hadoop-hdfs/src/main/bin/stop-balancer.sh
          • /hadoop/common/trunk/hadoop-hdfs/src/main/bin/start-secure-dns.sh
          • /hadoop/common/trunk/hadoop-common/src/main/bin/slaves.sh
          • /hadoop/common/trunk/hadoop-hdfs/src/main/bin/stop-secure-dns.sh
          • /hadoop/common/trunk/hadoop-common/src/main/bin/hadoop
          • /hadoop/common/trunk/hadoop-hdfs/src/main/bin/hdfs
          • /hadoop/common/trunk/hadoop-assemblies/src/main/resources/assemblies/hadoop-dist.xml
          • /hadoop/common/trunk/hadoop-common/src/main/bin/stop-all.sh
          • /hadoop/common/trunk/hadoop-common/src/main/bin/rcc
          Show
          Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk #787 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/787/ ) HADOOP-7563 . Setup HADOOP_HDFS_HOME, HADOOP_MAPRED_HOME and classpath correction. Contributed by Eric Yang. acmurthy : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1161329 Files : /hadoop/common/trunk/hadoop-hdfs/src/main/bin/refresh-namenodes.sh /hadoop/common/trunk/hadoop-common/src/main/bin/start-all.sh /hadoop/common/trunk/hadoop-hdfs/src/main/bin/distribute-exclude.sh /hadoop/common/trunk/hadoop-common/CHANGES.txt /hadoop/common/trunk/hadoop-mapreduce/pom.xml /hadoop/common/trunk/hadoop-common/src/main/bin/hadoop-daemons.sh /hadoop/common/trunk/hadoop-common/src/main/bin/hadoop-daemon.sh /hadoop/common/trunk/hadoop-common/src/main/bin/hadoop-config.sh /hadoop/common/trunk/hadoop-hdfs/src/main/bin/stop-dfs.sh /hadoop/common/trunk/hadoop-hdfs/src/main/bin/start-dfs.sh /hadoop/common/trunk/hadoop-hdfs/src/main/bin/hdfs-config.sh /hadoop/common/trunk/hadoop-hdfs/src/main/bin/start-balancer.sh /hadoop/common/trunk/hadoop-hdfs/src/main/bin/stop-balancer.sh /hadoop/common/trunk/hadoop-hdfs/src/main/bin/start-secure-dns.sh /hadoop/common/trunk/hadoop-common/src/main/bin/slaves.sh /hadoop/common/trunk/hadoop-hdfs/src/main/bin/stop-secure-dns.sh /hadoop/common/trunk/hadoop-common/src/main/bin/hadoop /hadoop/common/trunk/hadoop-hdfs/src/main/bin/hdfs /hadoop/common/trunk/hadoop-assemblies/src/main/resources/assemblies/hadoop-dist.xml /hadoop/common/trunk/hadoop-common/src/main/bin/stop-all.sh /hadoop/common/trunk/hadoop-common/src/main/bin/rcc
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Hdfs-trunk #765 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/765/)
          HADOOP-7563. Setup HADOOP_HDFS_HOME, HADOOP_MAPRED_HOME and classpath correction. Contributed by Eric Yang.

          acmurthy : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1161329
          Files :

          • /hadoop/common/trunk/hadoop-hdfs/src/main/bin/refresh-namenodes.sh
          • /hadoop/common/trunk/hadoop-common/src/main/bin/start-all.sh
          • /hadoop/common/trunk/hadoop-hdfs/src/main/bin/distribute-exclude.sh
          • /hadoop/common/trunk/hadoop-common/CHANGES.txt
          • /hadoop/common/trunk/hadoop-mapreduce/pom.xml
          • /hadoop/common/trunk/hadoop-common/src/main/bin/hadoop-daemons.sh
          • /hadoop/common/trunk/hadoop-common/src/main/bin/hadoop-daemon.sh
          • /hadoop/common/trunk/hadoop-common/src/main/bin/hadoop-config.sh
          • /hadoop/common/trunk/hadoop-hdfs/src/main/bin/stop-dfs.sh
          • /hadoop/common/trunk/hadoop-hdfs/src/main/bin/start-dfs.sh
          • /hadoop/common/trunk/hadoop-hdfs/src/main/bin/hdfs-config.sh
          • /hadoop/common/trunk/hadoop-hdfs/src/main/bin/start-balancer.sh
          • /hadoop/common/trunk/hadoop-hdfs/src/main/bin/stop-balancer.sh
          • /hadoop/common/trunk/hadoop-hdfs/src/main/bin/start-secure-dns.sh
          • /hadoop/common/trunk/hadoop-common/src/main/bin/slaves.sh
          • /hadoop/common/trunk/hadoop-hdfs/src/main/bin/stop-secure-dns.sh
          • /hadoop/common/trunk/hadoop-common/src/main/bin/hadoop
          • /hadoop/common/trunk/hadoop-hdfs/src/main/bin/hdfs
          • /hadoop/common/trunk/hadoop-assemblies/src/main/resources/assemblies/hadoop-dist.xml
          • /hadoop/common/trunk/hadoop-common/src/main/bin/stop-all.sh
          • /hadoop/common/trunk/hadoop-common/src/main/bin/rcc
          Show
          Hudson added a comment - Integrated in Hadoop-Hdfs-trunk #765 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/765/ ) HADOOP-7563 . Setup HADOOP_HDFS_HOME, HADOOP_MAPRED_HOME and classpath correction. Contributed by Eric Yang. acmurthy : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1161329 Files : /hadoop/common/trunk/hadoop-hdfs/src/main/bin/refresh-namenodes.sh /hadoop/common/trunk/hadoop-common/src/main/bin/start-all.sh /hadoop/common/trunk/hadoop-hdfs/src/main/bin/distribute-exclude.sh /hadoop/common/trunk/hadoop-common/CHANGES.txt /hadoop/common/trunk/hadoop-mapreduce/pom.xml /hadoop/common/trunk/hadoop-common/src/main/bin/hadoop-daemons.sh /hadoop/common/trunk/hadoop-common/src/main/bin/hadoop-daemon.sh /hadoop/common/trunk/hadoop-common/src/main/bin/hadoop-config.sh /hadoop/common/trunk/hadoop-hdfs/src/main/bin/stop-dfs.sh /hadoop/common/trunk/hadoop-hdfs/src/main/bin/start-dfs.sh /hadoop/common/trunk/hadoop-hdfs/src/main/bin/hdfs-config.sh /hadoop/common/trunk/hadoop-hdfs/src/main/bin/start-balancer.sh /hadoop/common/trunk/hadoop-hdfs/src/main/bin/stop-balancer.sh /hadoop/common/trunk/hadoop-hdfs/src/main/bin/start-secure-dns.sh /hadoop/common/trunk/hadoop-common/src/main/bin/slaves.sh /hadoop/common/trunk/hadoop-hdfs/src/main/bin/stop-secure-dns.sh /hadoop/common/trunk/hadoop-common/src/main/bin/hadoop /hadoop/common/trunk/hadoop-hdfs/src/main/bin/hdfs /hadoop/common/trunk/hadoop-assemblies/src/main/resources/assemblies/hadoop-dist.xml /hadoop/common/trunk/hadoop-common/src/main/bin/stop-all.sh /hadoop/common/trunk/hadoop-common/src/main/bin/rcc

            People

            • Assignee:
              Eric Yang
              Reporter:
              Eric Yang
            • Votes:
              0 Vote for this issue
              Watchers:
              6 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development