Hadoop Map/Reduce
  1. Hadoop Map/Reduce
  2. MAPREDUCE-2550

bin/mapred no longer works from a source checkout

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Blocker Blocker
    • Resolution: Fixed
    • Affects Version/s: 0.20.3
    • Fix Version/s: 0.23.0
    • Component/s: build
    • Labels:
      None
    • Environment:

      Java 6, Redhat 5.5

    • Hadoop Flags:
      Reviewed

      Description

      Developer may want to run hadoop without extracting tarball. It would be nice if existing method to run mapred scripts from source code is preserved for developers.

      1. MAPREDUCE-2550.patch
        2 kB
        Eric Yang
      2. MAPREDUCE-2550-1.patch
        3 kB
        Eric Yang
      3. MAPREDUCE-2550-2.patch
        3 kB
        Eric Yang

        Issue Links

          Activity

          Hide
          Eric Yang added a comment -

          Same type of issues for HDFS.

          Show
          Eric Yang added a comment - Same type of issues for HDFS.
          Hide
          Eric Yang added a comment -

          Restore HADOOP_COMMON_HOME and environment based control. This was removed per Owen's request to make the system more secure without depending on environment variables.

          This revision will check for $HADOOP_PREFIX/libexec/hadoop-config.sh exists. If libexec/hadoop-config.sh exists, it will be used. If libexec/hadoop-config.sh does not exist, then it falls back to developer environment setup.

          Show
          Eric Yang added a comment - Restore HADOOP_COMMON_HOME and environment based control. This was removed per Owen's request to make the system more secure without depending on environment variables. This revision will check for $HADOOP_PREFIX/libexec/hadoop-config.sh exists. If libexec/hadoop-config.sh exists, it will be used. If libexec/hadoop-config.sh does not exist, then it falls back to developer environment setup.
          Hide
          Eric Caspole added a comment -

          Does this change apply to trunk as well? I had been using the HADOOP_HDFS_HOME/HADOOP_COMMON_HOME/HADOOP_MAPRED_HOME vars with trunk and it was pretty much working until I updated today.

          Show
          Eric Caspole added a comment - Does this change apply to trunk as well? I had been using the HADOOP_HDFS_HOME/HADOOP_COMMON_HOME/HADOOP_MAPRED_HOME vars with trunk and it was pretty much working until I updated today.
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12480973/MAPREDUCE-2550.patch
          against trunk revision 1129771.

          +1 @author. The patch does not contain any @author tags.

          -1 tests included. The patch doesn't appear to include any new or modified tests.
          Please justify why no new tests are needed for this patch.
          Also please list what manual steps were performed to verify this patch.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          -1 findbugs. The patch appears to introduce 1 new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed core unit tests.

          +1 contrib tests. The patch passed contrib unit tests.

          +1 system test framework. The patch passed system test framework compile.

          Test results: https://builds.apache.org/hudson/job/PreCommit-MAPREDUCE-Build/323//testReport/
          Findbugs warnings: https://builds.apache.org/hudson/job/PreCommit-MAPREDUCE-Build/323//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
          Console output: https://builds.apache.org/hudson/job/PreCommit-MAPREDUCE-Build/323//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12480973/MAPREDUCE-2550.patch against trunk revision 1129771. +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. -1 findbugs. The patch appears to introduce 1 new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed core unit tests. +1 contrib tests. The patch passed contrib unit tests. +1 system test framework. The patch passed system test framework compile. Test results: https://builds.apache.org/hudson/job/PreCommit-MAPREDUCE-Build/323//testReport/ Findbugs warnings: https://builds.apache.org/hudson/job/PreCommit-MAPREDUCE-Build/323//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Console output: https://builds.apache.org/hudson/job/PreCommit-MAPREDUCE-Build/323//console This message is automatically generated.
          Hide
          Eric Yang added a comment -

          Modified the script to be more consistent for locating hadoop-config.sh.

          Show
          Eric Yang added a comment - Modified the script to be more consistent for locating hadoop-config.sh.
          Hide
          Eric Yang added a comment -

          Does this change apply to trunk as well? I had been using the

          HADOOP_HDFS_HOME/HADOOP_COMMON_HOME/HADOOP_MAPRED_HOME vars with trunk and it was pretty bq. much working until I updated today.

          Hi Eric, yes, this patch applies to trunk.

          Show
          Eric Yang added a comment - Does this change apply to trunk as well? I had been using the HADOOP_HDFS_HOME/HADOOP_COMMON_HOME/HADOOP_MAPRED_HOME vars with trunk and it was pretty bq. much working until I updated today. Hi Eric, yes, this patch applies to trunk.
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12481296/MAPREDUCE-2550-1.patch
          against trunk revision 1130554.

          +1 @author. The patch does not contain any @author tags.

          -1 tests included. The patch doesn't appear to include any new or modified tests.
          Please justify why no new tests are needed for this patch.
          Also please list what manual steps were performed to verify this patch.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          -1 findbugs. The patch appears to introduce 1 new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          -1 core tests. The patch failed these core unit tests:
          org.apache.hadoop.cli.TestMRCLI

          +1 contrib tests. The patch passed contrib unit tests.

          +1 system test framework. The patch passed system test framework compile.

          Test results: https://builds.apache.org/hudson/job/PreCommit-MAPREDUCE-Build/338//testReport/
          Findbugs warnings: https://builds.apache.org/hudson/job/PreCommit-MAPREDUCE-Build/338//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
          Console output: https://builds.apache.org/hudson/job/PreCommit-MAPREDUCE-Build/338//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12481296/MAPREDUCE-2550-1.patch against trunk revision 1130554. +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. -1 findbugs. The patch appears to introduce 1 new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these core unit tests: org.apache.hadoop.cli.TestMRCLI +1 contrib tests. The patch passed contrib unit tests. +1 system test framework. The patch passed system test framework compile. Test results: https://builds.apache.org/hudson/job/PreCommit-MAPREDUCE-Build/338//testReport/ Findbugs warnings: https://builds.apache.org/hudson/job/PreCommit-MAPREDUCE-Build/338//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Console output: https://builds.apache.org/hudson/job/PreCommit-MAPREDUCE-Build/338//console This message is automatically generated.
          Hide
          Aaron T. Myers added a comment -

          +1. I applied the patch to trunk and can confirm that `bin/mapred' did not work without this patch, and does work with it.

          Show
          Aaron T. Myers added a comment - +1. I applied the patch to trunk and can confirm that `bin/mapred' did not work without this patch, and does work with it.
          Hide
          Todd Lipcon added a comment -

          Do we need the following export?
          +export HADOOP_PREFIX="$

          {HADOOP_PREFIX:-$bin/..}

          "

          the sourcing of hadoop-config.sh should set this to point at COMMON_HOME in dev env or the appropriate prefix in an installed env, right?

          Show
          Todd Lipcon added a comment - Do we need the following export? +export HADOOP_PREFIX="$ {HADOOP_PREFIX:-$bin/..} " the sourcing of hadoop-config.sh should set this to point at COMMON_HOME in dev env or the appropriate prefix in an installed env, right?
          Hide
          Eric Yang added a comment -

          Do we need the following export?

          +export HADOOP_PREFIX="${HADOOP_PREFIX:-$bin/..}"

          This is required for "ant tar" release tarball deployment without merging the artificates to the same directory, and used COMMON_HOME to drive environments.

          Show
          Eric Yang added a comment - Do we need the following export? +export HADOOP_PREFIX="${HADOOP_PREFIX:-$bin/..}" This is required for "ant tar" release tarball deployment without merging the artificates to the same directory, and used COMMON_HOME to drive environments.
          Hide
          Eric Yang added a comment -

          Anyone wants to commit this?

          Show
          Eric Yang added a comment - Anyone wants to commit this?
          Hide
          Aaron T. Myers added a comment -

          Todd, are you satisfied with Eric's response to your question? If so, I'll go ahead and commit this.

          Show
          Aaron T. Myers added a comment - Todd, are you satisfied with Eric's response to your question? If so, I'll go ahead and commit this.
          Hide
          Aaron T. Myers added a comment -

          Never mind - I can't commit this since it's M/R.

          Todd, are you satisfied with Eric's response to your question? If so, would you mind committing this?

          Show
          Aaron T. Myers added a comment - Never mind - I can't commit this since it's M/R. Todd, are you satisfied with Eric's response to your question? If so, would you mind committing this?
          Hide
          Todd Lipcon added a comment -

          Sorry for slow response, I missed this in my jira box.

          Eric: my point is that the next thing after "export HADOOP_PREFIX=..." is to source hadoop-config.sh (wherever it might be found). The first export in that file is to export HADOOP_PREFIX again. So regardless of whether this is a dev env or installed env, the export in mapred-config.sh should be redundant, no?

          Show
          Todd Lipcon added a comment - Sorry for slow response, I missed this in my jira box. Eric: my point is that the next thing after "export HADOOP_PREFIX=..." is to source hadoop-config.sh (wherever it might be found). The first export in that file is to export HADOOP_PREFIX again. So regardless of whether this is a dev env or installed env, the export in mapred-config.sh should be redundant, no?
          Hide
          Eric Yang added a comment -

          Same patch with HADOOP_PREFIX removed from mapred-config.sh.

          Show
          Eric Yang added a comment - Same patch with HADOOP_PREFIX removed from mapred-config.sh.
          Hide
          Eric Yang added a comment -

          Todd, thanks for catching the duplicates.

          Show
          Eric Yang added a comment - Todd, thanks for catching the duplicates.
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12482952/MAPREDUCE-2550-2.patch
          against trunk revision 1136261.

          +1 @author. The patch does not contain any @author tags.

          -1 tests included. The patch doesn't appear to include any new or modified tests.
          Please justify why no new tests are needed for this patch.
          Also please list what manual steps were performed to verify this patch.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          -1 core tests. The patch failed these core unit tests:
          org.apache.hadoop.cli.TestMRCLI
          org.apache.hadoop.fs.TestFileSystem

          -1 contrib tests. The patch failed contrib unit tests.

          +1 system test framework. The patch passed system test framework compile.

          Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/404//testReport/
          Findbugs warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/404//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
          Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/404//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12482952/MAPREDUCE-2550-2.patch against trunk revision 1136261. +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these core unit tests: org.apache.hadoop.cli.TestMRCLI org.apache.hadoop.fs.TestFileSystem -1 contrib tests. The patch failed contrib unit tests. +1 system test framework. The patch passed system test framework compile. Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/404//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/404//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/404//console This message is automatically generated.
          Hide
          Todd Lipcon added a comment -

          +1, I'll check this into trunk momentarily.

          Show
          Todd Lipcon added a comment - +1, I'll check this into trunk momentarily.
          Hide
          Todd Lipcon added a comment -

          Committed to trunk, thanks Eric.

          Show
          Todd Lipcon added a comment - Committed to trunk, thanks Eric.
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Mapreduce-trunk-Commit #730 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/730/)
          MAPREDUCE-2550. Fix bin/mapred to work properly from within a source checkout. Contributed by Eric Yang.

          todd : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1137017
          Files :

          • /hadoop/common/trunk/mapreduce/bin/mapred-config.sh
          • /hadoop/common/trunk/mapreduce/CHANGES.txt
          • /hadoop/common/trunk/mapreduce/bin/start-mapred.sh
          • /hadoop/common/trunk/mapreduce/bin/stop-mapred.sh
          • /hadoop/common/trunk/mapreduce/bin/mapred
          Show
          Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk-Commit #730 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/730/ ) MAPREDUCE-2550 . Fix bin/mapred to work properly from within a source checkout. Contributed by Eric Yang. todd : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1137017 Files : /hadoop/common/trunk/mapreduce/bin/mapred-config.sh /hadoop/common/trunk/mapreduce/CHANGES.txt /hadoop/common/trunk/mapreduce/bin/start-mapred.sh /hadoop/common/trunk/mapreduce/bin/stop-mapred.sh /hadoop/common/trunk/mapreduce/bin/mapred
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Mapreduce-trunk #722 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/722/)

          Show
          Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk #722 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/722/ )
          Hide
          Ravi Prakash added a comment -

          Commit 103374ed71a22d614e77e79ac816fc72fbf93463 (Revision 161793) on branch-0.23
          mapred-config.sh searches for hadoop-config.sh in bin. Whereas after the recent mavenization and restructuring, if I had done

           $ mvn -Pdist install 

          in the top level directory then, hadoop-config.sh is put in libexec.

          Can we please include this case too?

          Show
          Ravi Prakash added a comment - Commit 103374ed71a22d614e77e79ac816fc72fbf93463 (Revision 161793) on branch-0.23 mapred-config.sh searches for hadoop-config.sh in bin. Whereas after the recent mavenization and restructuring, if I had done $ mvn -Pdist install in the top level directory then, hadoop-config.sh is put in libexec. Can we please include this case too?
          Hide
          Eric Yang added a comment -

          Arun updated the incremental changes of the script in MAPREDUCE-2885. Resolve this as fixed.

          Show
          Eric Yang added a comment - Arun updated the incremental changes of the script in MAPREDUCE-2885 . Resolve this as fixed.

            People

            • Assignee:
              Eric Yang
              Reporter:
              Eric Yang
            • Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development