Uploaded image for project: 'Hadoop Common'
  1. Hadoop Common
  2. HADOOP-13673

Update scripts to be smarter when running with privilege

    Details

    • Type: New Feature
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 3.0.0-alpha1, 3.0.0-alpha2
    • Fix Version/s: 3.0.0-alpha2
    • Component/s: scripts
    • Labels:
    • Target Version/s:
    • Release Note:
      Hide
      Apache Hadoop is now able to switch to the appropriate user prior to launching commands so long as the command is being run with a privileged user and the appropriate set of _USER variables are defined. This re-enables sbin/start-all.sh and sbin/stop-all.sh as well as fixes the sbin/start-dfs.sh and sbin/stop-dfs.sh to work with both secure and unsecure systems.
      Show
      Apache Hadoop is now able to switch to the appropriate user prior to launching commands so long as the command is being run with a privileged user and the appropriate set of _USER variables are defined. This re-enables sbin/start-all.sh and sbin/stop-all.sh as well as fixes the sbin/start-dfs.sh and sbin/stop-dfs.sh to work with both secure and unsecure systems.

      Description

      As work continues on HADOOP-13397, it's become evident that we need better hooks to start daemons as specifically configured users. Via the (command)_(subcommand)_USER environment variables in 3.x, we actually have a standardized way to do that. This in turn means we can make the sbin scripts super functional with a bit of updating:

      • Consolidate start-dfs.sh and start-secure-dns.sh into one script
      • Make start-*.sh and stop-*.sh know how to switch users when run as root
      • Undeprecate start/stop-all.sh so that it could be used as root for production purposes and as a single user for non-production users
      1. HADOOP-13673.00.patch
        21 kB
        Allen Wittenauer
      2. HADOOP-13673.01.patch
        30 kB
        Allen Wittenauer
      3. HADOOP-13673.02.patch
        35 kB
        Allen Wittenauer
      4. HADOOP-13673.03.patch
        36 kB
        Allen Wittenauer
      5. HADOOP-13673.04.patch
        37 kB
        Allen Wittenauer

        Issue Links

          Activity

          Hide
          aw Allen Wittenauer added a comment - - edited

          -00:

          • first pass

          Running the start-* and stop-* commands will fire off daemons either as the user they are being run as or if the effective user id is root, as the appropriate _user definition. Secure daemons will "do the right thing"--get started as root but then switch to the appropriate user when needed.

          At this point, the old start-secure and start-dfs are not merged. I may do that in a future pass.

          Show
          aw Allen Wittenauer added a comment - - edited -00: first pass Running the start-* and stop-* commands will fire off daemons either as the user they are being run as or if the effective user id is root, as the appropriate _user definition. Secure daemons will "do the right thing"--get started as root but then switch to the appropriate user when needed. At this point, the old start-secure and start-dfs are not merged. I may do that in a future pass.
          Hide
          aw Allen Wittenauer added a comment -

          Argh. I'll fix the hadoop-project/pom.xml issue on 01.

          Show
          aw Allen Wittenauer added a comment - Argh. I'll fix the hadoop-project/pom.xml issue on 01.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 18s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
          0 mvndep 0m 17s Maven dependency ordering for branch
          +1 mvninstall 9m 5s trunk passed
          +1 compile 9m 26s trunk passed
          +1 mvnsite 6m 5s trunk passed
          +1 mvneclipse 1m 24s trunk passed
          +1 javadoc 3m 20s trunk passed
          0 mvndep 0m 18s Maven dependency ordering for patch
          +1 mvninstall 4m 16s the patch passed
          +1 compile 7m 26s the patch passed
          +1 javac 7m 26s the patch passed
          +1 mvnsite 5m 17s the patch passed
          +1 mvneclipse 1m 32s the patch passed
          -1 shellcheck 0m 12s The patch generated 2 new + 75 unchanged - 0 fixed = 77 total (was 75)
          -0 shelldocs 0m 10s The patch generated 8 new + 122 unchanged - 2 fixed = 130 total (was 124)
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 xml 0m 1s The patch has no ill-formed XML file.
          +1 javadoc 3m 18s the patch passed
          +1 unit 0m 11s hadoop-project in the patch passed.
          +1 unit 9m 52s hadoop-common in the patch passed.
          -1 unit 73m 1s hadoop-hdfs in the patch failed.
          -1 unit 74m 7s hadoop-yarn in the patch failed.
          +1 asflicense 0m 28s The patch does not generate ASF License warnings.
          211m 7s



          Reason Tests
          Failed junit tests hadoop.hdfs.server.namenode.TestNameNodeMetadataConsistency
            hadoop.yarn.server.TestContainerManagerSecurity
            hadoop.yarn.server.TestMiniYarnClusterNodeUtilization



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:9560f25
          JIRA Issue HADOOP-13673
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12836422/HADOOP-13673.00.patch
          Optional Tests asflicense mvnsite unit shellcheck shelldocs compile javac javadoc mvninstall xml
          uname Linux 02e23a536e4f 3.13.0-96-generic #143-Ubuntu SMP Mon Aug 29 20:15:20 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / 34173a4
          Default Java 1.8.0_101
          shellcheck v0.4.4
          shellcheck https://builds.apache.org/job/PreCommit-HADOOP-Build/10948/artifact/patchprocess/diff-patch-shellcheck.txt
          shelldocs https://builds.apache.org/job/PreCommit-HADOOP-Build/10948/artifact/patchprocess/diff-patch-shelldocs.txt
          unit https://builds.apache.org/job/PreCommit-HADOOP-Build/10948/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
          unit https://builds.apache.org/job/PreCommit-HADOOP-Build/10948/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn.txt
          Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/10948/testReport/
          modules C: hadoop-project hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs hadoop-yarn-project/hadoop-yarn U: .
          Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/10948/console
          Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 18s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. 0 mvndep 0m 17s Maven dependency ordering for branch +1 mvninstall 9m 5s trunk passed +1 compile 9m 26s trunk passed +1 mvnsite 6m 5s trunk passed +1 mvneclipse 1m 24s trunk passed +1 javadoc 3m 20s trunk passed 0 mvndep 0m 18s Maven dependency ordering for patch +1 mvninstall 4m 16s the patch passed +1 compile 7m 26s the patch passed +1 javac 7m 26s the patch passed +1 mvnsite 5m 17s the patch passed +1 mvneclipse 1m 32s the patch passed -1 shellcheck 0m 12s The patch generated 2 new + 75 unchanged - 0 fixed = 77 total (was 75) -0 shelldocs 0m 10s The patch generated 8 new + 122 unchanged - 2 fixed = 130 total (was 124) +1 whitespace 0m 0s The patch has no whitespace issues. +1 xml 0m 1s The patch has no ill-formed XML file. +1 javadoc 3m 18s the patch passed +1 unit 0m 11s hadoop-project in the patch passed. +1 unit 9m 52s hadoop-common in the patch passed. -1 unit 73m 1s hadoop-hdfs in the patch failed. -1 unit 74m 7s hadoop-yarn in the patch failed. +1 asflicense 0m 28s The patch does not generate ASF License warnings. 211m 7s Reason Tests Failed junit tests hadoop.hdfs.server.namenode.TestNameNodeMetadataConsistency   hadoop.yarn.server.TestContainerManagerSecurity   hadoop.yarn.server.TestMiniYarnClusterNodeUtilization Subsystem Report/Notes Docker Image:yetus/hadoop:9560f25 JIRA Issue HADOOP-13673 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12836422/HADOOP-13673.00.patch Optional Tests asflicense mvnsite unit shellcheck shelldocs compile javac javadoc mvninstall xml uname Linux 02e23a536e4f 3.13.0-96-generic #143-Ubuntu SMP Mon Aug 29 20:15:20 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 34173a4 Default Java 1.8.0_101 shellcheck v0.4.4 shellcheck https://builds.apache.org/job/PreCommit-HADOOP-Build/10948/artifact/patchprocess/diff-patch-shellcheck.txt shelldocs https://builds.apache.org/job/PreCommit-HADOOP-Build/10948/artifact/patchprocess/diff-patch-shelldocs.txt unit https://builds.apache.org/job/PreCommit-HADOOP-Build/10948/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt unit https://builds.apache.org/job/PreCommit-HADOOP-Build/10948/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn.txt Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/10948/testReport/ modules C: hadoop-project hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs hadoop-yarn-project/hadoop-yarn U: . Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/10948/console Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          aw Allen Wittenauer added a comment - - edited

          -01:

          • some basic docs
          • hdfs/yarn/hadoop now support account switching
          • various bugs

          Some things I've been doing for testing:

          hadoop-env.sh:

          HDFS_NAMENODE_USER=hdfs
          HDFS_DATANODE_USER=root
          HDFS_DATANODE_SECURE_USER=hdfs
          YARN_RESOURCEMANAGER_USER=yarn
          
          root# yarn --daemon start resourcemanager
          yarn$ yarn --daemon start resourcemanager
          root# hdfs --daemon start datanode
          hdfs$ hdfs --daemon start namenode
          root# sbin/start-all.sh
          root# sbin/stop-all.sh
          hdfs$ start-dfs.sh
          root# start-dfs.sh
          yarn$ start-yarn.sh
          root# start-yarn.sh
          

          TODO:

          • verify that users can run daemons as root if they set _USER=root
          Show
          aw Allen Wittenauer added a comment - - edited -01: some basic docs hdfs/yarn/hadoop now support account switching various bugs Some things I've been doing for testing: hadoop-env.sh: HDFS_NAMENODE_USER=hdfs HDFS_DATANODE_USER=root HDFS_DATANODE_SECURE_USER=hdfs YARN_RESOURCEMANAGER_USER=yarn root# yarn --daemon start resourcemanager yarn$ yarn --daemon start resourcemanager root# hdfs --daemon start datanode hdfs$ hdfs --daemon start namenode root# sbin/start-all.sh root# sbin/stop-all.sh hdfs$ start-dfs.sh root# start-dfs.sh yarn$ start-yarn.sh root# start-yarn.sh TODO: verify that users can run daemons as root if they set _USER=root
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 16s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
          0 mvndep 1m 39s Maven dependency ordering for branch
          +1 mvninstall 7m 22s trunk passed
          +1 mvnsite 7m 13s trunk passed
          0 mvndep 0m 16s Maven dependency ordering for patch
          +1 mvnsite 6m 39s the patch passed
          -1 shellcheck 0m 11s The patch generated 2 new + 117 unchanged - 0 fixed = 119 total (was 117)
          +1 shelldocs 0m 9s The patch generated 0 new + 112 unchanged - 12 fixed = 112 total (was 124)
          -1 whitespace 0m 0s The patch has 1 line(s) that end in whitespace. Use git apply --whitespace=fix <<patch_file>>. Refer https://git-scm.com/docs/git-apply
          +1 unit 1m 52s hadoop-common in the patch passed.
          +1 unit 0m 48s hadoop-hdfs in the patch passed.
          +1 unit 5m 6s hadoop-yarn in the patch passed.
          +1 unit 4m 51s hadoop-mapreduce-project in the patch passed.
          +1 asflicense 1m 48s The patch does not generate ASF License warnings.
          38m 41s



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:a9ad5d6
          JIRA Issue HADOOP-13673
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12841317/HADOOP-13673.01.patch
          Optional Tests asflicense mvnsite unit shellcheck shelldocs
          uname Linux 9cfd9e08c35e 3.13.0-93-generic #140-Ubuntu SMP Mon Jul 18 21:21:05 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / e0fa492
          shellcheck v0.4.5
          shellcheck https://builds.apache.org/job/PreCommit-HADOOP-Build/11177/artifact/patchprocess/diff-patch-shellcheck.txt
          whitespace https://builds.apache.org/job/PreCommit-HADOOP-Build/11177/artifact/patchprocess/whitespace-eol.txt
          Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/11177/testReport/
          modules C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs hadoop-yarn-project/hadoop-yarn hadoop-mapreduce-project U: .
          Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/11177/console
          Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 16s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. 0 mvndep 1m 39s Maven dependency ordering for branch +1 mvninstall 7m 22s trunk passed +1 mvnsite 7m 13s trunk passed 0 mvndep 0m 16s Maven dependency ordering for patch +1 mvnsite 6m 39s the patch passed -1 shellcheck 0m 11s The patch generated 2 new + 117 unchanged - 0 fixed = 119 total (was 117) +1 shelldocs 0m 9s The patch generated 0 new + 112 unchanged - 12 fixed = 112 total (was 124) -1 whitespace 0m 0s The patch has 1 line(s) that end in whitespace. Use git apply --whitespace=fix <<patch_file>>. Refer https://git-scm.com/docs/git-apply +1 unit 1m 52s hadoop-common in the patch passed. +1 unit 0m 48s hadoop-hdfs in the patch passed. +1 unit 5m 6s hadoop-yarn in the patch passed. +1 unit 4m 51s hadoop-mapreduce-project in the patch passed. +1 asflicense 1m 48s The patch does not generate ASF License warnings. 38m 41s Subsystem Report/Notes Docker Image:yetus/hadoop:a9ad5d6 JIRA Issue HADOOP-13673 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12841317/HADOOP-13673.01.patch Optional Tests asflicense mvnsite unit shellcheck shelldocs uname Linux 9cfd9e08c35e 3.13.0-93-generic #140-Ubuntu SMP Mon Jul 18 21:21:05 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / e0fa492 shellcheck v0.4.5 shellcheck https://builds.apache.org/job/PreCommit-HADOOP-Build/11177/artifact/patchprocess/diff-patch-shellcheck.txt whitespace https://builds.apache.org/job/PreCommit-HADOOP-Build/11177/artifact/patchprocess/whitespace-eol.txt Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/11177/testReport/ modules C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs hadoop-yarn-project/hadoop-yarn hadoop-mapreduce-project U: . Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/11177/console Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          aw Allen Wittenauer added a comment - - edited

          -02:

          • minor bug fixes
          • add unit tests
          • doc fixes
          • shellcheck fixes
          • verified that users can run daemons as root if they set _USER=root (as ill-advised as that is)
          Show
          aw Allen Wittenauer added a comment - - edited -02: minor bug fixes add unit tests doc fixes shellcheck fixes verified that users can run daemons as root if they set _USER=root (as ill-advised as that is)
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 16s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 4 new or modified test files.
          0 mvndep 0m 18s Maven dependency ordering for branch
          +1 mvninstall 8m 48s trunk passed
          +1 mvnsite 7m 16s trunk passed
          0 mvndep 0m 17s Maven dependency ordering for patch
          +1 mvnsite 6m 28s the patch passed
          +1 shellcheck 0m 11s There were no new shellcheck issues.
          +1 shelldocs 0m 8s The patch generated 0 new + 112 unchanged - 12 fixed = 112 total (was 124)
          -1 whitespace 0m 0s The patch has 2 line(s) that end in whitespace. Use git apply --whitespace=fix <<patch_file>>. Refer https://git-scm.com/docs/git-apply
          +1 unit 2m 1s hadoop-common in the patch passed.
          +1 unit 0m 49s hadoop-hdfs in the patch passed.
          +1 unit 4m 59s hadoop-yarn in the patch passed.
          +1 unit 1m 51s hadoop-mapreduce-project in the patch passed.
          +1 asflicense 0m 35s The patch does not generate ASF License warnings.
          34m 25s



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:a9ad5d6
          JIRA Issue HADOOP-13673
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12842853/HADOOP-13673.02.patch
          Optional Tests asflicense mvnsite unit shellcheck shelldocs
          uname Linux b55eae3a42ae 3.13.0-103-generic #150-Ubuntu SMP Thu Nov 24 10:34:17 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / f66f618
          shellcheck v0.4.5
          whitespace https://builds.apache.org/job/PreCommit-HADOOP-Build/11250/artifact/patchprocess/whitespace-eol.txt
          Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/11250/testReport/
          modules C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs hadoop-yarn-project/hadoop-yarn hadoop-mapreduce-project U: .
          Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/11250/console
          Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 16s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 4 new or modified test files. 0 mvndep 0m 18s Maven dependency ordering for branch +1 mvninstall 8m 48s trunk passed +1 mvnsite 7m 16s trunk passed 0 mvndep 0m 17s Maven dependency ordering for patch +1 mvnsite 6m 28s the patch passed +1 shellcheck 0m 11s There were no new shellcheck issues. +1 shelldocs 0m 8s The patch generated 0 new + 112 unchanged - 12 fixed = 112 total (was 124) -1 whitespace 0m 0s The patch has 2 line(s) that end in whitespace. Use git apply --whitespace=fix <<patch_file>>. Refer https://git-scm.com/docs/git-apply +1 unit 2m 1s hadoop-common in the patch passed. +1 unit 0m 49s hadoop-hdfs in the patch passed. +1 unit 4m 59s hadoop-yarn in the patch passed. +1 unit 1m 51s hadoop-mapreduce-project in the patch passed. +1 asflicense 0m 35s The patch does not generate ASF License warnings. 34m 25s Subsystem Report/Notes Docker Image:yetus/hadoop:a9ad5d6 JIRA Issue HADOOP-13673 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12842853/HADOOP-13673.02.patch Optional Tests asflicense mvnsite unit shellcheck shelldocs uname Linux b55eae3a42ae 3.13.0-103-generic #150-Ubuntu SMP Thu Nov 24 10:34:17 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / f66f618 shellcheck v0.4.5 whitespace https://builds.apache.org/job/PreCommit-HADOOP-Build/11250/artifact/patchprocess/whitespace-eol.txt Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/11250/testReport/ modules C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs hadoop-yarn-project/hadoop-yarn hadoop-mapreduce-project U: . Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/11250/console Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          aw Allen Wittenauer added a comment -

          OK, found a bug with failure. Since the su isn't exec'd, we continue on and do weird things. Need to add some protection around that in a few spots.

          Show
          aw Allen Wittenauer added a comment - OK, found a bug with failure. Since the su isn't exec'd, we continue on and do weird things. Need to add some protection around that in a few spots.
          Hide
          aw Allen Wittenauer added a comment - - edited

          -03:

          • if the su operation isn't expected to return, then callers must do their own exec or exit or whatever. This ends up being a lot simpler than adding a param that will likely be false.
          • abs MYNAME so that if the command given is a relative path, we can su correctly. e.g., as root calling "hadoop/bin/hdfs namenode" would fail since su would try to call hadoop/bin/hdfs which was no longer the correct path

          At this point, I think everything is working and this should get reviewed.

          Show
          aw Allen Wittenauer added a comment - - edited -03: if the su operation isn't expected to return, then callers must do their own exec or exit or whatever. This ends up being a lot simpler than adding a param that will likely be false. abs MYNAME so that if the command given is a relative path, we can su correctly. e.g., as root calling "hadoop/bin/hdfs namenode" would fail since su would try to call hadoop/bin/hdfs which was no longer the correct path At this point, I think everything is working and this should get reviewed.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 13s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 4 new or modified test files.
          0 mvndep 0m 23s Maven dependency ordering for branch
          +1 mvninstall 13m 0s trunk passed
          +1 mvnsite 6m 1s trunk passed
          0 mvndep 0m 15s Maven dependency ordering for patch
          +1 mvnsite 5m 43s the patch passed
          +1 shellcheck 0m 12s There were no new shellcheck issues.
          +1 shelldocs 0m 8s The patch generated 0 new + 112 unchanged - 12 fixed = 112 total (was 124)
          -1 whitespace 0m 0s The patch has 3 line(s) that end in whitespace. Use git apply --whitespace=fix <<patch_file>>. Refer https://git-scm.com/docs/git-apply
          +1 unit 1m 55s hadoop-common in the patch passed.
          +1 unit 0m 51s hadoop-hdfs in the patch passed.
          +1 unit 5m 4s hadoop-yarn in the patch passed.
          +1 unit 1m 54s hadoop-mapreduce-project in the patch passed.
          +1 asflicense 0m 35s The patch does not generate ASF License warnings.
          36m 44s



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:a9ad5d6
          JIRA Issue HADOOP-13673
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12845825/HADOOP-13673.03.patch
          Optional Tests asflicense mvnsite unit shellcheck shelldocs
          uname Linux 21d535bf48ff 3.13.0-103-generic #150-Ubuntu SMP Thu Nov 24 10:34:17 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / a605ff3
          shellcheck v0.4.5
          whitespace https://builds.apache.org/job/PreCommit-HADOOP-Build/11372/artifact/patchprocess/whitespace-eol.txt
          Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/11372/testReport/
          modules C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs hadoop-yarn-project/hadoop-yarn hadoop-mapreduce-project U: .
          Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/11372/console
          Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 13s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 4 new or modified test files. 0 mvndep 0m 23s Maven dependency ordering for branch +1 mvninstall 13m 0s trunk passed +1 mvnsite 6m 1s trunk passed 0 mvndep 0m 15s Maven dependency ordering for patch +1 mvnsite 5m 43s the patch passed +1 shellcheck 0m 12s There were no new shellcheck issues. +1 shelldocs 0m 8s The patch generated 0 new + 112 unchanged - 12 fixed = 112 total (was 124) -1 whitespace 0m 0s The patch has 3 line(s) that end in whitespace. Use git apply --whitespace=fix <<patch_file>>. Refer https://git-scm.com/docs/git-apply +1 unit 1m 55s hadoop-common in the patch passed. +1 unit 0m 51s hadoop-hdfs in the patch passed. +1 unit 5m 4s hadoop-yarn in the patch passed. +1 unit 1m 54s hadoop-mapreduce-project in the patch passed. +1 asflicense 0m 35s The patch does not generate ASF License warnings. 36m 44s Subsystem Report/Notes Docker Image:yetus/hadoop:a9ad5d6 JIRA Issue HADOOP-13673 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12845825/HADOOP-13673.03.patch Optional Tests asflicense mvnsite unit shellcheck shelldocs uname Linux 21d535bf48ff 3.13.0-103-generic #150-Ubuntu SMP Thu Nov 24 10:34:17 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / a605ff3 shellcheck v0.4.5 whitespace https://builds.apache.org/job/PreCommit-HADOOP-Build/11372/artifact/patchprocess/whitespace-eol.txt Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/11372/testReport/ modules C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs hadoop-yarn-project/hadoop-yarn hadoop-mapreduce-project U: . Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/11372/console Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          aw Allen Wittenauer added a comment -

          ping Andrew Wang. I'd like to get this into -alpha2.

          Show
          aw Allen Wittenauer added a comment - ping Andrew Wang . I'd like to get this into -alpha2.
          Hide
          raviprak Ravi Prakash added a comment -

          Hi Allen!

          Thanks for the patch! It looks good. I only could find these nits:

          1. "Atempting" -> "Attempting"
          2. Remove "$ {EUID}

            comes from the shell itself!" in hadoop-functions.sh

          3. I'm not exactly sure how HADOOP_REEXECED_CMD is being used to prevent a fork bomb, but could a script set it to false explicitly as part of itself? i.e. what's preventing access to that variable from a user script?
            #pwd
          4. Is hadoop_abs supposed to resolve links? If yes, in hadoop_abs.bats could you please add a test for links?
          Show
          raviprak Ravi Prakash added a comment - Hi Allen! Thanks for the patch! It looks good. I only could find these nits: "Atempting" -> "Attempting" Remove "$ {EUID} comes from the shell itself!" in hadoop-functions.sh I'm not exactly sure how HADOOP_REEXECED_CMD is being used to prevent a fork bomb, but could a script set it to false explicitly as part of itself? i.e. what's preventing access to that variable from a user script? #pwd Is hadoop_abs supposed to resolve links? If yes, in hadoop_abs.bats could you please add a test for links?
          Hide
          aw Allen Wittenauer added a comment -

          Thanks for the feedback Ravi Prakash and Andrew Wang (who did his offline while JIRA was down). -04 should cover all of the very valid points you've raised.

          I'm not exactly sure how HADOOP_REEXECED_CMD is being used to prevent a fork bomb, but could a script set it to false explicitly as part of itself? i.e. what's preventing access to that variable from a user script?

          Anything that runs inside the environment can of course wreak havoc on anything. If we ignore bad actors, what happens is this:

          1. user runs command
          2. command determines that _USER has been set and it needs to get re-executed as a different user.
          3. command calls itself with same parameters, etc, but adds --reexec to the command line
          4. if for some reason command calls itself again, there will be two --reexec's on the command line (since those options aren't stripped) which will stop it during the param parasing. Additionally, hadoop_need_reexec will return false as well.

          Sure, it's not as strong as a semaphore, but I think it should stop most non-malicious code.

          Show
          aw Allen Wittenauer added a comment - Thanks for the feedback Ravi Prakash and Andrew Wang (who did his offline while JIRA was down). -04 should cover all of the very valid points you've raised. I'm not exactly sure how HADOOP_REEXECED_CMD is being used to prevent a fork bomb, but could a script set it to false explicitly as part of itself? i.e. what's preventing access to that variable from a user script? Anything that runs inside the environment can of course wreak havoc on anything. If we ignore bad actors, what happens is this: 1. user runs command 2. command determines that _USER has been set and it needs to get re-executed as a different user. 3. command calls itself with same parameters, etc, but adds --reexec to the command line 4. if for some reason command calls itself again, there will be two --reexec's on the command line (since those options aren't stripped) which will stop it during the param parasing. Additionally, hadoop_need_reexec will return false as well. Sure, it's not as strong as a semaphore, but I think it should stop most non-malicious code.
          Hide
          aw Allen Wittenauer added a comment -

          -04:

          • rebase
          • spelling fixes
          • test for symlinks
          • some whitespace fixes
          • more documentation + fixes
          Show
          aw Allen Wittenauer added a comment - -04: rebase spelling fixes test for symlinks some whitespace fixes more documentation + fixes
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 15s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 4 new or modified test files.
          0 mvndep 0m 45s Maven dependency ordering for branch
          +1 mvninstall 13m 58s trunk passed
          +1 mvnsite 6m 8s trunk passed
          0 mvndep 0m 15s Maven dependency ordering for patch
          +1 mvnsite 6m 0s the patch passed
          +1 shellcheck 0m 11s There were no new shellcheck issues.
          +1 shelldocs 0m 8s The patch generated 0 new + 108 unchanged - 12 fixed = 108 total (was 120)
          -1 whitespace 0m 0s The patch has 1 line(s) that end in whitespace. Use git apply --whitespace=fix <<patch_file>>. Refer https://git-scm.com/docs/git-apply
          +1 unit 1m 55s hadoop-common in the patch passed.
          +1 unit 0m 50s hadoop-hdfs in the patch passed.
          +1 unit 5m 16s hadoop-yarn in the patch passed.
          +1 unit 2m 16s hadoop-mapreduce-project in the patch passed.
          +1 asflicense 0m 31s The patch does not generate ASF License warnings.
          38m 58s



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:a9ad5d6
          JIRA Issue HADOOP-13673
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12847441/HADOOP-13673.04.patch
          Optional Tests asflicense mvnsite unit shellcheck shelldocs
          uname Linux 57303bef37c4 3.13.0-106-generic #153-Ubuntu SMP Tue Dec 6 15:44:32 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / d3170f9
          shellcheck v0.4.5
          whitespace https://builds.apache.org/job/PreCommit-HADOOP-Build/11437/artifact/patchprocess/whitespace-eol.txt
          Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/11437/testReport/
          modules C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs hadoop-yarn-project/hadoop-yarn hadoop-mapreduce-project U: .
          Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/11437/console
          Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 15s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 4 new or modified test files. 0 mvndep 0m 45s Maven dependency ordering for branch +1 mvninstall 13m 58s trunk passed +1 mvnsite 6m 8s trunk passed 0 mvndep 0m 15s Maven dependency ordering for patch +1 mvnsite 6m 0s the patch passed +1 shellcheck 0m 11s There were no new shellcheck issues. +1 shelldocs 0m 8s The patch generated 0 new + 108 unchanged - 12 fixed = 108 total (was 120) -1 whitespace 0m 0s The patch has 1 line(s) that end in whitespace. Use git apply --whitespace=fix <<patch_file>>. Refer https://git-scm.com/docs/git-apply +1 unit 1m 55s hadoop-common in the patch passed. +1 unit 0m 50s hadoop-hdfs in the patch passed. +1 unit 5m 16s hadoop-yarn in the patch passed. +1 unit 2m 16s hadoop-mapreduce-project in the patch passed. +1 asflicense 0m 31s The patch does not generate ASF License warnings. 38m 58s Subsystem Report/Notes Docker Image:yetus/hadoop:a9ad5d6 JIRA Issue HADOOP-13673 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12847441/HADOOP-13673.04.patch Optional Tests asflicense mvnsite unit shellcheck shelldocs uname Linux 57303bef37c4 3.13.0-106-generic #153-Ubuntu SMP Tue Dec 6 15:44:32 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / d3170f9 shellcheck v0.4.5 whitespace https://builds.apache.org/job/PreCommit-HADOOP-Build/11437/artifact/patchprocess/whitespace-eol.txt Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/11437/testReport/ modules C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs hadoop-yarn-project/hadoop-yarn hadoop-mapreduce-project U: . Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/11437/console Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          andrew.wang Andrew Wang added a comment -

          Just for completeness, here's the email exchange between myself and Allen (I hope Allen doesn't mind me posting this):

          > * hadoop_abs, does readlink -f accomplish the same thing?

          Effectively yes, but unfortunately, readlink isn't POSIX. It works differently on different operating systems, even to the point of having radically different parameters. So we can't rely upon it. hadoop_abs, while obviously slower, is super portable.

          > * Was it intentional to remove hadoop_usage from start-secure-dns.sh? The stop script still has a usage.

          I was going to replace it but I guess I got distracted. haha. I'll put it back for now.

          > * Few typos seen while reviewing: "legimately" "optinally" "definied" "description"

          I think i got all of these.

          > * I think there's an extra "resourcemanager" in this line:
          >
          >

          > +  hadoop_uservar_su yarn resourcemanager proxyserver "${HADOOP_YARN_HOME}/bin/yarn" \
          > 

          Yup, definitely.

          > * IIUC we we call hadoop_uservar_su directly in start-dfs.sh which requires that the user vars to be set when running as root. Noticed though that start-balancer.sh doesn't do this. Is this intentional or an omission?

          Intentional. All of the single daemon scripts will switch when they call the main hdfs/mapred/... script. For the others, --workers needs to get called with the appropriate user so that we don't try to use root's ssh key unless we really were meant to (e.g., secure datanode).

          > * Wondering if more needs to be said in the docs about what commands support this. For instance, HTTPFS is off on the side, but I guess that'll be fixed once John finishes the conversion from Tomcat to Jetty. Are there any other gaps you're aware of?

          Of the daemons, yeah, httpfs is a big outlier. The other ones are rumen and sls. Now that we have dynamic commands, we should probably make them inline as well.

          Show
          andrew.wang Andrew Wang added a comment - Just for completeness, here's the email exchange between myself and Allen (I hope Allen doesn't mind me posting this): > * hadoop_abs, does readlink -f accomplish the same thing? Effectively yes, but unfortunately, readlink isn't POSIX. It works differently on different operating systems, even to the point of having radically different parameters. So we can't rely upon it. hadoop_abs, while obviously slower, is super portable. > * Was it intentional to remove hadoop_usage from start-secure-dns.sh? The stop script still has a usage. I was going to replace it but I guess I got distracted. haha. I'll put it back for now. > * Few typos seen while reviewing: "legimately" "optinally" "definied" "description" I think i got all of these. > * I think there's an extra "resourcemanager" in this line: > > > + hadoop_uservar_su yarn resourcemanager proxyserver "${HADOOP_YARN_HOME}/bin/yarn" \ > Yup, definitely. > * IIUC we we call hadoop_uservar_su directly in start-dfs.sh which requires that the user vars to be set when running as root. Noticed though that start-balancer.sh doesn't do this. Is this intentional or an omission? Intentional. All of the single daemon scripts will switch when they call the main hdfs/mapred/... script. For the others, --workers needs to get called with the appropriate user so that we don't try to use root's ssh key unless we really were meant to (e.g., secure datanode). > * Wondering if more needs to be said in the docs about what commands support this. For instance, HTTPFS is off on the side, but I guess that'll be fixed once John finishes the conversion from Tomcat to Jetty. Are there any other gaps you're aware of? Of the daemons, yeah, httpfs is a big outlier. The other ones are rumen and sls. Now that we have dynamic commands, we should probably make them inline as well.
          Hide
          andrew.wang Andrew Wang added a comment -

          Nice rev Allen, LGTM +1 personally. I'd like to wait for Ravi's +1 too, though I believe his review comments are also addressed or explained.

          Show
          andrew.wang Andrew Wang added a comment - Nice rev Allen, LGTM +1 personally. I'd like to wait for Ravi's +1 too, though I believe his review comments are also addressed or explained.
          Hide
          raviprak Ravi Prakash added a comment -

          LGTM too! Thanks Allen and Andrew! +1

          Show
          raviprak Ravi Prakash added a comment - LGTM too! Thanks Allen and Andrew! +1
          Hide
          aw Allen Wittenauer added a comment -

          Thanks for the reviews folks!

          Committing to trunk.

          Show
          aw Allen Wittenauer added a comment - Thanks for the reviews folks! Committing to trunk.
          Hide
          hudson Hudson added a comment -

          SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #11137 (See https://builds.apache.org/job/Hadoop-trunk-Commit/11137/)
          HADOOP-13673. Update scripts to be smarter when running with privilege (aw: rev 0eb4b513b76bc944c31b15cd6558901ae44bf931)

          • (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/bin/start-dfs.sh
          • (add) hadoop-common-project/hadoop-common/src/test/scripts/hadoop_get_verify_uservar.bats
          • (edit) hadoop-common-project/hadoop-common/src/site/markdown/UnixShellGuide.md
          • (edit) hadoop-common-project/hadoop-common/src/test/scripts/hadoop-functions_test_helper.bash
          • (edit) hadoop-mapreduce-project/bin/mapred
          • (edit) hadoop-common-project/hadoop-common/src/main/bin/start-all.sh
          • (edit) hadoop-yarn-project/hadoop-yarn/bin/start-yarn.sh
          • (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/bin/stop-secure-dns.sh
          • (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/bin/hdfs
          • (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/bin/start-secure-dns.sh
          • (add) hadoop-common-project/hadoop-common/src/test/scripts/hadoop_abs.bats
          • (add) hadoop-common-project/hadoop-common/src/test/scripts/hadoop_privilege_check.bats
          • (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/bin/stop-dfs.sh
          • (edit) hadoop-common-project/hadoop-common/src/main/bin/hadoop
          • (edit) hadoop-common-project/hadoop-common/src/main/bin/hadoop-functions.sh
          • (edit) hadoop-yarn-project/hadoop-yarn/bin/stop-yarn.sh
          • (edit) hadoop-common-project/hadoop-common/src/main/bin/stop-all.sh
          • (edit) hadoop-yarn-project/hadoop-yarn/bin/yarn
          Show
          hudson Hudson added a comment - SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #11137 (See https://builds.apache.org/job/Hadoop-trunk-Commit/11137/ ) HADOOP-13673 . Update scripts to be smarter when running with privilege (aw: rev 0eb4b513b76bc944c31b15cd6558901ae44bf931) (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/bin/start-dfs.sh (add) hadoop-common-project/hadoop-common/src/test/scripts/hadoop_get_verify_uservar.bats (edit) hadoop-common-project/hadoop-common/src/site/markdown/UnixShellGuide.md (edit) hadoop-common-project/hadoop-common/src/test/scripts/hadoop-functions_test_helper.bash (edit) hadoop-mapreduce-project/bin/mapred (edit) hadoop-common-project/hadoop-common/src/main/bin/start-all.sh (edit) hadoop-yarn-project/hadoop-yarn/bin/start-yarn.sh (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/bin/stop-secure-dns.sh (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/bin/hdfs (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/bin/start-secure-dns.sh (add) hadoop-common-project/hadoop-common/src/test/scripts/hadoop_abs.bats (add) hadoop-common-project/hadoop-common/src/test/scripts/hadoop_privilege_check.bats (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/bin/stop-dfs.sh (edit) hadoop-common-project/hadoop-common/src/main/bin/hadoop (edit) hadoop-common-project/hadoop-common/src/main/bin/hadoop-functions.sh (edit) hadoop-yarn-project/hadoop-yarn/bin/stop-yarn.sh (edit) hadoop-common-project/hadoop-common/src/main/bin/stop-all.sh (edit) hadoop-yarn-project/hadoop-yarn/bin/yarn

            People

            • Assignee:
              aw Allen Wittenauer
              Reporter:
              aw Allen Wittenauer
            • Votes:
              0 Vote for this issue
              Watchers:
              7 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development