Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 3.0.0-alpha2
    • Fix Version/s: 3.0.0-alpha4
    • Component/s: scripts
    • Labels:
      None
    • Target Version/s:
    • Hadoop Flags:
      Incompatible change
    • Release Note:
      Hide
      <!-- markdown -->

      The secure user variables have been changed to be consistent with the rest of the environment variable changes:

      | Old | New |
      |:---- |:---- |
      | HADOOP\_SECURE\_DN\_USER | HDFS\_DATANODE\_SECURE\_USER |
      | HADOO\P_PRIVILEGED\_NFS\_USER | HDFS\_NFS3\_SECURE\_USER |
      Show
      <!-- markdown --> The secure user variables have been changed to be consistent with the rest of the environment variable changes: | Old | New | |:---- |:---- | | HADOOP\_SECURE\_DN\_USER | HDFS\_DATANODE\_SECURE\_USER | | HADOO\P_PRIVILEGED\_NFS\_USER | HDFS\_NFS3\_SECURE\_USER |

      Description

      Post-HADOOP-13341 and (more importantly) HADOOP-13673, there has been a major effort on making the configuration environment variables consistent among all the projects. The vast majority of vars now look like (command)(subcommand)(etc). Two hold outs are HADOOP_SECURE_DN_USER and HADOOP_PRIVILEGED_NFS_USER.

      Additionally, there is

      • no generic handling
      • no documentation for anyone
      • no safety checks to make sure things are defined

      In order to fix all of this, we should:

      • deprecate the previous vars using the deprecation function, updating the HDFS documentation that references them
      • add generic (command)_(subcommand)_SECURE_USER support
      • add some verification for the previously mentioned var
      • add some docs to UnixShellGuide.md
      1. HADOOP-14202.00.patch
        54 kB
        Allen Wittenauer
      2. HADOOP-14202.01.patch
        58 kB
        Allen Wittenauer
      3. HADOOP-14202.02.patch
        59 kB
        Allen Wittenauer
      4. HADOOP-14202.03.patch
        68 kB
        Allen Wittenauer
      5. HADOOP-14202.04.patch
        68 kB
        Allen Wittenauer

        Issue Links

          Activity

          Hide
          hudson Hudson added a comment -

          SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #11591 (See https://builds.apache.org/job/Hadoop-trunk-Commit/11591/)
          HADOOP-14202. fix jsvc/secure user var inconsistencies (aw: rev 96cbb4fce2278c80b7e7e389d801ad31b97c0a61)

          • (add) hadoop-common-project/hadoop-common/src/test/scripts/hadoop_verify_user_perm.bats
          • (edit) hadoop-mapreduce-project/bin/mapred-config.sh
          • (edit) hadoop-common-project/hadoop-common/src/main/bin/hadoop-functions.sh
          • (edit) hadoop-common-project/hadoop-common/src/main/bin/hadoop-config.sh
          • (edit) hadoop-common-project/hadoop-common/src/main/conf/hadoop-env.sh
          • (add) hadoop-common-project/hadoop-common/src/test/scripts/hadoop_detect_priv_subcmd.bats
          • (edit) hadoop-common-project/hadoop-common/src/main/bin/hadoop
          • (edit) hadoop-common-project/hadoop-common/src/site/markdown/UnixShellGuide.md
          • (add) hadoop-common-project/hadoop-common/src/test/scripts/hadoop_build_custom_subcmd_var.bats
          • (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/bin/hdfs
          • (edit) hadoop-hdfs-project/hadoop-hdfs/src/site/markdown/HdfsNfsGateway.md
          • (edit) hadoop-yarn-project/hadoop-yarn/bin/yarn
          • (delete) hadoop-common-project/hadoop-common/src/test/scripts/hadoop_verify_user.bats
          • (edit) hadoop-common-project/hadoop-common/src/site/markdown/SecureMode.md
          • (add) hadoop-common-project/hadoop-common/src/test/scripts/hadoop_verify_user_resolves.bats
          • (edit) hadoop-yarn-project/hadoop-yarn/bin/yarn-config.sh
          • (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/bin/hdfs-config.sh
          • (edit) hadoop-mapreduce-project/bin/mapred
          • (delete) hadoop-common-project/hadoop-common/src/test/scripts/hadoop_get_verify_uservar.bats
          Show
          hudson Hudson added a comment - SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #11591 (See https://builds.apache.org/job/Hadoop-trunk-Commit/11591/ ) HADOOP-14202 . fix jsvc/secure user var inconsistencies (aw: rev 96cbb4fce2278c80b7e7e389d801ad31b97c0a61) (add) hadoop-common-project/hadoop-common/src/test/scripts/hadoop_verify_user_perm.bats (edit) hadoop-mapreduce-project/bin/mapred-config.sh (edit) hadoop-common-project/hadoop-common/src/main/bin/hadoop-functions.sh (edit) hadoop-common-project/hadoop-common/src/main/bin/hadoop-config.sh (edit) hadoop-common-project/hadoop-common/src/main/conf/hadoop-env.sh (add) hadoop-common-project/hadoop-common/src/test/scripts/hadoop_detect_priv_subcmd.bats (edit) hadoop-common-project/hadoop-common/src/main/bin/hadoop (edit) hadoop-common-project/hadoop-common/src/site/markdown/UnixShellGuide.md (add) hadoop-common-project/hadoop-common/src/test/scripts/hadoop_build_custom_subcmd_var.bats (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/bin/hdfs (edit) hadoop-hdfs-project/hadoop-hdfs/src/site/markdown/HdfsNfsGateway.md (edit) hadoop-yarn-project/hadoop-yarn/bin/yarn (delete) hadoop-common-project/hadoop-common/src/test/scripts/hadoop_verify_user.bats (edit) hadoop-common-project/hadoop-common/src/site/markdown/SecureMode.md (add) hadoop-common-project/hadoop-common/src/test/scripts/hadoop_verify_user_resolves.bats (edit) hadoop-yarn-project/hadoop-yarn/bin/yarn-config.sh (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/bin/hdfs-config.sh (edit) hadoop-mapreduce-project/bin/mapred (delete) hadoop-common-project/hadoop-common/src/test/scripts/hadoop_get_verify_uservar.bats
          Hide
          hudson Hudson added a comment -

          SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #11547 (See https://builds.apache.org/job/Hadoop-trunk-Commit/11547/)
          HADOOP-14202. fix jsvc/secure user var inconsistencies (aw: rev 96cbb4fce2278c80b7e7e389d801ad31b97c0a61)

          • (edit) hadoop-hdfs-project/hadoop-hdfs/src/site/markdown/HdfsNfsGateway.md
          • (add) hadoop-common-project/hadoop-common/src/test/scripts/hadoop_detect_priv_subcmd.bats
          • (edit) hadoop-common-project/hadoop-common/src/main/conf/hadoop-env.sh
          • (edit) hadoop-common-project/hadoop-common/src/site/markdown/UnixShellGuide.md
          • (add) hadoop-common-project/hadoop-common/src/test/scripts/hadoop_verify_user_resolves.bats
          • (delete) hadoop-common-project/hadoop-common/src/test/scripts/hadoop_get_verify_uservar.bats
          • (add) hadoop-common-project/hadoop-common/src/test/scripts/hadoop_verify_user_perm.bats
          • (edit) hadoop-mapreduce-project/bin/mapred
          • (edit) hadoop-common-project/hadoop-common/src/site/markdown/SecureMode.md
          • (add) hadoop-common-project/hadoop-common/src/test/scripts/hadoop_build_custom_subcmd_var.bats
          • (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/bin/hdfs-config.sh
          • (edit) hadoop-yarn-project/hadoop-yarn/bin/yarn-config.sh
          • (delete) hadoop-common-project/hadoop-common/src/test/scripts/hadoop_verify_user.bats
          • (edit) hadoop-mapreduce-project/bin/mapred-config.sh
          • (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/bin/hdfs
          • (edit) hadoop-common-project/hadoop-common/src/main/bin/hadoop-config.sh
          • (edit) hadoop-common-project/hadoop-common/src/main/bin/hadoop
          • (edit) hadoop-common-project/hadoop-common/src/main/bin/hadoop-functions.sh
          • (edit) hadoop-yarn-project/hadoop-yarn/bin/yarn
          Show
          hudson Hudson added a comment - SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #11547 (See https://builds.apache.org/job/Hadoop-trunk-Commit/11547/ ) HADOOP-14202 . fix jsvc/secure user var inconsistencies (aw: rev 96cbb4fce2278c80b7e7e389d801ad31b97c0a61) (edit) hadoop-hdfs-project/hadoop-hdfs/src/site/markdown/HdfsNfsGateway.md (add) hadoop-common-project/hadoop-common/src/test/scripts/hadoop_detect_priv_subcmd.bats (edit) hadoop-common-project/hadoop-common/src/main/conf/hadoop-env.sh (edit) hadoop-common-project/hadoop-common/src/site/markdown/UnixShellGuide.md (add) hadoop-common-project/hadoop-common/src/test/scripts/hadoop_verify_user_resolves.bats (delete) hadoop-common-project/hadoop-common/src/test/scripts/hadoop_get_verify_uservar.bats (add) hadoop-common-project/hadoop-common/src/test/scripts/hadoop_verify_user_perm.bats (edit) hadoop-mapreduce-project/bin/mapred (edit) hadoop-common-project/hadoop-common/src/site/markdown/SecureMode.md (add) hadoop-common-project/hadoop-common/src/test/scripts/hadoop_build_custom_subcmd_var.bats (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/bin/hdfs-config.sh (edit) hadoop-yarn-project/hadoop-yarn/bin/yarn-config.sh (delete) hadoop-common-project/hadoop-common/src/test/scripts/hadoop_verify_user.bats (edit) hadoop-mapreduce-project/bin/mapred-config.sh (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/bin/hdfs (edit) hadoop-common-project/hadoop-common/src/main/bin/hadoop-config.sh (edit) hadoop-common-project/hadoop-common/src/main/bin/hadoop (edit) hadoop-common-project/hadoop-common/src/main/bin/hadoop-functions.sh (edit) hadoop-yarn-project/hadoop-yarn/bin/yarn
          Hide
          aw Allen Wittenauer added a comment -

          Thanks!

          Committed.

          Show
          aw Allen Wittenauer added a comment - Thanks! Committed.
          Hide
          jzhuge John Zhuge added a comment -

          Thanks Allen Wittenauer.

          +1 LGTM

          Show
          jzhuge John Zhuge added a comment - Thanks Allen Wittenauer . +1 LGTM
          Hide
          hadoopqa Hadoop QA added a comment -
          +1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 23s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 6 new or modified test files.
          0 mvndep 1m 50s Maven dependency ordering for branch
          +1 mvninstall 14m 53s trunk passed
          +1 mvnsite 6m 57s trunk passed
          0 mvndep 0m 15s Maven dependency ordering for patch
          +1 mvnsite 5m 52s the patch passed
          +1 shellcheck 1m 28s The patch generated 0 new + 81 unchanged - 19 fixed = 81 total (was 100)
          +1 shelldocs 0m 9s There were no new shelldocs issues.
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 unit 2m 1s hadoop-common in the patch passed.
          +1 unit 0m 59s hadoop-hdfs in the patch passed.
          +1 unit 11m 40s hadoop-yarn in the patch passed.
          +1 unit 2m 15s hadoop-mapreduce-project in the patch passed.
          +1 asflicense 0m 38s The patch does not generate ASF License warnings.
          49m 48s



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:a9ad5d6
          JIRA Issue HADOOP-14202
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12862435/HADOOP-14202.04.patch
          Optional Tests asflicense mvnsite unit shellcheck shelldocs
          uname Linux f4dbb2207119 3.13.0-106-generic #153-Ubuntu SMP Tue Dec 6 15:44:32 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / e7167e4
          shellcheck v0.4.5
          Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/12059/testReport/
          modules C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs hadoop-yarn-project/hadoop-yarn hadoop-mapreduce-project U: .
          Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/12059/console
          Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - +1 overall Vote Subsystem Runtime Comment 0 reexec 0m 23s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 6 new or modified test files. 0 mvndep 1m 50s Maven dependency ordering for branch +1 mvninstall 14m 53s trunk passed +1 mvnsite 6m 57s trunk passed 0 mvndep 0m 15s Maven dependency ordering for patch +1 mvnsite 5m 52s the patch passed +1 shellcheck 1m 28s The patch generated 0 new + 81 unchanged - 19 fixed = 81 total (was 100) +1 shelldocs 0m 9s There were no new shelldocs issues. +1 whitespace 0m 0s The patch has no whitespace issues. +1 unit 2m 1s hadoop-common in the patch passed. +1 unit 0m 59s hadoop-hdfs in the patch passed. +1 unit 11m 40s hadoop-yarn in the patch passed. +1 unit 2m 15s hadoop-mapreduce-project in the patch passed. +1 asflicense 0m 38s The patch does not generate ASF License warnings. 49m 48s Subsystem Report/Notes Docker Image:yetus/hadoop:a9ad5d6 JIRA Issue HADOOP-14202 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12862435/HADOOP-14202.04.patch Optional Tests asflicense mvnsite unit shellcheck shelldocs uname Linux f4dbb2207119 3.13.0-106-generic #153-Ubuntu SMP Tue Dec 6 15:44:32 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / e7167e4 shellcheck v0.4.5 Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/12059/testReport/ modules C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs hadoop-yarn-project/hadoop-yarn hadoop-mapreduce-project U: . Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/12059/console Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          aw Allen Wittenauer added a comment -

          new blank line at EOF. "git am" complains.

          I always commit with smart-apply-patch in committer mode which will fix these at commit time.

          Show
          aw Allen Wittenauer added a comment - new blank line at EOF. "git am" complains. I always commit with smart-apply-patch in committer mode which will fix these at commit time.
          Hide
          jzhuge John Zhuge added a comment -

          +1 LGTM. A few nitpicks, up to you to fix or not.

          hadoop-functions.sh

          • 2142-2143, …: match local var names with @param names. No need to enforce since there are so many functions with the same issue.

          UnixShellGuide.md

          • new blank line at EOF. "git am" complains.

          hadoop_detect_priv_subcmd.bats

          • new blank line at EOF. "git am" complains.

          hadoop_verify_user_resolves.bats

          • new blank line at EOF. "git am" complains.
          Show
          jzhuge John Zhuge added a comment - +1 LGTM. A few nitpicks, up to you to fix or not. hadoop-functions.sh 2142-2143, …: match local var names with @param names. No need to enforce since there are so many functions with the same issue. UnixShellGuide.md new blank line at EOF. "git am" complains. hadoop_detect_priv_subcmd.bats new blank line at EOF. "git am" complains. hadoop_verify_user_resolves.bats new blank line at EOF. "git am" complains.
          Hide
          aw Allen Wittenauer added a comment -

          -04:

          • fix the tabs
          Show
          aw Allen Wittenauer added a comment - -04: fix the tabs
          Hide
          aw Allen Wittenauer added a comment -

          The patch 5 line(s) with tabs.

          Argh. I switched computers and I don't have my editor settings correct yet.

          Show
          aw Allen Wittenauer added a comment - The patch 5 line(s) with tabs. Argh. I switched computers and I don't have my editor settings correct yet.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 14s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 6 new or modified test files.
          0 mvndep 0m 14s Maven dependency ordering for branch
          +1 mvninstall 12m 56s trunk passed
          +1 mvnsite 6m 6s trunk passed
          0 mvndep 0m 16s Maven dependency ordering for patch
          +1 mvnsite 6m 1s the patch passed
          +1 shellcheck 1m 22s The patch generated 0 new + 81 unchanged - 19 fixed = 81 total (was 100)
          +1 shelldocs 0m 8s There were no new shelldocs issues.
          -1 whitespace 0m 0s The patch 5 line(s) with tabs.
          +1 unit 1m 50s hadoop-common in the patch passed.
          +1 unit 0m 58s hadoop-hdfs in the patch passed.
          +1 unit 10m 58s hadoop-yarn in the patch passed.
          +1 unit 2m 16s hadoop-mapreduce-project in the patch passed.
          +1 asflicense 0m 34s The patch does not generate ASF License warnings.
          44m 22s



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:a9ad5d6
          JIRA Issue HADOOP-14202
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12862392/HADOOP-14202.03.patch
          Optional Tests asflicense mvnsite unit shellcheck shelldocs
          uname Linux 2f43b69033e7 3.13.0-106-generic #153-Ubuntu SMP Tue Dec 6 15:44:32 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / a49fac5
          shellcheck v0.4.5
          whitespace https://builds.apache.org/job/PreCommit-HADOOP-Build/12051/artifact/patchprocess/whitespace-tabs.txt
          Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/12051/testReport/
          modules C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs hadoop-yarn-project/hadoop-yarn hadoop-mapreduce-project U: .
          Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/12051/console
          Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 14s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 6 new or modified test files. 0 mvndep 0m 14s Maven dependency ordering for branch +1 mvninstall 12m 56s trunk passed +1 mvnsite 6m 6s trunk passed 0 mvndep 0m 16s Maven dependency ordering for patch +1 mvnsite 6m 1s the patch passed +1 shellcheck 1m 22s The patch generated 0 new + 81 unchanged - 19 fixed = 81 total (was 100) +1 shelldocs 0m 8s There were no new shelldocs issues. -1 whitespace 0m 0s The patch 5 line(s) with tabs. +1 unit 1m 50s hadoop-common in the patch passed. +1 unit 0m 58s hadoop-hdfs in the patch passed. +1 unit 10m 58s hadoop-yarn in the patch passed. +1 unit 2m 16s hadoop-mapreduce-project in the patch passed. +1 asflicense 0m 34s The patch does not generate ASF License warnings. 44m 22s Subsystem Report/Notes Docker Image:yetus/hadoop:a9ad5d6 JIRA Issue HADOOP-14202 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12862392/HADOOP-14202.03.patch Optional Tests asflicense mvnsite unit shellcheck shelldocs uname Linux 2f43b69033e7 3.13.0-106-generic #153-Ubuntu SMP Tue Dec 6 15:44:32 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / a49fac5 shellcheck v0.4.5 whitespace https://builds.apache.org/job/PreCommit-HADOOP-Build/12051/artifact/patchprocess/whitespace-tabs.txt Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/12051/testReport/ modules C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs hadoop-yarn-project/hadoop-yarn hadoop-mapreduce-project U: . Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/12051/console Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          aw Allen Wittenauer added a comment -

          -03:

          • remove some more shellcheck overrides that are no longer needed
          • updated documentation to address feedback
          • fix a bug with JSVC_HOME where the default wasn't actually being set
          • flip the return state for hadoop_detect_priv_subcmd
          • add a test for hadoop_detect_priv_subcmd
          • add a check for SECURE_USER in hadoop_uservar_su to default to a working state if the current process is privileged but _USER is undefined
          • check the error message for hadoop_uservar_su because if we are trying to stop a daemon "launch" makes zero sense.
          • rework hadoop_generic_java_subcmd_handler a bit to better handle 3rd party stuff and make the comments more sane
          • clean up hadoop-env.sh to reflect reality
          Show
          aw Allen Wittenauer added a comment - -03: remove some more shellcheck overrides that are no longer needed updated documentation to address feedback fix a bug with JSVC_HOME where the default wasn't actually being set flip the return state for hadoop_detect_priv_subcmd add a test for hadoop_detect_priv_subcmd add a check for SECURE_USER in hadoop_uservar_su to default to a working state if the current process is privileged but _USER is undefined check the error message for hadoop_uservar_su because if we are trying to stop a daemon "launch" makes zero sense. rework hadoop_generic_java_subcmd_handler a bit to better handle 3rd party stuff and make the comments more sane clean up hadoop-env.sh to reflect reality
          Hide
          jzhuge John Zhuge added a comment -

          Thanks for the clarification on Bash 3.2.

          Ok to keep HADOOP_SUBCMD_SECURESERVICE. Should you put the var back to UnixShellGuide.md?

          Show
          jzhuge John Zhuge added a comment - Thanks for the clarification on Bash 3.2. Ok to keep HADOOP_SUBCMD_SECURESERVICE. Should you put the var back to UnixShellGuide.md?
          Hide
          aw Allen Wittenauer added a comment -

          Thanks!

          Which feature requires Bash 3.2 ?

          I changed this not so much for the changes made in this patch, but to get rid of the last shellcheck error in hadoop-config.sh while I was fixing the other ones. That said, the one change in 3.2 I'm specifically targeting is this one:

          The =~ Regular Expression match operator no longer requires quoting of the pattern within [[ ... ]].
          

          We do this all over the place and if someone were to run this on 3.0 or 3.1, I'm fairly certain the world will tip over.

          478: move to 2536?

            HADOOP_SUBCMD_SECURESERVICE=false
          

          We set this here so that 3rd parties can override via shell profile later if they have some other way to do the work. (e.g., maybe they overlay hadoop_secure_daemon_handler to use pfexec using RBAC instead of jsvc. No extra pids or logs required, so a lot of this extra handling goes out the window.)

          Has HADOOP_SUBCMD_SECUREUSER become a local var in hadoop_generic_java_subcmd_handler?

          Good catch. I was thinking that it might be useful for 3rd parties (i.e. function replacement), but that doesn't make much sense. HADOOP_SECURE_USER is really what they should be checking, even if the user later changes it via cmd_subcmd_SECURE_USER.

          Show
          aw Allen Wittenauer added a comment - Thanks! Which feature requires Bash 3.2 ? I changed this not so much for the changes made in this patch, but to get rid of the last shellcheck error in hadoop-config.sh while I was fixing the other ones. That said, the one change in 3.2 I'm specifically targeting is this one: The =~ Regular Expression match operator no longer requires quoting of the pattern within [[ ... ]]. We do this all over the place and if someone were to run this on 3.0 or 3.1, I'm fairly certain the world will tip over. 478: move to 2536? HADOOP_SUBCMD_SECURESERVICE= false We set this here so that 3rd parties can override via shell profile later if they have some other way to do the work. (e.g., maybe they overlay hadoop_secure_daemon_handler to use pfexec using RBAC instead of jsvc. No extra pids or logs required, so a lot of this extra handling goes out the window.) Has HADOOP_SUBCMD_SECUREUSER become a local var in hadoop_generic_java_subcmd_handler? Good catch. I was thinking that it might be useful for 3rd parties (i.e. function replacement), but that doesn't make much sense. HADOOP_SECURE_USER is really what they should be checking, even if the user later changes it via cmd_subcmd_SECURE_USER.
          Hide
          jzhuge John Zhuge added a comment -

          Thanks Allen Wittenauer for the great effort. Nice deduplication with hadoop_generic_java_subcmd_handler.

          hadoop-config.sh

          • Which feature requires Bash 3.2 ?

          hadoop-functions.sh

          • 118, 120: such a perfectionist
          • 478: move to 2536?
          • 2134-2135, 2157-2159, 2185, 2204-2205, 2266, 2324: match local var names, or change local var names to match these. Not sure about this though, because there are so many functions with the same issue.
          • 2517: Has HADOOP_SUBCMD_SECUREUSER become a local var in hadoop_generic_java_subcmd_handler? It should be lower case then. Same for HADOOP_SUBCMD_SECURESERVICE if we move 478 to 2536

          UnixShellGuide.md

          • Blank line at EOF

          hadoop_verify_user_resolves.bats

          • Blank line at EOF
          Show
          jzhuge John Zhuge added a comment - Thanks Allen Wittenauer for the great effort. Nice deduplication with hadoop_generic_java_subcmd_handler. hadoop-config.sh Which feature requires Bash 3.2 ? hadoop-functions.sh 118, 120: such a perfectionist 478: move to 2536? 2134-2135, 2157-2159, 2185, 2204-2205, 2266, 2324: match local var names, or change local var names to match these. Not sure about this though, because there are so many functions with the same issue. 2517: Has HADOOP_SUBCMD_SECUREUSER become a local var in hadoop_generic_java_subcmd_handler? It should be lower case then. Same for HADOOP_SUBCMD_SECURESERVICE if we move 478 to 2536 UnixShellGuide.md Blank line at EOF hadoop_verify_user_resolves.bats Blank line at EOF
          Hide
          busbey Sean Busbey added a comment -

          yeah, stating it in the concrete terms of "note that the daemon_* env variables will also be different if we're running with privilege" or "the daemon stuff" instead of "daemon_* env variables" depending on what kind of tone you want to keep.

          Show
          busbey Sean Busbey added a comment - yeah, stating it in the concrete terms of "note that the daemon_* env variables will also be different if we're running with privilege" or "the daemon stuff" instead of "daemon_* env variables" depending on what kind of tone you want to keep.
          Hide
          aw Allen Wittenauer added a comment -

          The intent is to warn people and to make sure they understand what is going on. I have clearly failed. haha. The big issue is that the if clause needs to be there for the daemon_* env vars ("these settings") are getting configured slightly differently. I guess I should just state that though.

          Show
          aw Allen Wittenauer added a comment - The intent is to warn people and to make sure they understand what is going on. I have clearly failed. haha. The big issue is that the if clause needs to be there for the daemon_* env vars ("these settings") are getting configured slightly differently. I guess I should just state that though.
          Hide
          busbey Sean Busbey added a comment -
          2510	  # are we expected to be running with privilege?
          2511	  # if yes, then we need to define all of the priv and daemon stuff
          2512	  # if not, then we just need to define daemon stuff.
          2513	  # note that these settings are different between the two
          

          This phrasing is a bit awkward. What is "these settings" referring to?

          Show
          busbey Sean Busbey added a comment - 2510 # are we expected to be running with privilege? 2511 # if yes, then we need to define all of the priv and daemon stuff 2512 # if not, then we just need to define daemon stuff. 2513 # note that these settings are different between the two This phrasing is a bit awkward. What is "these settings" referring to?
          Hide
          hadoopqa Hadoop QA added a comment -
          +1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 19s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 5 new or modified test files.
          0 mvndep 0m 20s Maven dependency ordering for branch
          +1 mvninstall 13m 0s trunk passed
          +1 mvnsite 5m 54s trunk passed
          0 mvndep 0m 15s Maven dependency ordering for patch
          +1 mvnsite 5m 39s the patch passed
          +1 shellcheck 1m 17s The patch generated 0 new + 81 unchanged - 19 fixed = 81 total (was 100)
          +1 shelldocs 0m 10s There were no new shelldocs issues.
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 unit 2m 3s hadoop-common in the patch passed.
          +1 unit 0m 50s hadoop-hdfs in the patch passed.
          +1 unit 11m 39s hadoop-yarn in the patch passed.
          +1 unit 1m 51s hadoop-mapreduce-project in the patch passed.
          +1 asflicense 0m 37s The patch does not generate ASF License warnings.
          44m 27s



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:a9ad5d6
          JIRA Issue HADOOP-14202
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12860740/HADOOP-14202.02.patch
          Optional Tests asflicense mvnsite unit shellcheck shelldocs
          uname Linux 4c757f3e83f2 3.13.0-106-generic #153-Ubuntu SMP Tue Dec 6 15:44:32 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / cd014d5
          shellcheck v0.4.5
          Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/11946/testReport/
          modules C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs hadoop-yarn-project/hadoop-yarn hadoop-mapreduce-project U: .
          Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/11946/console
          Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - +1 overall Vote Subsystem Runtime Comment 0 reexec 0m 19s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 5 new or modified test files. 0 mvndep 0m 20s Maven dependency ordering for branch +1 mvninstall 13m 0s trunk passed +1 mvnsite 5m 54s trunk passed 0 mvndep 0m 15s Maven dependency ordering for patch +1 mvnsite 5m 39s the patch passed +1 shellcheck 1m 17s The patch generated 0 new + 81 unchanged - 19 fixed = 81 total (was 100) +1 shelldocs 0m 10s There were no new shelldocs issues. +1 whitespace 0m 0s The patch has no whitespace issues. +1 unit 2m 3s hadoop-common in the patch passed. +1 unit 0m 50s hadoop-hdfs in the patch passed. +1 unit 11m 39s hadoop-yarn in the patch passed. +1 unit 1m 51s hadoop-mapreduce-project in the patch passed. +1 asflicense 0m 37s The patch does not generate ASF License warnings. 44m 27s Subsystem Report/Notes Docker Image:yetus/hadoop:a9ad5d6 JIRA Issue HADOOP-14202 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12860740/HADOOP-14202.02.patch Optional Tests asflicense mvnsite unit shellcheck shelldocs uname Linux 4c757f3e83f2 3.13.0-106-generic #153-Ubuntu SMP Tue Dec 6 15:44:32 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / cd014d5 shellcheck v0.4.5 Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/11946/testReport/ modules C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs hadoop-yarn-project/hadoop-yarn hadoop-mapreduce-project U: . Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/11946/console Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          aw Allen Wittenauer added a comment -

          -02:

          • fix shellcheck errors
          • remove uid test since it doesn't work consistently; leave a skip test telling why
          Show
          aw Allen Wittenauer added a comment - -02: fix shellcheck errors remove uid test since it doesn't work consistently; leave a skip test telling why
          Hide
          aw Allen Wittenauer added a comment - - edited

          With shellcheck -x enabled, that's a lot better.

          6 new + 100 unchanged - 0 fixed = 106 total (was 100)
          

          to

          3 new + 88 unchanged - 12 fixed = 91 total (was 100)
          
          Show
          aw Allen Wittenauer added a comment - - edited With shellcheck -x enabled, that's a lot better. 6 new + 100 unchanged - 0 fixed = 106 total (was 100) to 3 new + 88 unchanged - 12 fixed = 91 total (was 100)
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 25s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 5 new or modified test files.
          0 mvndep 0m 21s Maven dependency ordering for branch
          +1 mvninstall 13m 28s trunk passed
          +1 mvnsite 6m 7s trunk passed
          0 mvndep 0m 16s Maven dependency ordering for patch
          +1 mvnsite 6m 22s the patch passed
          -1 shellcheck 0m 56s The patch generated 3 new + 88 unchanged - 12 fixed = 91 total (was 100)
          +1 shelldocs 0m 9s There were no new shelldocs issues.
          +1 whitespace 0m 0s The patch has no whitespace issues.
          -1 unit 2m 7s hadoop-common in the patch failed.
          +1 unit 0m 51s hadoop-hdfs in the patch passed.
          +1 unit 12m 31s hadoop-yarn in the patch passed.
          +1 unit 1m 53s hadoop-mapreduce-project in the patch passed.
          +1 asflicense 0m 37s The patch does not generate ASF License warnings.
          46m 33s



          Reason Tests
          Failed TAP tests hadoop_verify_user_resolves.bats.tap



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:a9ad5d6
          JIRA Issue HADOOP-14202
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12860039/HADOOP-14202.01.patch
          Optional Tests asflicense mvnsite unit shellcheck shelldocs
          uname Linux 096e90bbc35e 3.13.0-106-generic #153-Ubuntu SMP Tue Dec 6 15:44:32 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / db2adf3
          shellcheck v0.4.5
          shellcheck https://builds.apache.org/job/PreCommit-HADOOP-Build/11944/artifact/patchprocess/diff-patch-shellcheck.txt
          TAP logs https://builds.apache.org/job/PreCommit-HADOOP-Build/11944/artifact/patchprocess/patch-hadoop-common-project_hadoop-common.tap
          unit https://builds.apache.org/job/PreCommit-HADOOP-Build/11944/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common.txt
          Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/11944/testReport/
          modules C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs hadoop-yarn-project/hadoop-yarn hadoop-mapreduce-project U: .
          Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/11944/console
          Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 25s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 5 new or modified test files. 0 mvndep 0m 21s Maven dependency ordering for branch +1 mvninstall 13m 28s trunk passed +1 mvnsite 6m 7s trunk passed 0 mvndep 0m 16s Maven dependency ordering for patch +1 mvnsite 6m 22s the patch passed -1 shellcheck 0m 56s The patch generated 3 new + 88 unchanged - 12 fixed = 91 total (was 100) +1 shelldocs 0m 9s There were no new shelldocs issues. +1 whitespace 0m 0s The patch has no whitespace issues. -1 unit 2m 7s hadoop-common in the patch failed. +1 unit 0m 51s hadoop-hdfs in the patch passed. +1 unit 12m 31s hadoop-yarn in the patch passed. +1 unit 1m 53s hadoop-mapreduce-project in the patch passed. +1 asflicense 0m 37s The patch does not generate ASF License warnings. 46m 33s Reason Tests Failed TAP tests hadoop_verify_user_resolves.bats.tap Subsystem Report/Notes Docker Image:yetus/hadoop:a9ad5d6 JIRA Issue HADOOP-14202 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12860039/HADOOP-14202.01.patch Optional Tests asflicense mvnsite unit shellcheck shelldocs uname Linux 096e90bbc35e 3.13.0-106-generic #153-Ubuntu SMP Tue Dec 6 15:44:32 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / db2adf3 shellcheck v0.4.5 shellcheck https://builds.apache.org/job/PreCommit-HADOOP-Build/11944/artifact/patchprocess/diff-patch-shellcheck.txt TAP logs https://builds.apache.org/job/PreCommit-HADOOP-Build/11944/artifact/patchprocess/patch-hadoop-common-project_hadoop-common.tap unit https://builds.apache.org/job/PreCommit-HADOOP-Build/11944/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common.txt Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/11944/testReport/ modules C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs hadoop-yarn-project/hadoop-yarn hadoop-mapreduce-project U: . Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/11944/console Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          aw Allen Wittenauer added a comment -

          hadoop_verify_user_resolves.bats.tap

          OK, this is failing because id -u uid isn't POSIX. I'll just remove that test.

          Filed YETUS-498 to add -x support for Apache Yetus' shellcheck test.

          Show
          aw Allen Wittenauer added a comment - hadoop_verify_user_resolves.bats.tap OK, this is failing because id -u uid isn't POSIX. I'll just remove that test. Filed YETUS-498 to add -x support for Apache Yetus' shellcheck test.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 22s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 5 new or modified test files.
          0 mvndep 0m 14s Maven dependency ordering for branch
          +1 mvninstall 13m 9s trunk passed
          +1 mvnsite 6m 31s trunk passed
          0 mvndep 0m 33s Maven dependency ordering for patch
          +1 mvnsite 5m 48s the patch passed
          -1 shellcheck 0m 12s The patch generated 16 new + 88 unchanged - 12 fixed = 104 total (was 100)
          +1 shelldocs 0m 10s There were no new shelldocs issues.
          +1 whitespace 0m 0s The patch has no whitespace issues.
          -1 unit 2m 4s hadoop-common in the patch failed.
          +1 unit 0m 50s hadoop-hdfs in the patch passed.
          +1 unit 16m 5s hadoop-yarn in the patch passed.
          +1 unit 1m 51s hadoop-mapreduce-project in the patch passed.
          +1 asflicense 0m 37s The patch does not generate ASF License warnings.
          48m 56s



          Reason Tests
          Failed TAP tests hadoop_verify_user_resolves.bats.tap



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:a9ad5d6
          JIRA Issue HADOOP-14202
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12860039/HADOOP-14202.01.patch
          Optional Tests asflicense mvnsite unit shellcheck shelldocs
          uname Linux 542293b570ca 3.13.0-106-generic #153-Ubuntu SMP Tue Dec 6 15:44:32 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / f462e1f
          shellcheck v0.4.5
          shellcheck https://builds.apache.org/job/PreCommit-HADOOP-Build/11885/artifact/patchprocess/diff-patch-shellcheck.txt
          TAP logs https://builds.apache.org/job/PreCommit-HADOOP-Build/11885/artifact/patchprocess/patch-hadoop-common-project_hadoop-common.tap
          unit https://builds.apache.org/job/PreCommit-HADOOP-Build/11885/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common.txt
          Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/11885/testReport/
          modules C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs hadoop-yarn-project/hadoop-yarn hadoop-mapreduce-project U: .
          Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/11885/console
          Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 22s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 5 new or modified test files. 0 mvndep 0m 14s Maven dependency ordering for branch +1 mvninstall 13m 9s trunk passed +1 mvnsite 6m 31s trunk passed 0 mvndep 0m 33s Maven dependency ordering for patch +1 mvnsite 5m 48s the patch passed -1 shellcheck 0m 12s The patch generated 16 new + 88 unchanged - 12 fixed = 104 total (was 100) +1 shelldocs 0m 10s There were no new shelldocs issues. +1 whitespace 0m 0s The patch has no whitespace issues. -1 unit 2m 4s hadoop-common in the patch failed. +1 unit 0m 50s hadoop-hdfs in the patch passed. +1 unit 16m 5s hadoop-yarn in the patch passed. +1 unit 1m 51s hadoop-mapreduce-project in the patch passed. +1 asflicense 0m 37s The patch does not generate ASF License warnings. 48m 56s Reason Tests Failed TAP tests hadoop_verify_user_resolves.bats.tap Subsystem Report/Notes Docker Image:yetus/hadoop:a9ad5d6 JIRA Issue HADOOP-14202 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12860039/HADOOP-14202.01.patch Optional Tests asflicense mvnsite unit shellcheck shelldocs uname Linux 542293b570ca 3.13.0-106-generic #153-Ubuntu SMP Tue Dec 6 15:44:32 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / f462e1f shellcheck v0.4.5 shellcheck https://builds.apache.org/job/PreCommit-HADOOP-Build/11885/artifact/patchprocess/diff-patch-shellcheck.txt TAP logs https://builds.apache.org/job/PreCommit-HADOOP-Build/11885/artifact/patchprocess/patch-hadoop-common-project_hadoop-common.tap unit https://builds.apache.org/job/PreCommit-HADOOP-Build/11885/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common.txt Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/11885/testReport/ modules C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs hadoop-yarn-project/hadoop-yarn hadoop-mapreduce-project U: . Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/11885/console Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          aw Allen Wittenauer added a comment -

          -01:

          • Add a shellcheck source for yetus to knock out some disable line issues caused by the mass deletion/consolidation of code. Hopefully this works.
          • Move the DN and NFS deprecations to happen only when DN and NFS are called. This prevents the wrong values going in too early, conflicts, etc.
          Show
          aw Allen Wittenauer added a comment - -01: Add a shellcheck source for yetus to knock out some disable line issues caused by the mass deletion/consolidation of code. Hopefully this works. Move the DN and NFS deprecations to happen only when DN and NFS are called. This prevents the wrong values going in too early, conflicts, etc.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 25s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 5 new or modified test files.
          0 mvndep 1m 49s Maven dependency ordering for branch
          +1 mvninstall 12m 26s trunk passed
          +1 mvnsite 5m 54s trunk passed
          0 mvndep 0m 15s Maven dependency ordering for patch
          +1 mvnsite 5m 37s the patch passed
          -1 shellcheck 0m 12s The patch generated 6 new + 100 unchanged - 0 fixed = 106 total (was 100)
          +1 shelldocs 0m 9s There were no new shelldocs issues.
          +1 whitespace 0m 0s The patch has no whitespace issues.
          -1 unit 2m 6s hadoop-common in the patch failed.
          +1 unit 0m 51s hadoop-hdfs in the patch passed.
          +1 unit 6m 48s hadoop-yarn in the patch passed.
          +1 unit 2m 8s hadoop-mapreduce-project in the patch passed.
          +1 asflicense 0m 48s The patch does not generate ASF License warnings.
          40m 6s



          Reason Tests
          Failed TAP tests hadoop_verify_user_resolves.bats.tap



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:a9ad5d6
          JIRA Issue HADOOP-14202
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12859628/HADOOP-14202.00.patch
          Optional Tests asflicense mvnsite unit shellcheck shelldocs
          uname Linux 71a654949d9e 3.13.0-106-generic #153-Ubuntu SMP Tue Dec 6 15:44:32 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / 6c399a8
          shellcheck v0.4.5
          shellcheck https://builds.apache.org/job/PreCommit-HADOOP-Build/11859/artifact/patchprocess/diff-patch-shellcheck.txt
          TAP logs https://builds.apache.org/job/PreCommit-HADOOP-Build/11859/artifact/patchprocess/patch-hadoop-common-project_hadoop-common.tap
          unit https://builds.apache.org/job/PreCommit-HADOOP-Build/11859/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common.txt
          Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/11859/testReport/
          modules C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs hadoop-yarn-project/hadoop-yarn hadoop-mapreduce-project U: .
          Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/11859/console
          Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 25s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 5 new or modified test files. 0 mvndep 1m 49s Maven dependency ordering for branch +1 mvninstall 12m 26s trunk passed +1 mvnsite 5m 54s trunk passed 0 mvndep 0m 15s Maven dependency ordering for patch +1 mvnsite 5m 37s the patch passed -1 shellcheck 0m 12s The patch generated 6 new + 100 unchanged - 0 fixed = 106 total (was 100) +1 shelldocs 0m 9s There were no new shelldocs issues. +1 whitespace 0m 0s The patch has no whitespace issues. -1 unit 2m 6s hadoop-common in the patch failed. +1 unit 0m 51s hadoop-hdfs in the patch passed. +1 unit 6m 48s hadoop-yarn in the patch passed. +1 unit 2m 8s hadoop-mapreduce-project in the patch passed. +1 asflicense 0m 48s The patch does not generate ASF License warnings. 40m 6s Reason Tests Failed TAP tests hadoop_verify_user_resolves.bats.tap Subsystem Report/Notes Docker Image:yetus/hadoop:a9ad5d6 JIRA Issue HADOOP-14202 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12859628/HADOOP-14202.00.patch Optional Tests asflicense mvnsite unit shellcheck shelldocs uname Linux 71a654949d9e 3.13.0-106-generic #153-Ubuntu SMP Tue Dec 6 15:44:32 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 6c399a8 shellcheck v0.4.5 shellcheck https://builds.apache.org/job/PreCommit-HADOOP-Build/11859/artifact/patchprocess/diff-patch-shellcheck.txt TAP logs https://builds.apache.org/job/PreCommit-HADOOP-Build/11859/artifact/patchprocess/patch-hadoop-common-project_hadoop-common.tap unit https://builds.apache.org/job/PreCommit-HADOOP-Build/11859/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common.txt Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/11859/testReport/ modules C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs hadoop-yarn-project/hadoop-yarn hadoop-mapreduce-project U: . Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/11859/console Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          aw Allen Wittenauer added a comment - - edited
          • Make everything use secure_user
          • rename hadoop_verify_user to hadoop_verify_user_perm to better reflect reality
          • add a new var creator function to condense code further
          • remove vast amounts of boiler plate from the base bin commands. required moving mapred.jobsummary.logger definition in mapred:
           13 files changed, 285 insertions(+), 322 deletions(-)
          
          Show
          aw Allen Wittenauer added a comment - - edited Make everything use secure_user rename hadoop_verify_user to hadoop_verify_user_perm to better reflect reality add a new var creator function to condense code further remove vast amounts of boiler plate from the base bin commands. required moving mapred.jobsummary.logger definition in mapred: 13 files changed, 285 insertions(+), 322 deletions(-)
          Hide
          aw Allen Wittenauer added a comment -

          Now I remember why I pushed this off: the user vars are used to determine if something is running with privilege. This make cleaning this up significantly more complicated.

          Show
          aw Allen Wittenauer added a comment - Now I remember why I pushed this off: the user vars are used to determine if something is running with privilege. This make cleaning this up significantly more complicated.

            People

            • Assignee:
              aw Allen Wittenauer
              Reporter:
              aw Allen Wittenauer
            • Votes:
              0 Vote for this issue
              Watchers:
              8 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development