Uploaded image for project: 'Hadoop YARN'
  1. Hadoop YARN
  2. YARN-3768

ArrayIndexOutOfBoundsException with empty environment variables

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 2.5.0
    • Fix Version/s: 2.8.0, 3.0.0-alpha1
    • Component/s: yarn
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      Looking at line 80 of org.apache.hadoop.yarn.util.Apps an index out of range exception occurs if an environment variable is encountered without a value.

      java.lang.ArrayIndexOutOfBoundsException: 1
      	at org.apache.hadoop.yarn.util.Apps.setEnvFromInputString(Apps.java:80)
      

      I believe this occurs because java will not return empty strings from the split method. Similar to this http://stackoverflow.com/questions/14602062/java-string-split-removed-empty-values

      1. YARN-3768.000.patch
        3 kB
        zhihai xu
      2. YARN-3768.001.patch
        3 kB
        zhihai xu
      3. YARN-3768.002.patch
        6 kB
        Gera Shegalov
      4. YARN-3768.003.patch
        6 kB
        Gera Shegalov
      5. YARN-3768.004.patch
        6 kB
        zhihai xu

        Issue Links

          Activity

          Hide
          zxu zhihai xu added a comment -

          Hi Joe Ferner, That is a good find. I can see the change at MAPREDUCE-5965 may trigger this bug. I can take up this issue if you don't mind. thanks for reporting this issue.

          Show
          zxu zhihai xu added a comment - Hi Joe Ferner , That is a good find. I can see the change at MAPREDUCE-5965 may trigger this bug. I can take up this issue if you don't mind. thanks for reporting this issue.
          Hide
          zxu zhihai xu added a comment -

          I uploaded a patch YARN-3768.000.patch which use cEnv.split("=", -1) to parse the string, so trailing empty strings won't be discarded.
          It will also skip bad format environment variable.

          Show
          zxu zhihai xu added a comment - I uploaded a patch YARN-3768 .000.patch which use cEnv.split("=", -1) to parse the string, so trailing empty strings won't be discarded. It will also skip bad format environment variable.
          Hide
          hadoopqa Hadoop QA added a comment -



          -1 overall



          Vote Subsystem Runtime Comment
          0 pre-patch 16m 19s Pre-patch trunk compilation is healthy.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 tests included 0m 0s The patch appears to include 1 new or modified test files.
          -1 javac 7m 35s The applied patch generated 1 additional warning messages.
          +1 javadoc 9m 35s There were no new javadoc warning messages.
          +1 release audit 0m 24s The applied patch does not increase the total number of release audit warnings.
          +1 checkstyle 0m 53s There were no new checkstyle issues.
          +1 whitespace 0m 0s The patch has no lines that end in whitespace.
          +1 install 1m 37s mvn install still works.
          +1 eclipse:eclipse 0m 33s The patch built with eclipse:eclipse.
          +1 findbugs 1m 35s The patch does not introduce any new Findbugs (version 3.0.0) warnings.
          +1 yarn tests 1m 57s Tests passed in hadoop-yarn-common.
              40m 31s  



          Subsystem Report/Notes
          Patch URL http://issues.apache.org/jira/secure/attachment/12739163/YARN-3768.000.patch
          Optional Tests javadoc javac unit findbugs checkstyle
          git revision trunk / 83e8110
          javac https://builds.apache.org/job/PreCommit-YARN-Build/8240/artifact/patchprocess/diffJavacWarnings.txt
          hadoop-yarn-common test log https://builds.apache.org/job/PreCommit-YARN-Build/8240/artifact/patchprocess/testrun_hadoop-yarn-common.txt
          Test Results https://builds.apache.org/job/PreCommit-YARN-Build/8240/testReport/
          Java 1.7.0_55
          uname Linux asf906.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Console output https://builds.apache.org/job/PreCommit-YARN-Build/8240/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 pre-patch 16m 19s Pre-patch trunk compilation is healthy. +1 @author 0m 0s The patch does not contain any @author tags. +1 tests included 0m 0s The patch appears to include 1 new or modified test files. -1 javac 7m 35s The applied patch generated 1 additional warning messages. +1 javadoc 9m 35s There were no new javadoc warning messages. +1 release audit 0m 24s The applied patch does not increase the total number of release audit warnings. +1 checkstyle 0m 53s There were no new checkstyle issues. +1 whitespace 0m 0s The patch has no lines that end in whitespace. +1 install 1m 37s mvn install still works. +1 eclipse:eclipse 0m 33s The patch built with eclipse:eclipse. +1 findbugs 1m 35s The patch does not introduce any new Findbugs (version 3.0.0) warnings. +1 yarn tests 1m 57s Tests passed in hadoop-yarn-common.     40m 31s   Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12739163/YARN-3768.000.patch Optional Tests javadoc javac unit findbugs checkstyle git revision trunk / 83e8110 javac https://builds.apache.org/job/PreCommit-YARN-Build/8240/artifact/patchprocess/diffJavacWarnings.txt hadoop-yarn-common test log https://builds.apache.org/job/PreCommit-YARN-Build/8240/artifact/patchprocess/testrun_hadoop-yarn-common.txt Test Results https://builds.apache.org/job/PreCommit-YARN-Build/8240/testReport/ Java 1.7.0_55 uname Linux asf906.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Console output https://builds.apache.org/job/PreCommit-YARN-Build/8240/console This message was automatically generated.
          Hide
          hadoopqa Hadoop QA added a comment -



          +1 overall



          Vote Subsystem Runtime Comment
          0 pre-patch 15m 57s Pre-patch trunk compilation is healthy.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 tests included 0m 0s The patch appears to include 1 new or modified test files.
          +1 javac 7m 33s There were no new javac warning messages.
          +1 javadoc 9m 36s There were no new javadoc warning messages.
          +1 release audit 0m 22s The applied patch does not increase the total number of release audit warnings.
          +1 checkstyle 0m 55s There were no new checkstyle issues.
          +1 whitespace 0m 0s The patch has no lines that end in whitespace.
          +1 install 1m 33s mvn install still works.
          +1 eclipse:eclipse 0m 32s The patch built with eclipse:eclipse.
          +1 findbugs 1m 35s The patch does not introduce any new Findbugs (version 3.0.0) warnings.
          +1 yarn tests 1m 56s Tests passed in hadoop-yarn-common.
              40m 2s  



          Subsystem Report/Notes
          Patch URL http://issues.apache.org/jira/secure/attachment/12739173/YARN-3768.000.patch
          Optional Tests javadoc javac unit findbugs checkstyle
          git revision trunk / 83e8110
          hadoop-yarn-common test log https://builds.apache.org/job/PreCommit-YARN-Build/8241/artifact/patchprocess/testrun_hadoop-yarn-common.txt
          Test Results https://builds.apache.org/job/PreCommit-YARN-Build/8241/testReport/
          Java 1.7.0_55
          uname Linux asf906.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Console output https://builds.apache.org/job/PreCommit-YARN-Build/8241/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - +1 overall Vote Subsystem Runtime Comment 0 pre-patch 15m 57s Pre-patch trunk compilation is healthy. +1 @author 0m 0s The patch does not contain any @author tags. +1 tests included 0m 0s The patch appears to include 1 new or modified test files. +1 javac 7m 33s There were no new javac warning messages. +1 javadoc 9m 36s There were no new javadoc warning messages. +1 release audit 0m 22s The applied patch does not increase the total number of release audit warnings. +1 checkstyle 0m 55s There were no new checkstyle issues. +1 whitespace 0m 0s The patch has no lines that end in whitespace. +1 install 1m 33s mvn install still works. +1 eclipse:eclipse 0m 32s The patch built with eclipse:eclipse. +1 findbugs 1m 35s The patch does not introduce any new Findbugs (version 3.0.0) warnings. +1 yarn tests 1m 56s Tests passed in hadoop-yarn-common.     40m 2s   Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12739173/YARN-3768.000.patch Optional Tests javadoc javac unit findbugs checkstyle git revision trunk / 83e8110 hadoop-yarn-common test log https://builds.apache.org/job/PreCommit-YARN-Build/8241/artifact/patchprocess/testrun_hadoop-yarn-common.txt Test Results https://builds.apache.org/job/PreCommit-YARN-Build/8241/testReport/ Java 1.7.0_55 uname Linux asf906.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Console output https://builds.apache.org/job/PreCommit-YARN-Build/8241/console This message was automatically generated.
          Hide
          xgong Xuan Gong added a comment -

          Thanks for working on this. zhihai xu. Could you also add a testcase which can verify if we give a bad environment, it will not throw out the exception ?

          Also, why do we need to keep trailing empty strings ?

          Show
          xgong Xuan Gong added a comment - Thanks for working on this. zhihai xu . Could you also add a testcase which can verify if we give a bad environment, it will not throw out the exception ? Also, why do we need to keep trailing empty strings ?
          Hide
          zxu zhihai xu added a comment -

          Hi Xuan Gong, thanks for the review. I uploaded a new patch YARN-3768.001.patch, in which I add a test case to verify bad environment variables are skipped.
          About keeping trailing empty strings, it will depend on whether an Environment Variable with empty value is a valid use case.
          MAPREDUCE-5965 adds option to configure Environment Variable with empty value if stream.jobconf.truncate.limit is 0.
          It looks like an Environment Variable with empty value may be a valid use case.

          Show
          zxu zhihai xu added a comment - Hi Xuan Gong , thanks for the review. I uploaded a new patch YARN-3768 .001.patch, in which I add a test case to verify bad environment variables are skipped. About keeping trailing empty strings, it will depend on whether an Environment Variable with empty value is a valid use case. MAPREDUCE-5965 adds option to configure Environment Variable with empty value if stream.jobconf.truncate.limit is 0. It looks like an Environment Variable with empty value may be a valid use case.
          Hide
          hadoopqa Hadoop QA added a comment -



          +1 overall



          Vote Subsystem Runtime Comment
          0 pre-patch 16m 5s Pre-patch trunk compilation is healthy.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 tests included 0m 0s The patch appears to include 1 new or modified test files.
          +1 javac 7m 32s There were no new javac warning messages.
          +1 javadoc 9m 35s There were no new javadoc warning messages.
          +1 release audit 0m 22s The applied patch does not increase the total number of release audit warnings.
          +1 checkstyle 0m 53s There were no new checkstyle issues.
          +1 whitespace 0m 0s The patch has no lines that end in whitespace.
          +1 install 1m 32s mvn install still works.
          +1 eclipse:eclipse 0m 33s The patch built with eclipse:eclipse.
          +1 findbugs 1m 32s The patch does not introduce any new Findbugs (version 3.0.0) warnings.
          +1 yarn tests 1m 57s Tests passed in hadoop-yarn-common.
              40m 4s  



          Subsystem Report/Notes
          Patch URL http://issues.apache.org/jira/secure/attachment/12740968/YARN-3768.001.patch
          Optional Tests javadoc javac unit findbugs checkstyle
          git revision trunk / 445b132
          hadoop-yarn-common test log https://builds.apache.org/job/PreCommit-YARN-Build/8307/artifact/patchprocess/testrun_hadoop-yarn-common.txt
          Test Results https://builds.apache.org/job/PreCommit-YARN-Build/8307/testReport/
          Java 1.7.0_55
          uname Linux asf905.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Console output https://builds.apache.org/job/PreCommit-YARN-Build/8307/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - +1 overall Vote Subsystem Runtime Comment 0 pre-patch 16m 5s Pre-patch trunk compilation is healthy. +1 @author 0m 0s The patch does not contain any @author tags. +1 tests included 0m 0s The patch appears to include 1 new or modified test files. +1 javac 7m 32s There were no new javac warning messages. +1 javadoc 9m 35s There were no new javadoc warning messages. +1 release audit 0m 22s The applied patch does not increase the total number of release audit warnings. +1 checkstyle 0m 53s There were no new checkstyle issues. +1 whitespace 0m 0s The patch has no lines that end in whitespace. +1 install 1m 32s mvn install still works. +1 eclipse:eclipse 0m 33s The patch built with eclipse:eclipse. +1 findbugs 1m 32s The patch does not introduce any new Findbugs (version 3.0.0) warnings. +1 yarn tests 1m 57s Tests passed in hadoop-yarn-common.     40m 4s   Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12740968/YARN-3768.001.patch Optional Tests javadoc javac unit findbugs checkstyle git revision trunk / 445b132 hadoop-yarn-common test log https://builds.apache.org/job/PreCommit-YARN-Build/8307/artifact/patchprocess/testrun_hadoop-yarn-common.txt Test Results https://builds.apache.org/job/PreCommit-YARN-Build/8307/testReport/ Java 1.7.0_55 uname Linux asf905.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Console output https://builds.apache.org/job/PreCommit-YARN-Build/8307/console This message was automatically generated.
          Hide
          jira.shegalov Gera Shegalov added a comment -

          Instead of executing two regexes:

          first directly via Pattern p = Pattern.compile(Shell.getEnvironmentVariableRegex()) and then via split

          can we simply match via a single regex? we can use a capture group to get the value.

          Show
          jira.shegalov Gera Shegalov added a comment - Instead of executing two regexes: first directly via Pattern p = Pattern.compile(Shell.getEnvironmentVariableRegex()) and then via split can we simply match via a single regex? we can use a capture group to get the value.
          Hide
          zxu zhihai xu added a comment -

          Gera Shegalov, thanks for the suggestion! The current code will only look up the pattern getEnvironmentVariableRegex in value(parts[1]) and replace the matched substring with the stored Env Variable's value. I looked at java Matcher class, I couldn't find a way to do capture group and replacement at the same time with a single regex. Is it possible to use a single regex with capture group to do both split and replacement with different variable? If it is possible, Could you tell me how to do that?

          Show
          zxu zhihai xu added a comment - Gera Shegalov , thanks for the suggestion! The current code will only look up the pattern getEnvironmentVariableRegex in value( parts [1] ) and replace the matched substring with the stored Env Variable's value. I looked at java Matcher class, I couldn't find a way to do capture group and replacement at the same time with a single regex. Is it possible to use a single regex with capture group to do both split and replacement with different variable? If it is possible, Could you tell me how to do that?
          Hide
          jira.shegalov Gera Shegalov added a comment -

          You are right zhihai xu, and I actually meant to combine matching k=v pairs and capturing k and v in one shot.

          Show
          jira.shegalov Gera Shegalov added a comment - You are right zhihai xu , and I actually meant to combine matching k=v pairs and capturing k and v in one shot.
          Hide
          jira.shegalov Gera Shegalov added a comment -

          002 attached, with this idea and proper name validation.

          Show
          jira.shegalov Gera Shegalov added a comment - 002 attached, with this idea and proper name validation.
          Hide
          hadoopqa Hadoop QA added a comment -



          +1 overall



          Vote Subsystem Runtime Comment
          0 pre-patch 17m 45s Pre-patch trunk compilation is healthy.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 tests included 0m 0s The patch appears to include 1 new or modified test files.
          +1 javac 7m 33s There were no new javac warning messages.
          +1 javadoc 9m 37s There were no new javadoc warning messages.
          +1 release audit 0m 23s The applied patch does not increase the total number of release audit warnings.
          +1 checkstyle 1m 47s There were no new checkstyle issues.
          +1 whitespace 0m 0s The patch has no lines that end in whitespace.
          +1 install 1m 33s mvn install still works.
          +1 eclipse:eclipse 0m 33s The patch built with eclipse:eclipse.
          +1 findbugs 3m 25s The patch does not introduce any new Findbugs (version 3.0.0) warnings.
          +1 common tests 22m 1s Tests passed in hadoop-common.
          +1 yarn tests 1m 56s Tests passed in hadoop-yarn-common.
              66m 36s  



          Subsystem Report/Notes
          Patch URL http://issues.apache.org/jira/secure/attachment/12742353/YARN-3768.002.patch
          Optional Tests javadoc javac unit findbugs checkstyle
          git revision trunk / 79ed0f9
          hadoop-common test log https://builds.apache.org/job/PreCommit-YARN-Build/8364/artifact/patchprocess/testrun_hadoop-common.txt
          hadoop-yarn-common test log https://builds.apache.org/job/PreCommit-YARN-Build/8364/artifact/patchprocess/testrun_hadoop-yarn-common.txt
          Test Results https://builds.apache.org/job/PreCommit-YARN-Build/8364/testReport/
          Java 1.7.0_55
          uname Linux asf905.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Console output https://builds.apache.org/job/PreCommit-YARN-Build/8364/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - +1 overall Vote Subsystem Runtime Comment 0 pre-patch 17m 45s Pre-patch trunk compilation is healthy. +1 @author 0m 0s The patch does not contain any @author tags. +1 tests included 0m 0s The patch appears to include 1 new or modified test files. +1 javac 7m 33s There were no new javac warning messages. +1 javadoc 9m 37s There were no new javadoc warning messages. +1 release audit 0m 23s The applied patch does not increase the total number of release audit warnings. +1 checkstyle 1m 47s There were no new checkstyle issues. +1 whitespace 0m 0s The patch has no lines that end in whitespace. +1 install 1m 33s mvn install still works. +1 eclipse:eclipse 0m 33s The patch built with eclipse:eclipse. +1 findbugs 3m 25s The patch does not introduce any new Findbugs (version 3.0.0) warnings. +1 common tests 22m 1s Tests passed in hadoop-common. +1 yarn tests 1m 56s Tests passed in hadoop-yarn-common.     66m 36s   Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12742353/YARN-3768.002.patch Optional Tests javadoc javac unit findbugs checkstyle git revision trunk / 79ed0f9 hadoop-common test log https://builds.apache.org/job/PreCommit-YARN-Build/8364/artifact/patchprocess/testrun_hadoop-common.txt hadoop-yarn-common test log https://builds.apache.org/job/PreCommit-YARN-Build/8364/artifact/patchprocess/testrun_hadoop-yarn-common.txt Test Results https://builds.apache.org/job/PreCommit-YARN-Build/8364/testReport/ Java 1.7.0_55 uname Linux asf905.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Console output https://builds.apache.org/job/PreCommit-YARN-Build/8364/console This message was automatically generated.
          Hide
          zxu zhihai xu added a comment -

          Hi Gera Shegalov, thanks for the new patch. I noticed the patch will accept "a=b=c" instead of discarding it. If the input is "a=b=c", it saves Env variable "a" with value "b". Is it correct? I also noticed the patch will discard Env Variable with empty string value. I am ok with it, but I just want to make sure we don't support Env Variable with empty string value.

          Show
          zxu zhihai xu added a comment - Hi Gera Shegalov , thanks for the new patch. I noticed the patch will accept "a=b=c" instead of discarding it. If the input is "a=b=c", it saves Env variable "a" with value "b". Is it correct? I also noticed the patch will discard Env Variable with empty string value. I am ok with it, but I just want to make sure we don't support Env Variable with empty string value.
          Hide
          jira.shegalov Gera Shegalov added a comment -

          Thanks for reviewing the patch, zhihai xu!

          If the input is "a=b=c", it saves Env variable "a" with value "b". Is it correct?

          Correct, and I agree it does not look the behavior we want. I think the right behavior is to accept any value between the first = and the next ,. The value should be b=c in your example.

          I also noticed the patch will discard Env Variable with empty string value. I am ok with it.

          I think it might be desirable sometimes to clear a variable that is is set globally. Let us allow it.

          003 patch attached!

          Show
          jira.shegalov Gera Shegalov added a comment - Thanks for reviewing the patch, zhihai xu ! If the input is "a=b=c", it saves Env variable "a" with value "b". Is it correct? Correct, and I agree it does not look the behavior we want. I think the right behavior is to accept any value between the first = and the next , . The value should be b=c in your example. I also noticed the patch will discard Env Variable with empty string value. I am ok with it. I think it might be desirable sometimes to clear a variable that is is set globally. Let us allow it. 003 patch attached!
          Hide
          hadoopqa Hadoop QA added a comment -



          +1 overall



          Vote Subsystem Runtime Comment
          0 pre-patch 17m 48s Pre-patch trunk compilation is healthy.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 tests included 0m 0s The patch appears to include 1 new or modified test files.
          +1 javac 7m 33s There were no new javac warning messages.
          +1 javadoc 9m 33s There were no new javadoc warning messages.
          +1 release audit 0m 22s The applied patch does not increase the total number of release audit warnings.
          +1 checkstyle 1m 49s There were no new checkstyle issues.
          +1 whitespace 0m 0s The patch has no lines that end in whitespace.
          +1 install 1m 36s mvn install still works.
          +1 eclipse:eclipse 0m 32s The patch built with eclipse:eclipse.
          +1 findbugs 3m 25s The patch does not introduce any new Findbugs (version 3.0.0) warnings.
          +1 common tests 21m 59s Tests passed in hadoop-common.
          +1 yarn tests 1m 57s Tests passed in hadoop-yarn-common.
              66m 38s  



          Subsystem Report/Notes
          Patch URL http://issues.apache.org/jira/secure/attachment/12742659/YARN-3768.003.patch
          Optional Tests javadoc javac unit findbugs checkstyle
          git revision trunk / d3797f9
          hadoop-common test log https://builds.apache.org/job/PreCommit-YARN-Build/8381/artifact/patchprocess/testrun_hadoop-common.txt
          hadoop-yarn-common test log https://builds.apache.org/job/PreCommit-YARN-Build/8381/artifact/patchprocess/testrun_hadoop-yarn-common.txt
          Test Results https://builds.apache.org/job/PreCommit-YARN-Build/8381/testReport/
          Java 1.7.0_55
          uname Linux asf906.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Console output https://builds.apache.org/job/PreCommit-YARN-Build/8381/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - +1 overall Vote Subsystem Runtime Comment 0 pre-patch 17m 48s Pre-patch trunk compilation is healthy. +1 @author 0m 0s The patch does not contain any @author tags. +1 tests included 0m 0s The patch appears to include 1 new or modified test files. +1 javac 7m 33s There were no new javac warning messages. +1 javadoc 9m 33s There were no new javadoc warning messages. +1 release audit 0m 22s The applied patch does not increase the total number of release audit warnings. +1 checkstyle 1m 49s There were no new checkstyle issues. +1 whitespace 0m 0s The patch has no lines that end in whitespace. +1 install 1m 36s mvn install still works. +1 eclipse:eclipse 0m 32s The patch built with eclipse:eclipse. +1 findbugs 3m 25s The patch does not introduce any new Findbugs (version 3.0.0) warnings. +1 common tests 21m 59s Tests passed in hadoop-common. +1 yarn tests 1m 57s Tests passed in hadoop-yarn-common.     66m 38s   Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12742659/YARN-3768.003.patch Optional Tests javadoc javac unit findbugs checkstyle git revision trunk / d3797f9 hadoop-common test log https://builds.apache.org/job/PreCommit-YARN-Build/8381/artifact/patchprocess/testrun_hadoop-common.txt hadoop-yarn-common test log https://builds.apache.org/job/PreCommit-YARN-Build/8381/artifact/patchprocess/testrun_hadoop-yarn-common.txt Test Results https://builds.apache.org/job/PreCommit-YARN-Build/8381/testReport/ Java 1.7.0_55 uname Linux asf906.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Console output https://builds.apache.org/job/PreCommit-YARN-Build/8381/console This message was automatically generated.
          Hide
          zxu zhihai xu added a comment -

          thanks for the new patch Gera Shegalov! I uploaded a patch YARN-3768.004.patch based on your patch YARN-3768.003.patch to support Env Variable with empty string value, since you think it might be desirable.
          it is a minor change: "([^,]+)" => "([^,]*)". I also updated the test to verify it.

          Show
          zxu zhihai xu added a comment - thanks for the new patch Gera Shegalov ! I uploaded a patch YARN-3768 .004.patch based on your patch YARN-3768 .003.patch to support Env Variable with empty string value, since you think it might be desirable. it is a minor change: "( [^,] +)" => "( [^,] *)" . I also updated the test to verify it.
          Hide
          jira.shegalov Gera Shegalov added a comment -

          Good catch, zhihai xu. I rushed on the way home and forgot to regenerate the patch with the * change after making it locally.

          +1 pending Jenkins

          Show
          jira.shegalov Gera Shegalov added a comment - Good catch, zhihai xu . I rushed on the way home and forgot to regenerate the patch with the * change after making it locally. +1 pending Jenkins
          Hide
          hadoopqa Hadoop QA added a comment -



          -1 overall



          Vote Subsystem Runtime Comment
          -1 pre-patch 17m 8s Findbugs (version 3.0.0) appears to be broken on trunk.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 tests included 0m 0s The patch appears to include 1 new or modified test files.
          +1 javac 8m 2s There were no new javac warning messages.
          +1 javadoc 10m 7s There were no new javadoc warning messages.
          +1 release audit 0m 21s The applied patch does not increase the total number of release audit warnings.
          -1 checkstyle 1m 23s The applied patch generated 97 new checkstyle issues (total was 0, now 97).
          +1 whitespace 0m 0s The patch has no lines that end in whitespace.
          +1 install 1m 34s mvn install still works.
          +1 eclipse:eclipse 0m 33s The patch built with eclipse:eclipse.
          +1 findbugs 3m 25s The patch does not introduce any new Findbugs (version 3.0.0) warnings.
          -1 common tests 21m 19s Tests failed in hadoop-common.
          -1 yarn tests 0m 19s Tests failed in hadoop-yarn-common.
              64m 29s  



          Reason Tests
          Failed unit tests hadoop.security.ssl.TestReloadingX509TrustManager
          Timed out tests org.apache.hadoop.security.token.delegation.TestZKDelegationTokenSecretManager
          Failed build hadoop-yarn-common



          Subsystem Report/Notes
          Patch URL http://issues.apache.org/jira/secure/attachment/12742690/YARN-3768.004.patch
          Optional Tests javadoc javac unit findbugs checkstyle
          git revision trunk / d3797f9
          checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/8383/artifact/patchprocess/diffcheckstylehadoop-common.txt
          hadoop-common test log https://builds.apache.org/job/PreCommit-YARN-Build/8383/artifact/patchprocess/testrun_hadoop-common.txt
          hadoop-yarn-common test log https://builds.apache.org/job/PreCommit-YARN-Build/8383/artifact/patchprocess/testrun_hadoop-yarn-common.txt
          Test Results https://builds.apache.org/job/PreCommit-YARN-Build/8383/testReport/
          Java 1.7.0_55
          uname Linux asf904.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Console output https://builds.apache.org/job/PreCommit-YARN-Build/8383/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment -1 pre-patch 17m 8s Findbugs (version 3.0.0) appears to be broken on trunk. +1 @author 0m 0s The patch does not contain any @author tags. +1 tests included 0m 0s The patch appears to include 1 new or modified test files. +1 javac 8m 2s There were no new javac warning messages. +1 javadoc 10m 7s There were no new javadoc warning messages. +1 release audit 0m 21s The applied patch does not increase the total number of release audit warnings. -1 checkstyle 1m 23s The applied patch generated 97 new checkstyle issues (total was 0, now 97). +1 whitespace 0m 0s The patch has no lines that end in whitespace. +1 install 1m 34s mvn install still works. +1 eclipse:eclipse 0m 33s The patch built with eclipse:eclipse. +1 findbugs 3m 25s The patch does not introduce any new Findbugs (version 3.0.0) warnings. -1 common tests 21m 19s Tests failed in hadoop-common. -1 yarn tests 0m 19s Tests failed in hadoop-yarn-common.     64m 29s   Reason Tests Failed unit tests hadoop.security.ssl.TestReloadingX509TrustManager Timed out tests org.apache.hadoop.security.token.delegation.TestZKDelegationTokenSecretManager Failed build hadoop-yarn-common Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12742690/YARN-3768.004.patch Optional Tests javadoc javac unit findbugs checkstyle git revision trunk / d3797f9 checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/8383/artifact/patchprocess/diffcheckstylehadoop-common.txt hadoop-common test log https://builds.apache.org/job/PreCommit-YARN-Build/8383/artifact/patchprocess/testrun_hadoop-common.txt hadoop-yarn-common test log https://builds.apache.org/job/PreCommit-YARN-Build/8383/artifact/patchprocess/testrun_hadoop-yarn-common.txt Test Results https://builds.apache.org/job/PreCommit-YARN-Build/8383/testReport/ Java 1.7.0_55 uname Linux asf904.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Console output https://builds.apache.org/job/PreCommit-YARN-Build/8383/console This message was automatically generated.
          Hide
          hadoopqa Hadoop QA added a comment -



          +1 overall



          Vote Subsystem Runtime Comment
          0 pre-patch 18m 12s Pre-patch trunk compilation is healthy.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 tests included 0m 0s The patch appears to include 1 new or modified test files.
          +1 javac 7m 37s There were no new javac warning messages.
          +1 javadoc 9m 38s There were no new javadoc warning messages.
          +1 release audit 0m 22s The applied patch does not increase the total number of release audit warnings.
          +1 checkstyle 1m 47s There were no new checkstyle issues.
          +1 whitespace 0m 0s The patch has no lines that end in whitespace.
          +1 install 1m 36s mvn install still works.
          +1 eclipse:eclipse 0m 31s The patch built with eclipse:eclipse.
          +1 findbugs 3m 27s The patch does not introduce any new Findbugs (version 3.0.0) warnings.
          +1 common tests 22m 36s Tests passed in hadoop-common.
          +1 yarn tests 1m 57s Tests passed in hadoop-yarn-common.
              67m 47s  



          Subsystem Report/Notes
          Patch URL http://issues.apache.org/jira/secure/attachment/12742721/YARN-3768.004.patch
          Optional Tests javadoc javac unit findbugs checkstyle
          git revision trunk / d3797f9
          hadoop-common test log https://builds.apache.org/job/PreCommit-YARN-Build/8384/artifact/patchprocess/testrun_hadoop-common.txt
          hadoop-yarn-common test log https://builds.apache.org/job/PreCommit-YARN-Build/8384/artifact/patchprocess/testrun_hadoop-yarn-common.txt
          Test Results https://builds.apache.org/job/PreCommit-YARN-Build/8384/testReport/
          Java 1.7.0_55
          uname Linux asf907.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Console output https://builds.apache.org/job/PreCommit-YARN-Build/8384/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - +1 overall Vote Subsystem Runtime Comment 0 pre-patch 18m 12s Pre-patch trunk compilation is healthy. +1 @author 0m 0s The patch does not contain any @author tags. +1 tests included 0m 0s The patch appears to include 1 new or modified test files. +1 javac 7m 37s There were no new javac warning messages. +1 javadoc 9m 38s There were no new javadoc warning messages. +1 release audit 0m 22s The applied patch does not increase the total number of release audit warnings. +1 checkstyle 1m 47s There were no new checkstyle issues. +1 whitespace 0m 0s The patch has no lines that end in whitespace. +1 install 1m 36s mvn install still works. +1 eclipse:eclipse 0m 31s The patch built with eclipse:eclipse. +1 findbugs 3m 27s The patch does not introduce any new Findbugs (version 3.0.0) warnings. +1 common tests 22m 36s Tests passed in hadoop-common. +1 yarn tests 1m 57s Tests passed in hadoop-yarn-common.     67m 47s   Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12742721/YARN-3768.004.patch Optional Tests javadoc javac unit findbugs checkstyle git revision trunk / d3797f9 hadoop-common test log https://builds.apache.org/job/PreCommit-YARN-Build/8384/artifact/patchprocess/testrun_hadoop-common.txt hadoop-yarn-common test log https://builds.apache.org/job/PreCommit-YARN-Build/8384/artifact/patchprocess/testrun_hadoop-yarn-common.txt Test Results https://builds.apache.org/job/PreCommit-YARN-Build/8384/testReport/ Java 1.7.0_55 uname Linux asf907.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Console output https://builds.apache.org/job/PreCommit-YARN-Build/8384/console This message was automatically generated.
          Hide
          jira.shegalov Gera Shegalov added a comment -

          Thanks zhihai xu for your contribution and Xuan Gong for additional review. Committed to trunk and branch-2 after cleaning unused imports in TestApps.

          Show
          jira.shegalov Gera Shegalov added a comment - Thanks zhihai xu for your contribution and Xuan Gong for additional review. Committed to trunk and branch-2 after cleaning unused imports in TestApps.
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-trunk-Commit #8097 (See https://builds.apache.org/job/Hadoop-trunk-Commit/8097/)
          YARN-3768. ArrayIndexOutOfBoundsException with empty environment variables. (Zhihai Xu via gera) (gera: rev 6f2a41e37d0b36cdafcfff75125165f212c612a6)

          • hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/Shell.java
          • hadoop-yarn-project/CHANGES.txt
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/test/java/org/apache/hadoop/yarn/util/TestApps.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/util/Apps.java
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-trunk-Commit #8097 (See https://builds.apache.org/job/Hadoop-trunk-Commit/8097/ ) YARN-3768 . ArrayIndexOutOfBoundsException with empty environment variables. (Zhihai Xu via gera) (gera: rev 6f2a41e37d0b36cdafcfff75125165f212c612a6) hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/Shell.java hadoop-yarn-project/CHANGES.txt hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/test/java/org/apache/hadoop/yarn/util/TestApps.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/util/Apps.java
          Hide
          zxu zhihai xu added a comment -

          thanks Xuan Gong for the review! thanks Gera Shegalov for the good idea, review and committing the patch.

          Show
          zxu zhihai xu added a comment - thanks Xuan Gong for the review! thanks Gera Shegalov for the good idea, review and committing the patch.
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Yarn-trunk #975 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/975/)
          YARN-3768. ArrayIndexOutOfBoundsException with empty environment variables. (Zhihai Xu via gera) (gera: rev 6f2a41e37d0b36cdafcfff75125165f212c612a6)

          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/test/java/org/apache/hadoop/yarn/util/TestApps.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/util/Apps.java
          • hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/Shell.java
          • hadoop-yarn-project/CHANGES.txt
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Yarn-trunk #975 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/975/ ) YARN-3768 . ArrayIndexOutOfBoundsException with empty environment variables. (Zhihai Xu via gera) (gera: rev 6f2a41e37d0b36cdafcfff75125165f212c612a6) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/test/java/org/apache/hadoop/yarn/util/TestApps.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/util/Apps.java hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/Shell.java hadoop-yarn-project/CHANGES.txt
          Hide
          hudson Hudson added a comment -

          SUCCESS: Integrated in Hadoop-Yarn-trunk-Java8 #245 (See https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/245/)
          YARN-3768. ArrayIndexOutOfBoundsException with empty environment variables. (Zhihai Xu via gera) (gera: rev 6f2a41e37d0b36cdafcfff75125165f212c612a6)

          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/test/java/org/apache/hadoop/yarn/util/TestApps.java
          • hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/Shell.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/util/Apps.java
          • hadoop-yarn-project/CHANGES.txt
          Show
          hudson Hudson added a comment - SUCCESS: Integrated in Hadoop-Yarn-trunk-Java8 #245 (See https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/245/ ) YARN-3768 . ArrayIndexOutOfBoundsException with empty environment variables. (Zhihai Xu via gera) (gera: rev 6f2a41e37d0b36cdafcfff75125165f212c612a6) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/test/java/org/apache/hadoop/yarn/util/TestApps.java hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/Shell.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/util/Apps.java hadoop-yarn-project/CHANGES.txt
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Hdfs-trunk #2172 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/2172/)
          YARN-3768. ArrayIndexOutOfBoundsException with empty environment variables. (Zhihai Xu via gera) (gera: rev 6f2a41e37d0b36cdafcfff75125165f212c612a6)

          • hadoop-yarn-project/CHANGES.txt
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/util/Apps.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/test/java/org/apache/hadoop/yarn/util/TestApps.java
          • hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/Shell.java
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Hdfs-trunk #2172 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/2172/ ) YARN-3768 . ArrayIndexOutOfBoundsException with empty environment variables. (Zhihai Xu via gera) (gera: rev 6f2a41e37d0b36cdafcfff75125165f212c612a6) hadoop-yarn-project/CHANGES.txt hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/util/Apps.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/test/java/org/apache/hadoop/yarn/util/TestApps.java hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/Shell.java
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Hdfs-trunk-Java8 #233 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/233/)
          YARN-3768. ArrayIndexOutOfBoundsException with empty environment variables. (Zhihai Xu via gera) (gera: rev 6f2a41e37d0b36cdafcfff75125165f212c612a6)

          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/util/Apps.java
          • hadoop-yarn-project/CHANGES.txt
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/test/java/org/apache/hadoop/yarn/util/TestApps.java
          • hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/Shell.java
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Hdfs-trunk-Java8 #233 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/233/ ) YARN-3768 . ArrayIndexOutOfBoundsException with empty environment variables. (Zhihai Xu via gera) (gera: rev 6f2a41e37d0b36cdafcfff75125165f212c612a6) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/util/Apps.java hadoop-yarn-project/CHANGES.txt hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/test/java/org/apache/hadoop/yarn/util/TestApps.java hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/Shell.java
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Mapreduce-trunk #2191 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2191/)
          YARN-3768. ArrayIndexOutOfBoundsException with empty environment variables. (Zhihai Xu via gera) (gera: rev 6f2a41e37d0b36cdafcfff75125165f212c612a6)

          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/test/java/org/apache/hadoop/yarn/util/TestApps.java
          • hadoop-yarn-project/CHANGES.txt
          • hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/Shell.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/util/Apps.java
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Mapreduce-trunk #2191 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2191/ ) YARN-3768 . ArrayIndexOutOfBoundsException with empty environment variables. (Zhihai Xu via gera) (gera: rev 6f2a41e37d0b36cdafcfff75125165f212c612a6) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/test/java/org/apache/hadoop/yarn/util/TestApps.java hadoop-yarn-project/CHANGES.txt hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/Shell.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/util/Apps.java
          Hide
          hudson Hudson added a comment -

          SUCCESS: Integrated in Hadoop-Mapreduce-trunk-Java8 #243 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/243/)
          YARN-3768. ArrayIndexOutOfBoundsException with empty environment variables. (Zhihai Xu via gera) (gera: rev 6f2a41e37d0b36cdafcfff75125165f212c612a6)

          • hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/Shell.java
          • hadoop-yarn-project/CHANGES.txt
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/test/java/org/apache/hadoop/yarn/util/TestApps.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/util/Apps.java
          Show
          hudson Hudson added a comment - SUCCESS: Integrated in Hadoop-Mapreduce-trunk-Java8 #243 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/243/ ) YARN-3768 . ArrayIndexOutOfBoundsException with empty environment variables. (Zhihai Xu via gera) (gera: rev 6f2a41e37d0b36cdafcfff75125165f212c612a6) hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/Shell.java hadoop-yarn-project/CHANGES.txt hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/test/java/org/apache/hadoop/yarn/util/TestApps.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/util/Apps.java
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user bellemare commented on the issue:

          https://github.com/apache/incubator-predictionio/issues/337

          Ah geez, I realize now that I may have made a mistake.
          ```
          "SPARK_YARN_USER_ENV" -> sys.env.filter(kv => kv.1.startsWith("PIO")).
          map(kv => s"$

          {kv._1}=${kv._2}").mkString(",")).run())

          Note the map: kv => s"${kv._1}

          =$

          {kv._2}

          "
          ```
          This indicates that it does INDEED create a key=value pair.... so I must have an empty parameter that I am setting, which is a known Yarn bug (https://issues.apache.org/jira/browse/YARN-3768). That being said, I could not find an empty parameter in my debug logging. Weird.

          I'll close this for now, since it does not seem to be the case that my reported issue is a bug.

          Show
          githubbot ASF GitHub Bot added a comment - Github user bellemare commented on the issue: https://github.com/apache/incubator-predictionio/issues/337 Ah geez, I realize now that I may have made a mistake. ``` "SPARK_YARN_USER_ENV" -> sys.env.filter(kv => kv. 1.startsWith("PIO ")). map(kv => s"$ {kv._1}=${kv._2}").mkString(",")).run()) Note the map: kv => s"${kv._1} =$ {kv._2} " ``` This indicates that it does INDEED create a key=value pair.... so I must have an empty parameter that I am setting, which is a known Yarn bug ( https://issues.apache.org/jira/browse/YARN-3768 ). That being said, I could not find an empty parameter in my debug logging. Weird. I'll close this for now, since it does not seem to be the case that my reported issue is a bug.

            People

            • Assignee:
              zxu zhihai xu
              Reporter:
              joeferner Joe Ferner
            • Votes:
              0 Vote for this issue
              Watchers:
              9 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development