Description
There are two issues w/ Spark action's argument parsing within SparkMain
Driver and executor extra classpaths: equals sign used
When the user specifies --conf spark.executor.extraClassPath=XYZ or --conf spark.driver.extraClassPath=ABC, the option --conf will be added to sparkArgs. Then when the code tries to evaluate spark.executor.extraClassPath=XYZ, it uses special logic and set addToSparkArgs = false. As a result there will be a extra --conf in the sparkArgs eventually.
For example: --conf spark.executor.extraClassPath=XYZ --conf otherProperty=ABC will become --conf --conf otherProperty=ABC, which will cause spark job submit failure later.
We might need to remove one prior --conf in sparkArgs if the current evaluated opt is EXECUTOR_CLASSPATH or DRIVER_CLASSPATH.
User provided files and archives: equals sign used
For the following workflow XML snippet:
<spark-opts>--files=${nameNode}/home/share/hive-site.xml --num-executors 4 --executor-memory 7g --driver-memory 7g</spark-opts>
the --files=${nameNode}/home/share/hive-site.xml opt will be placed into sparkArgs in previous Oozie version without any modification, because we don't have special handling for --files opt.
If the user specifies --files=${nameNode}/home/share/hive-site.xml --num-executor 4, then SparkMain code treats --num-executor as a file path / name. That caused the issue as I described in my previous comment. We might need to change the handling logic for FILES_OPTION and ARCHIVES_OPTION to be the same to DRIVER_CLASSPATH_OPTION.
Attachments
Attachments
- OOZIE-2923.001.patch
- 60 kB
- Andras Piros
- OOZIE-2923.002.patch
- 60 kB
- Andras Piros
- OOZIE-2923.003.patch
- 72 kB
- Andras Piros
- OOZIE-2923.004.patch
- 72 kB
- Andras Piros
- OOZIE-2923.005.patch
- 72 kB
- Andras Piros
- OOZIE-2923.006.patch
- 72 kB
- Andras Piros
Issue Links
- is related to
-
OOZIE-3228 [Spark action] Can't load properties from spark-defaults.conf
- Closed
- links to
Activity
Testing JIRA OOZIE-2923
Cleaning local git workspace
----------------------------
+1 PATCH_APPLIES
+1 CLEAN
+1 RAW_PATCH_ANALYSIS
. +1 the patch does not introduce any @author tags
. +1 the patch does not introduce any tabs
. +1 the patch does not introduce any trailing spaces
. +1 the patch does not introduce any line longer than 132
. +1 the patch does adds/modifies 3 testcase(s)
+1 RAT
. +1 the patch does not seem to introduce new RAT warnings
+1 JAVADOC
. +1 the patch does not seem to introduce new Javadoc warnings
. WARNING: the current HEAD has 6 Javadoc warning(s)
+1 COMPILE
. +1 HEAD compiles
. +1 patch compiles
. +1 the patch does not seem to introduce new javac warnings
+1 There are no new bugs found in total.
. +1 There are no new bugs found in [server].
. +1 There are no new bugs found in [client].
. +1 There are no new bugs found in [docs].
. +1 There are no new bugs found in [sharelib/hive].
. +1 There are no new bugs found in [sharelib/spark].
. +1 There are no new bugs found in [sharelib/hcatalog].
. +1 There are no new bugs found in [sharelib/hive2].
. +1 There are no new bugs found in [sharelib/streaming].
. +1 There are no new bugs found in [sharelib/pig].
. +1 There are no new bugs found in [sharelib/sqoop].
. +1 There are no new bugs found in [sharelib/distcp].
. +1 There are no new bugs found in [sharelib/oozie].
. +1 There are no new bugs found in [core].
. +1 There are no new bugs found in [tools].
. +1 There are no new bugs found in [examples].
+1 BACKWARDS_COMPATIBILITY
. +1 the patch does not change any JPA Entity/Colum/Basic/Lob/Transient annotations
. +1 the patch does not modify JPA files
-1 TESTS
. Tests run: 1961
. Tests failed: 1
. Tests errors: 2
. The patch failed the following testcases:
. testNofindLogs(org.apache.oozie.util.TestTimestampedMessageParser)
. Tests failing with errors:
. testProcessRemainingLog(org.apache.oozie.util.TestTimestampedMessageParser)
. testProcessRemainingCoordinatorLogForActions(org.apache.oozie.util.TestTimestampedMessageParser)
+1 DISTRO
. +1 distro tarball builds with the patch
----------------------------
-1 Overall result, please check the reported -1(s)
. There is at least one warning, please check
The full output of the test-patch run is available at
. https://builds.apache.org/job/oozie-trunk-precommit-build/3841/
Thanks for the contribution!
I like the idea of extracting things from SparkMain to make things more testable. Left some comments on RB
Testing JIRA OOZIE-2923
Cleaning local git workspace
----------------------------
+1 PATCH_APPLIES
+1 CLEAN
+1 RAW_PATCH_ANALYSIS
. +1 the patch does not introduce any @author tags
. +1 the patch does not introduce any tabs
. +1 the patch does not introduce any trailing spaces
. +1 the patch does not introduce any line longer than 132
. +1 the patch does adds/modifies 4 testcase(s)
+1 RAT
. +1 the patch does not seem to introduce new RAT warnings
+1 JAVADOC
. +1 the patch does not seem to introduce new Javadoc warnings
. WARNING: the current HEAD has 6 Javadoc warning(s)
+1 COMPILE
. +1 HEAD compiles
. +1 patch compiles
. +1 the patch does not seem to introduce new javac warnings
-1 There are [1] new bugs found below threshold in total that must be fixed.
. +1 There are no new bugs found in [server].
. +1 There are no new bugs found in [client].
. +1 There are no new bugs found in [docs].
. +1 There are no new bugs found in [sharelib/hive].
. -1 There are [1] new bugs found below threshold in [sharelib/spark] that must be fixed.
. You can find the FindBugs diff here (look for the red and orange ones): sharelib/spark/findbugs-new.html
. The most important FindBugs errors are:
. At HadoopUriFinder.java:[line 34]: org.apache.oozie.action.hadoop.HadoopUriFinder.getJarVersion(File) may fail to close stream
. +1 There are no new bugs found in [sharelib/hcatalog].
. +1 There are no new bugs found in [sharelib/hive2].
. +1 There are no new bugs found in [sharelib/streaming].
. +1 There are no new bugs found in [sharelib/pig].
. +1 There are no new bugs found in [sharelib/sqoop].
. +1 There are no new bugs found in [sharelib/distcp].
. +1 There are no new bugs found in [sharelib/oozie].
. +1 There are no new bugs found in [core].
. +1 There are no new bugs found in [tools].
. +1 There are no new bugs found in [examples].
+1 BACKWARDS_COMPATIBILITY
. +1 the patch does not change any JPA Entity/Colum/Basic/Lob/Transient annotations
. +1 the patch does not modify JPA files
-1 TESTS
. Tests run: 1963
. Tests failed: 1
. Tests errors: 2
. The patch failed the following testcases:
. testNofindLogs(org.apache.oozie.util.TestTimestampedMessageParser)
. Tests failing with errors:
. testProcessRemainingLog(org.apache.oozie.util.TestTimestampedMessageParser)
. testProcessRemainingCoordinatorLogForActions(org.apache.oozie.util.TestTimestampedMessageParser)
+1 DISTRO
. +1 distro tarball builds with the patch
----------------------------
-1 Overall result, please check the reported -1(s)
. There is at least one warning, please check
The full output of the test-patch run is available at
. https://builds.apache.org/job/oozie-trunk-precommit-build/3844/
Testing JIRA OOZIE-2923
Cleaning local git workspace
----------------------------
+1 PATCH_APPLIES
+1 CLEAN
+1 RAW_PATCH_ANALYSIS
. +1 the patch does not introduce any @author tags
. +1 the patch does not introduce any tabs
. +1 the patch does not introduce any trailing spaces
. +1 the patch does not introduce any line longer than 132
. +1 the patch does adds/modifies 4 testcase(s)
+1 RAT
. +1 the patch does not seem to introduce new RAT warnings
+1 JAVADOC
. +1 the patch does not seem to introduce new Javadoc warnings
. WARNING: the current HEAD has 6 Javadoc warning(s)
+1 COMPILE
. +1 HEAD compiles
. +1 patch compiles
. +1 the patch does not seem to introduce new javac warnings
+1 There are no new bugs found in total.
. +1 There are no new bugs found in [server].
. +1 There are no new bugs found in [client].
. +1 There are no new bugs found in [core].
. +1 There are no new bugs found in [docs].
. +1 There are no new bugs found in [tools].
. +1 There are no new bugs found in [examples].
. +1 There are no new bugs found in [sharelib/streaming].
. +1 There are no new bugs found in [sharelib/sqoop].
. +1 There are no new bugs found in [sharelib/distcp].
. +1 There are no new bugs found in [sharelib/oozie].
. +1 There are no new bugs found in [sharelib/hcatalog].
. +1 There are no new bugs found in [sharelib/hive].
. +1 There are no new bugs found in [sharelib/hive2].
. +1 There are no new bugs found in [sharelib/pig].
. +1 There are no new bugs found in [sharelib/spark].
+1 BACKWARDS_COMPATIBILITY
. +1 the patch does not change any JPA Entity/Colum/Basic/Lob/Transient annotations
. +1 the patch does not modify JPA files
+1 TESTS
. Tests run: 1963
. Tests rerun: 79
. Tests failed at first run: org.apache.oozie.action.hadoop.TestJavaActionExecutor,org.apache.oozie.jms.TestJMSJobEventListener,org.apache.oozie.action.hadoop.TestLauncherAM,
+1 DISTRO
. +1 distro tarball builds with the patch
----------------------------
+1 Overall result, good!, no -1s
. There is at least one warning, please check
The full output of the test-patch run is available at
. https://builds.apache.org/job/oozie-trunk-precommit-build/3848/
Testing JIRA OOZIE-2923
Cleaning local git workspace
----------------------------
+1 PATCH_APPLIES
+1 CLEAN
+1 RAW_PATCH_ANALYSIS
. +1 the patch does not introduce any @author tags
. +1 the patch does not introduce any tabs
. +1 the patch does not introduce any trailing spaces
. +1 the patch does not introduce any line longer than 132
. +1 the patch does adds/modifies 4 testcase(s)
+1 RAT
. +1 the patch does not seem to introduce new RAT warnings
+1 JAVADOC
. +1 the patch does not seem to introduce new Javadoc warnings
. WARNING: the current HEAD has 6 Javadoc warning(s)
+1 COMPILE
. +1 HEAD compiles
. +1 patch compiles
. +1 the patch does not seem to introduce new javac warnings
+1 There are no new bugs found in total.
. +1 There are no new bugs found in [server].
. +1 There are no new bugs found in [client].
. +1 There are no new bugs found in [core].
. +1 There are no new bugs found in [docs].
. +1 There are no new bugs found in [tools].
. +1 There are no new bugs found in [examples].
. +1 There are no new bugs found in [sharelib/streaming].
. +1 There are no new bugs found in [sharelib/sqoop].
. +1 There are no new bugs found in [sharelib/distcp].
. +1 There are no new bugs found in [sharelib/oozie].
. +1 There are no new bugs found in [sharelib/hcatalog].
. +1 There are no new bugs found in [sharelib/hive].
. +1 There are no new bugs found in [sharelib/hive2].
. +1 There are no new bugs found in [sharelib/pig].
. +1 There are no new bugs found in [sharelib/spark].
+1 BACKWARDS_COMPATIBILITY
. +1 the patch does not change any JPA Entity/Colum/Basic/Lob/Transient annotations
. +1 the patch does not modify JPA files
+1 TESTS
. Tests run: 1963
. Tests rerun: 79
. Tests failed at first run: org.apache.oozie.action.hadoop.TestJavaActionExecutor,org.apache.oozie.jms.TestJMSJobEventListener,org.apache.oozie.action.hadoop.TestLauncherAM,
+1 DISTRO
. +1 distro tarball builds with the patch
----------------------------
+1 Overall result, good!, no -1s
. There is at least one warning, please check
The full output of the test-patch run is available at
. https://builds.apache.org/job/oozie-trunk-precommit-build/3862/
Thanks andras.piros!
Some of the tests are failing for me because of some differences:
--files, hive-site.xml,spark-log4j.properties,
--files, spark-log4j.properties,hive-site.xml,
I think it may be a HashMap ordering thing that changes in Java8. Please don't depend on the order or fix the order in the Extractor.
Addressing fixed order of parameters using LinkedHashMap for JDK7 and JDK8.
Testing JIRA OOZIE-2923
Cleaning local git workspace
----------------------------
+1 PATCH_APPLIES
+1 CLEAN
+1 RAW_PATCH_ANALYSIS
. +1 the patch does not introduce any @author tags
. +1 the patch does not introduce any tabs
. +1 the patch does not introduce any trailing spaces
. +1 the patch does not introduce any line longer than 132
. +1 the patch does adds/modifies 4 testcase(s)
+1 RAT
. +1 the patch does not seem to introduce new RAT warnings
+1 JAVADOC
. +1 the patch does not seem to introduce new Javadoc warnings
. WARNING: the current HEAD has 6 Javadoc warning(s)
+1 COMPILE
. +1 HEAD compiles
. +1 patch compiles
. +1 the patch does not seem to introduce new javac warnings
+1 There are no new bugs found in total.
. +1 There are no new bugs found in [server].
. +1 There are no new bugs found in [client].
. +1 There are no new bugs found in [core].
. +1 There are no new bugs found in [docs].
. +1 There are no new bugs found in [tools].
. +1 There are no new bugs found in [examples].
. +1 There are no new bugs found in [sharelib/streaming].
. +1 There are no new bugs found in [sharelib/sqoop].
. +1 There are no new bugs found in [sharelib/distcp].
. +1 There are no new bugs found in [sharelib/oozie].
. +1 There are no new bugs found in [sharelib/hcatalog].
. +1 There are no new bugs found in [sharelib/hive].
. +1 There are no new bugs found in [sharelib/hive2].
. +1 There are no new bugs found in [sharelib/pig].
. +1 There are no new bugs found in [sharelib/spark].
+1 BACKWARDS_COMPATIBILITY
. +1 the patch does not change any JPA Entity/Colum/Basic/Lob/Transient annotations
. +1 the patch does not modify JPA files
+1 TESTS
. Tests run: 1963
. Tests rerun: 64
. Tests failed at first run: org.apache.oozie.action.hadoop.TestJavaActionExecutor,org.apache.oozie.action.hadoop.TestLauncherAM,
+1 DISTRO
. +1 distro tarball builds with the patch
----------------------------
+1 Overall result, good!, no -1s
. There is at least one warning, please check
The full output of the test-patch run is available at
. https://builds.apache.org/job/oozie-trunk-precommit-build/3866/
Testing JIRA
OOZIE-2923Cleaning local git workspace
----------------------------
+1 PATCH_APPLIES
+1 CLEAN
+1 RAW_PATCH_ANALYSIS
. +1 the patch does not introduce any @author tags
. +1 the patch does not introduce any tabs
. +1 the patch does not introduce any trailing spaces
. +1 the patch does not introduce any line longer than 132
. +1 the patch does adds/modifies 3 testcase(s)
+1 RAT
. +1 the patch does not seem to introduce new RAT warnings
+1 JAVADOC
. +1 the patch does not seem to introduce new Javadoc warnings
. WARNING: the current HEAD has 6 Javadoc warning(s)
+1 COMPILE
. +1 HEAD compiles
. +1 patch compiles
. +1 the patch does not seem to introduce new javac warnings
-1 There are [1] new bugs found below threshold in total that must be fixed.
. +1 There are no new bugs found in [sharelib/hive].
. +1 There are no new bugs found in [sharelib/hcatalog].
. +1 There are no new bugs found in [sharelib/sqoop].
. +1 There are no new bugs found in [sharelib/pig].
. +1 There are no new bugs found in [sharelib/streaming].
. -1 There are [1] new bugs found below threshold in [sharelib/spark] that must be fixed.
. You can find the FindBugs diff here (look for the red and orange ones): sharelib/spark/findbugs-new.html
. The most important FindBugs errors are:
. Dereferenced at SparkMain.java:[line 165]: Possible null pointer dereference in org.apache.oozie.action.hadoop.SparkMain.getMatchingFile(Pattern) due to return value of called method
. Known null at SparkMain.java:[line 165]
. +1 There are no new bugs found in [sharelib/oozie].
. +1 There are no new bugs found in [sharelib/distcp].
. +1 There are no new bugs found in [sharelib/hive2].
. +1 There are no new bugs found in [server].
. +1 There are no new bugs found in [core].
. +1 There are no new bugs found in [tools].
. +1 There are no new bugs found in [docs].
. +1 There are no new bugs found in [client].
. +1 There are no new bugs found in [examples].
+1 BACKWARDS_COMPATIBILITY
. +1 the patch does not change any JPA Entity/Colum/Basic/Lob/Transient annotations
. +1 the patch does not modify JPA files
+1 TESTS
. Tests run: 1961
. Tests rerun: 39
. Tests failed at first run: org.apache.oozie.action.hadoop.TestLauncherAM,org.apache.oozie.action.hadoop.TestHdfsOperations,org.apache.oozie.event.TestEventGeneration,org.apache.oozie.command.coord.TestAbandonedCoordChecker,
+1 DISTRO
. +1 distro tarball builds with the patch
----------------------------
-1 Overall result, please check the reported -1(s)
. There is at least one warning, please check
The full output of the test-patch run is available at
. https://builds.apache.org/job/oozie-trunk-precommit-build/3839/