[SPARK-48217] Spark stdout and stderr getting removed at end of spark job triggered from cloudera hue workflow

• Type: Improvement
• Status: Open
• Priority: Critical
• Resolution: Unresolved
• Affects Version/s: 1.6.0
• Component/s: Spark Submit
• Assignee: Unassigned Assign to me
• Reporter: Noam Shemesh
• Created: 09/May/24 10:04
• Updated: 12/May/24 06:08

Description

Hello,

we are running spark job triggered from cloudera hue workflow

and spark printing stdout and stderr logs during execution as expected:

e.g. -

 

But stdout and stderr logs getting cleaned when workflows finished/status succeeded

 

following is spark-submit command workflow is triggering:

_/usr/bin/spark-submit _
  _--master yarn-client _
  _--driver-memory 4g _
  _--executor-memory 16g _
  _--executor-cores 4 _
  _--class tst _
  _--files `ls -m *.conf | tr -d '\n '` _
  _--conf "spark.dynamicAllocation.maxExecutors=4" _
  _--conf "spark.kryoserializer.buffer.max=1024" _
  tst.jar $*

 

 

does someone familiar with this spark job behavior or can advise ideas to fix it?

 

Thanks in advance