[SPARK-33668] Fix flaky test "Verify logging configuration is picked from the provided SPARK_CONF_DIR/log4j.properties."

• Type: Sub-task
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 3.1.0
• Fix Version/s: 3.1.0
• Component/s: Kubernetes, Tests
• Assignee: Prashant Sharma
• Reporter: Prashant Sharma
• Created: 05/Dec/20 07:23
• Updated: 15/Jan/21 21:15
• Resolved: 06/Dec/20 07:05

Description

The test is flaking, with multiple flaked instances - the reason for the failure has been similar to:

  The code passed to eventually never returned normally. Attempted 109 times over 3.0079882413999997 minutes. Last failure message: Failure executing: GET at: https://192.168.39.167:8443/api/v1/namespaces/b37fc72a991b49baa68a2eaaa1516463/pods/spark-pi-97a9bc76308e7fe3-exec-1/log?pretty=false. Message: pods "spark-pi-97a9bc76308e7fe3-exec-1" not found. Received status: Status(apiVersion=v1, code=404, details=StatusDetails(causes=[], group=null, kind=pods, name=spark-pi-97a9bc76308e7fe3-exec-1, retryAfterSeconds=null, uid=null, additionalProperties={}), kind=Status, message=pods "spark-pi-97a9bc76308e7fe3-exec-1" not found, metadata=ListMeta(_continue=null, remainingItemCount=null, resourceVersion=null, selfLink=null, additionalProperties={}), reason=NotFound, status=Failure, additionalProperties={}).. (KubernetesSuite.scala:402)

https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/36854/console
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/36852/console
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/36850/console
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/36848/console

From the above failures, it seems, that executor finishes too quickly and is removed by spark before the test can complete.

So, in order to mitigate this situation, one way is to turn on the flag

   "spark.kubernetes.executor.deleteOnTermination"

Issue Links

• links to: [Github] Pull Request #30616 (ScrapCodes)
• links to: [Github] Pull Request #30616 (ScrapCodes)