Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-5402

Log executor ID at executor-construction time

Attach filesAttach ScreenshotVotersWatch issueWatchersCreate sub-taskLinkCloneUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

Details

    • Improvement
    • Status: Closed
    • Minor
    • Resolution: Fixed
    • 1.2.0
    • 1.2.1, 1.3.0
    • Spark Core
    • None

    Description

      One stumbling block I've hit while debugging Spark-on-YARN jobs is that yarn logs presents each executor's stderr/stdout by container name, but I often need to find the logs for a specific executor ID; the executor ID isn't printed anywhere convenient in each executor's logs, afaict.

      I added a simple logInfo to Executor.scala locally and it's been useful, so I'd like to merge it upstream.

      PR forthcoming.

      Attachments

        Activity

          This comment will be Viewable by All Users Viewable by All Users
          Cancel

          People

            rdub Ryan Williams
            rdub Ryan Williams
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Slack

                Issue deployment