Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-5925

YARN - Spark progress bar stucks at 10% but after finishing shows 100%

    XMLWordPrintableJSON

    Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Minor
    • Resolution: Won't Fix
    • Affects Version/s: 1.2.1
    • Fix Version/s: None
    • Component/s: YARN
    • Labels:
      None

      Description

      I did set up a yarn cluster (CDH5) and spark (1.2.1), and also started Spark History Server. Now I am able to click on more details on yarn's web interface and get redirected to the appropriate spark logs during both job execution and also after the job has finished.

      My only concern is that while a spark job is being executed (either yarn-client or yarn-cluster), the progress bar stucks at 10% and doesn't increase as for MapReduce jobs. After finishing, it shows 100% properly, but we are loosing the real-time tracking capability of the status bar.

      Also tested yarn restful web interface, and it retrieves again 10% during (yarn) spark job execution, and works well again after finishing. (I suppose for the while being I should have a look on Spark Job Server and see if it's possible to track the job via its restful web interface.)

      Did anyone else experience this behaviour? Thanks in advance.

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                Unassigned
                Reporter:
                funnyboy Laszlo Fesus
              • Votes:
                5 Vote for this issue
                Watchers:
                10 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: