Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-5034

Spark on Yarn launch failure on HDInsight on Windows

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Cannot Reproduce
    • Affects Version/s: 1.1.0, 1.1.1, 1.2.0
    • Fix Version/s: None
    • Component/s: Windows, YARN
    • Labels:
      None
    • Environment:

      Spark on Yarn within HDInsight on Windows Azure

      Description

      Windows Environment

      I'm trying to run JavaSparkPi example on YARN with master = yarn-client but I have a problem.
      It runs smoothly with submitting application, first container for Application Master works too.
      When job is starting and there are some tasks to do I'm getting this warning on console (I'm using windows cmd if this makes any difference):
      WARN cluster.YarnClientClusterScheduler: Initial job has not accepted any resources; check your cluster UI to ensure that workers are registered and have sufficient memory
      When I'm checking logs for container with Application Masters it is launching containers for executors "properly", then goes with:
      INFO YarnAllocationHandler: Completed container container_1409217202587_0003_01_000002 (state: COMPLETE, exit status: 1)
      INFO YarnAllocationHandler: Container marked as failed: container_1409217202587_0003_01_000002
      And tries to re-launch them.

      On failed container log there is only this:
      Error: Could not find or load main class PWD..spark@GBV06758291.my.secret.address.net:63680.user.CoarseGrainedScheduler

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                Unassigned
                Reporter:
                riceyang Rice
              • Votes:
                0 Vote for this issue
                Watchers:
                4 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: