Uploaded image for project: 'Livy'
  1. Livy
  2. LIVY-267

Livy yarn-cluster mode will fail on cluster with firewall as launcher.port uses random port

    XMLWordPrintableJSON

    Details

    • Type: Improvement
    • Status: Open
    • Priority: Major
    • Resolution: Unresolved
    • Affects Version/s: 0.3
    • Fix Version/s: None
    • Component/s: Core
    • Labels:
      None

      Description

      Livy Server when launches a Spark Job on yarn-cluster mode, RpcServer binds a random port and ContextLauncher sets that as livy.rsc.launcher.port. The RSCDriver which runs on some NodeManager Machine will try to connect to livy server at livy.rsc.launcher.port. In cluster where only fixed ports are open for security purposes, this job will fail. The livy.rsc.launcher.port port has to be a configurable one with a single or range of ports which RpcServer has to bind.

        Attachments

          Activity

            People

            • Assignee:
              jerryshao Saisai Shao
              Reporter:
              Prabhu Joseph Prabhu Joseph
            • Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

              • Created:
                Updated: