Uploaded image for project: 'Livy'
  1. Livy
  2. LIVY-451

Failed to use Livy in Docker container launched on same node as YARN

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Open
    • Major
    • Resolution: Unresolved
    • 0.3, 0.5.0
    • 0.9.0
    • RSC
    • None

    Description

      I've tried to run Livy inside of Docker container on same node as I have YARN.

      As I've found, to make it works I need to expose some ports (LIVY-337), so I've added -p 10000-10010:10000-10010 option to Docker run command.

      It works when Livy container launched on other machine, but when it launched on same node as YARN, Spark Driver fails to start.

      After some investigation, I've found that Spark Driver (which is inside of YARN) tries to use same ports that configured for Spark Launcher (livy.rsc.launcher.port.range option), and it was not able to start as all ports from that range was already bound by Docker daemon.

      Attachments

        Activity

          People

            Unassigned Unassigned
            idzikovsky Ivan Dzikovsky
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

              Created:
              Updated: