Details

    • Type: Sub-task
    • Status: Resolved
    • Priority: Blocker
    • Resolution: Fixed
    • Affects Version/s: 1.1.0
    • Fix Version/s: 1.1.0
    • Component/s: spark
    • Labels:
      None
    • Environment:

      bigtop-slaves:trunk-ubuntu-14.04

      Description

      As has been discussed here Spark master 1.5.1 ignores the setting of SPARK_MASTER_IP and binds to the IP address instead of the hostname. This makes workers being unable to associate with the master if they use hostname of the master host.

      1. BIGTOP-2113.1.patch
        1 kB
        Sergey Soldatov
      2. BIGTOP-2113.2.patch
        1 kB
        Sergey Soldatov

        Activity

        Hide
        sergey.soldatov Sergey Soldatov added a comment -

        Explanation is provided in SPARK-11570

        Show
        sergey.soldatov Sergey Soldatov added a comment - Explanation is provided in SPARK-11570
        Hide
        sergey.soldatov Sergey Soldatov added a comment -

        Mimics the logic of start-master.sh from spark distribution.

        Show
        sergey.soldatov Sergey Soldatov added a comment - Mimics the logic of start-master.sh from spark distribution.
        Hide
        cos Konstantin Boudnik added a comment -

        I am disagree with this part

        +    if [ "$SPARK_MASTER_IP" = "" ]; then
        +        SPARK_MASTER_IP=`hostname`
        +    fi
        

        as I think it'd more harm than good. Mimicking the faulty logic of the Spark script is really no excuse for that. But as Sergey Soldatov explained to me offline, the inner-logic Spark master has around the binding is pretty screwed-up, so perhaps this snippet won't be worst that the code itself.

        +1 I will commit it.

        Show
        cos Konstantin Boudnik added a comment - I am disagree with this part + if [ "$SPARK_MASTER_IP" = "" ]; then + SPARK_MASTER_IP=`hostname` + fi as I think it'd more harm than good. Mimicking the faulty logic of the Spark script is really no excuse for that. But as Sergey Soldatov explained to me offline, the inner-logic Spark master has around the binding is pretty screwed-up, so perhaps this snippet won't be worst that the code itself. +1 I will commit it.
        Hide
        cos Konstantin Boudnik added a comment -

        No, I won't - $SPARK_MASTER_IP needs to be passed to the master with --host

        Show
        cos Konstantin Boudnik added a comment - No, I won't - $SPARK_MASTER_IP needs to be passed to the master with --host
        Hide
        sergey.soldatov Sergey Soldatov added a comment -

        Added --ip

        Show
        sergey.soldatov Sergey Soldatov added a comment - Added --ip
        Hide
        cos Konstantin Boudnik added a comment -

        IIRC --ip is getting deprecated in Spark, but let's use it considering their scripts are using it for now ;(

        +1 committing

        Show
        cos Konstantin Boudnik added a comment - IIRC --ip is getting deprecated in Spark, but let's use it considering their scripts are using it for now ;( +1 committing
        Hide
        cos Konstantin Boudnik added a comment -

        Pushed to the master, thanks Sergey Soldatov

        Show
        cos Konstantin Boudnik added a comment - Pushed to the master, thanks Sergey Soldatov

          People

          • Assignee:
            sergey.soldatov Sergey Soldatov
            Reporter:
            cos Konstantin Boudnik
          • Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development