Uploaded image for project: 'Bigtop'
  1. Bigtop
  2. BIGTOP-2441

spark smoke test doesn't work relying on yarn-client submittion

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 1.0.0
    • Fix Version/s: 1.2.0
    • Component/s: tests
    • Labels:
      None

      Description

      We aren't using YARN to deploy services like Spark, as we relying on standalone cluster deployments. Hence, the test needs to be fixed.

      I wonder if anyone was even running this stuff ;(

        Issue Links

          Activity

          Hide
          cos Konstantin Boudnik added a comment -

          I have committed and pushed to the master.

          Show
          cos Konstantin Boudnik added a comment - I have committed and pushed to the master.
          Hide
          cos Konstantin Boudnik added a comment -

          Your comment makes all the sense. And let's totally see if this improvement needs to be done indeed. I just want to go ahead with this fix, so we have Spark smokes working again. Would you mind opening up a separate ticket to figure out how to do it? Thanks!

          Show
          cos Konstantin Boudnik added a comment - Your comment makes all the sense. And let's totally see if this improvement needs to be done indeed. I just want to go ahead with this fix, so we have Spark smokes working again. Would you mind opening up a separate ticket to figure out how to do it? Thanks!
          Hide
          ktsakalozos Konstantinos Tsakalozos added a comment -

          LGTM. Do you think we could be using the "spark.master" url from spark-defaults.conf . It might be beneficial to cross-check that as well especially in an HA setup.

          Show
          ktsakalozos Konstantinos Tsakalozos added a comment - LGTM. Do you think we could be using the "spark.master" url from spark-defaults.conf . It might be beneficial to cross-check that as well especially in an HA setup.
          Hide
          cos Konstantin Boudnik added a comment -

          Here's the proposed fix/improvement.

          Show
          cos Konstantin Boudnik added a comment - Here's the proposed fix/improvement.
          Hide
          cos Konstantin Boudnik added a comment - - edited

          The best approach I can think of right now is to check if SPARK_MASTER_IP and SPARK_MASTER_PORT are set. If so, use --master spark://$SPARK_MASTER_IP:SPARK_MASTER_PORT. Otherwise, --master yarn-client would do.

          Thoughts?

          Show
          cos Konstantin Boudnik added a comment - - edited The best approach I can think of right now is to check if SPARK_MASTER_IP and SPARK_MASTER_PORT are set. If so, use --master spark://$SPARK_MASTER_IP:SPARK_MASTER_PORT . Otherwise, --master yarn-client would do. Thoughts?

            People

            • Assignee:
              cos Konstantin Boudnik
              Reporter:
              cos Konstantin Boudnik
            • Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development