Uploaded image for project: 'Apache DataLab (Retired)'
  1. Apache DataLab (Retired)
  2. DATALAB-1823

[Branch-515][AWS][GCP][Zeppelin]: Playbooks flight/airbnb preparation Py2/Py3/R/Scala run with error for local/remote kernels

ResolvedReopenedClosedVerifiedBlockedon hold
    XMLWordPrintableJSON

Details

    Description

      Preconditions:

      1. Zeppelin is created on AWS/GCP

      Steps to reproduce:

      1. Run playbooks flight preparation Py2/Py3/R/Scala for local/remote kernels

      Actual result: 

      1. Playbooks run with error

      Expected result:

      1. Playbooks run successful

      ------------------------------------------------------------------------------------------

       For GCP Playbook preparation fails only for airbnb see attachments

      AWS

      Flight preparation Py2

      Local runs with error (04/06/2020)
      EMR 5.30.0 runs with error (09/06/2020)
      EMR 6.0.0 runs with error (09/06/2020)
      Spark runs with error (04/06/2020)

       Flight preparation R

      Local py2 runs with error (04/06/2020)
      Local py3 runs with error (04/06/2020)
      EMR 5.30.0 py2  verified (09/06/2020)
      EMR 5.30.0 py3  verified (09/06/2020)
      EMR 6.0.0 py2  runs with error (09/06/2020)
      EMR 6.0.0 py3  runs with error (09/06/2020)
      Spark py2  runs with error (04/06/2020)
      Spark py3  runs with error (04/06/2020

      Flight preparation scala

      Local py2 runs with error (04/06/2020)
      Local py3 runs with error (04/06/2020)
      EMR 5.30.0 py2  verified (09/06/2020)
      EMR 5.30.0 py3  verified (09/06/2020)
      EMR 6.0.0 py2  runs with error (09/06/2020)
      EMR 6.0.0 py3  runs with error (09/06/2020)
      Spark py2 runs with error (04/06/2020)
      Spark py3 runs with error (04/06/2020)

       AIRBNB

      Local py2 runs with error (04/06/2020)
      Local py3 runs with error (04/06/2020)
      EMR 5.30.0 py2 runs with error (09/06/2020)
      EMR 5.30.0 py3 runs with error (09/06/2020)
      EMR 6.0.0 py2 runs with error (09/06/2020) 
      EMR 6.0.0 py3 runs with error (09/06/2020
      Spark py2 runs with error (04/06/2020)
      Spark py3 runs with error (04/06/2020)

       GCP

      AIRBNB

      Local py2 runs with error  (11/06/2020)
      Local py3 runs with error -failed to start pyspark (11/06/2020)
      DP 1.3 py2 runs with error (11/06/2020)
      DP 1.3 py3 runs with error (11/06/2020)
      DP 1.4 py2 runs with error (11/06/2020) 
      DP 1.4 py3 runs with error (11/06/2020)
      DP 1.5 ubuntu py2 runs with error (16/06/2020)
      DP 1.5 ubuntu py3 runs with error (16/06/2020)
      Spark py2 runs with error  (11/06/2020)
      Spark py3 runs with error  -failed to start pyspark (11/06/2020)

      Flight preparation Py2

      Local verfied (11/06/2020)
      DP 1.3 runs with error (11/06/2020)
      DP 1.4 verified (11/06/2020)
      DP1.5 ubuntu runs with error (16/06/2020)
      Spark verified (11/06/2020)

       Flight preparation R

      Local py2 verified (11/06/2020)
      Local py3 verified (11/06/2020)
      DP 1.3 py2 runs with error (11/06/2020)
      DP 1.3 py3 runs with error (11/06/2020)
      DP 1.4 py2 verified (11/06/2020)
      DP 1.4 py3 verified (11/06/2020)
      DP 1.5 ubuntu py2 runs with error (16/06/2020)
      DP 1.5 ubuntu py3 runs with error ** (16/06/2020)
      Spark py2  verified (11/06/2020)
      Spark py3 verified (11/06/2020)

      Flight preparation scala

      Local py2 verified (11/06/2020)
      Local py3 verified (11/06/2020)
      DP 1.3 py2  runs with error (11/06/2020)
      DP 1.3 py3 runs with error (11/06/2020)
      DP 1.4 py2 verified (11/06/2020)
      DP 1.4 py3 verified (11/06/2020)
      DP 1.5 ubuntu py2  runs with error (16/06/2020)
      DP 1.5 ubuntu py3 runs with error (16/06/2020)
      Spark py2 verified (11/06/2020)
      Spark py3 verified (11/06/2020)

      Attachments

        1. Python 2 data preparation-local.png
          449 kB
          Vira Vitanska
        2. Python 2 data preparation -local logs.txt
          9 kB
          Vira Vitanska
        3. Python 2 data preparation-spark.png
          444 kB
          Vira Vitanska
        4. Python 2 data preparation-Spark-logs.txt
          9 kB
          Vira Vitanska
        5. R data preparation local remote logs.txt
          9 kB
          Vira Vitanska
        6. Scala -py2-py3- local-remote.png
          402 kB
          Vira Vitanska
        7. Scala py2-py3 remote-local logs.txt
          5 kB
          Vira Vitanska
        8. AIRBNB local-remote.png
          398 kB
          Vira Vitanska
        9. GCP Preparation airbnb py2.png
          118 kB
          Vira Vitanska
        10. GCP preparation airbnb py3.png
          42 kB
          Vira Vitanska
        11. Verified1-AWS-local-flight-prapare-Py2.png
          461 kB
          Vira Vitanska
        12. Verified-AWS-Local-Fight-R-Py3.png
          464 kB
          Vira Vitanska
        13. Verified1-AWS-Scala-Flight-Prepare-Py3.png
          435 kB
          Vira Vitanska
        14. Verified1-AWS-Scala-Flight-Prepare-Py2.png
          421 kB
          Vira Vitanska
        15. Verified1-AWS-Spark-Flight-Prapare-Py2.png
          477 kB
          Vira Vitanska
        16. Verified1-AWS-Spark-Flight-Prepare-Py2.png
          398 kB
          Vira Vitanska
        17. Verified1-AWS-spark-flight-prepare-Scala-Py2.png
          432 kB
          Vira Vitanska
        18. Verified1-AWS-Spark-Flight-Prepare-Py3.png
          395 kB
          Vira Vitanska
        19. Verified1-AWS-Spark-Flight-Prepare-R-py3.png
          463 kB
          Vira Vitanska
        20. Verifight-AWS-Spark-AIRBNB-Prepare-Py2.png
          425 kB
          Vira Vitanska
        21. Verified1-AWS-Spark-AIRBNB-Prepare-Py3.png
          291 kB
          Vira Vitanska
        22. Verified1-AWS-EMR530-Flight-preparetion-PY2.png
          413 kB
          Vira Vitanska
        23. Verified1-AWS-EMR6-Flight-Prepare-py2.png
          487 kB
          Vira Vitanska
        24. Verified1-AWS-EMR6-Flight-R-Py2.png
          476 kB
          Vira Vitanska
        25. Verified1-AWS-EMR6-Flight-R-Py3.png
          457 kB
          Vira Vitanska
        26. Verified1-AWS-EMR6-Flight-Scala-Py2.png
          360 kB
          Vira Vitanska
        27. Verified1-AWS-EMR6-Flight-Scala-Py3.png
          367 kB
          Vira Vitanska
        28. Verified1-AWS-EMR530-AIRBNB-Prapare-Py3.png
          510 kB
          Vira Vitanska
        29. Verified1-AWS-EMR530-AIRBNB-Prapare-Py2.png
          490 kB
          Vira Vitanska
        30. Verified1-AWS-EMR6-AIRBNB-Prapare-Py2-py3.png
          382 kB
          Vira Vitanska
        31. Verified1-GCP-local-AIRBNB-py2.jpg
          381 kB
          Vira Vitanska
        32. Verified1-GCP-local-AirbNB-prepare-py3.jpg
          128 kB
          Vira Vitanska
        33. verified1-GCP-Spark-AIRBNB-Prapare-Py3.jpg
          92 kB
          Vira Vitanska
        34. Verified1-GCP-DP13-Flight-prepare-Py2.jpg
          225 kB
          Vira Vitanska
        35. Verified1-GCP-DP13-Flight-visualize-Py2.jpg
          241 kB
          Vira Vitanska
        36. verified1-GCP-DP13-Flight-prepare-R-py2.jpg
          168 kB
          Vira Vitanska
        37. Verified-GCP-DP15-Flight-Prepare-Py2.png
          490 kB
          Vira Vitanska
        38. image-2020-06-16-10-20-51-231.png
          431 kB
          Vira Vitanska
        39. Verified1-GCP-DP15-Flight-prepare-py2-R.png
          423 kB
          Vira Vitanska
        40. Verified1-GCP-DP15-Flight0Prepare-Scala.png
          337 kB
          Vira Vitanska
        41. Verified1-GCP-DP15-AIRBNB-Prepare-py2.png
          337 kB
          Vira Vitanska

        Issue Links

          is related to

          Task - A task that needs to be done. DATALAB-1857 [Branch-515][AWS][GCP][Zeppelin]: Investigate Spark cluster Py3 for correct work

          • Major - Major loss of function.
          • Closed

          Activity

            People

              mykolabodnar Mykola Bodnar
              vvitanska Vira Vitanska
              Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: