Uploaded image for project: 'Bigtop'
  1. Bigtop
  2. BIGTOP-2434

Change Spark test to unpack jar file and retrieve resources

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 1.1.0
    • Fix Version/s: 1.2.0
    • Component/s: tests
    • Labels:
      None

      Description

      CDH doesn't create examples resources (namely, the source code and data files) in Spark, so the smoke test is failing because it looks in this directory for certain files.

      The workaround for this issue is to unpack the spark-examples.jar file and grab the necessary directory/files from there.

      1. BIGTOP-2434.patch
        2 kB
        Dasha Boudnik
      2. BIGTOP-2434.patch
        2 kB
        Dasha Boudnik

        Issue Links

          Activity

          Hide
          dasha.boudnik Dasha Boudnik added a comment -

          First version of patch attached, just a draft, not in working condition just yet

          Show
          dasha.boudnik Dasha Boudnik added a comment - First version of patch attached, just a draft, not in working condition just yet
          Hide
          cos Konstantin Boudnik added a comment -

          Clearly, this test is completely broken in the Bigtop... It is using yarn-client mode of submission, which we aren't even bother to touch in the deployment. I will open a separate ticket to fix that problem, once your patch is in.

          Show
          cos Konstantin Boudnik added a comment - Clearly, this test is completely broken in the Bigtop... It is using yarn-client mode of submission, which we aren't even bother to touch in the deployment. I will open a separate ticket to fix that problem, once your patch is in.
          Hide
          dasha.boudnik Dasha Boudnik added a comment -

          Attached an updated version of the patch which is now passing on a Bigtop cluster. Let me know if I can commit

          Show
          dasha.boudnik Dasha Boudnik added a comment - Attached an updated version of the patch which is now passing on a Bigtop cluster. Let me know if I can commit
          Hide
          cos Konstantin Boudnik added a comment -

          We are Commit-Then-Review project. So if you're confident that the fix is fine and you don't seek any feedback - just commit it.

          Show
          cos Konstantin Boudnik added a comment - We are Commit-Then-Review project. So if you're confident that the fix is fine and you don't seek any feedback - just commit it.
          Hide
          dasha.boudnik Dasha Boudnik added a comment -

          Got it Committed and pushed, thanks!

          5e4bf8e23f4f220fe7247df9ee4126fa72bb0557

          Show
          dasha.boudnik Dasha Boudnik added a comment - Got it Committed and pushed, thanks! 5e4bf8e23f4f220fe7247df9ee4126fa72bb0557
          Hide
          dasha.boudnik Dasha Boudnik added a comment -

          Because of this bug – SPARK-11102 – this test won't pass on CDH 5.5.

          Show
          dasha.boudnik Dasha Boudnik added a comment - Because of this bug – SPARK-11102 – this test won't pass on CDH 5.5.
          Hide
          cos Konstantin Boudnik added a comment -

          Thanks for the fix Dasha Boudnik. BTW, the issues with downstream distros aren't of our concern, because we can't control what they put into their versions of official Apache release.

          Show
          cos Konstantin Boudnik added a comment - Thanks for the fix Dasha Boudnik . BTW, the issues with downstream distros aren't of our concern, because we can't control what they put into their versions of official Apache release.

            People

            • Assignee:
              dasha.boudnik Dasha Boudnik
              Reporter:
              dasha.boudnik Dasha Boudnik
            • Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development