Uploaded image for project: 'Bigtop'
  1. Bigtop
  2. BIGTOP-2133

Running BPSGenerator using fat-jar fails on Mac OS X

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Minor
    • Resolution: Fixed
    • Affects Version/s: 1.1.0
    • Fix Version/s: 1.2.0
    • Component/s: blueprints
    • Labels:

      Description

      I tried bigpetstore-mapreduce following the "Running on a hadoop cluster" section in bigtop-bigpetstore/bigpetstore-mapreduce/README.md on OS X.
      First, I created fat-jar using the following command:

      gradle clean shadowJar -Pfor-cluster
      

      Then, I ran BPSGenerator, but encountered the following error:

      [sekikn@mobile bigpetstore-mapreduce]$ hadoop jar build/libs/BigPetStore-1.1.0-SNAPSHOT-all.jar org.apache.bigtop.bigpetstore.generator.BPSGenerator 1000000 bigpetstore/gen
      Exception in thread "main" java.io.IOException: Mkdirs failed to create /var/folders/n2/1bnspz7j4q7100jmh610zd200000gn/T/hadoop-unjar4851530034973666155/META-INF/license
      	at org.apache.hadoop.util.RunJar.ensureDirectory(RunJar.java:128)
      	at org.apache.hadoop.util.RunJar.unJar(RunJar.java:104)
      	at org.apache.hadoop.util.RunJar.unJar(RunJar.java:81)
      	at org.apache.hadoop.util.RunJar.run(RunJar.java:209)
      	at org.apache.hadoop.util.RunJar.main(RunJar.java:136)
      

      This is because both META-INF/LICENSE (file) and META-INF/license (directory) exist in the fat-jar and the default filesystem of OS X doesn't differentiate them.

      [sekikn@mobile bigpetstore-mapreduce]$ jar tf build/libs/BigPetStore-1.1.0-SNAPSHOT-all.jar | grep -i -E META-INF/license/?$
      META-INF/LICENSE
      META-INF/license/
      

      If either of them is removed, generated jar works well.

      1. BIGTOP-2133.01.patch
        1.0 kB
        Kengo Seki
      2. BIGTOP-2133.patch
        0.7 kB
        Kengo Seki

        Activity

        Hide
        sekikn Kengo Seki added a comment -

        Attaching a patch, but I think Bigtop doesn't support OS X officially. Please close this JIRA as "won't fix" if it's not a problem. Thanks!

        Show
        sekikn Kengo Seki added a comment - Attaching a patch, but I think Bigtop doesn't support OS X officially. Please close this JIRA as "won't fix" if it's not a problem. Thanks!
        Hide
        oflebbe Olaf Flebbe added a comment -

        I would accept this patch, if you put a comment line before why this is necessary.

        The FatJar is surely a mess, and leaving out an apache License from a unusual place is o.k., since an additional Copy is contained in the root of the FatJar, too.

        Show
        oflebbe Olaf Flebbe added a comment - I would accept this patch, if you put a comment line before why this is necessary. The FatJar is surely a mess, and leaving out an apache License from a unusual place is o.k., since an additional Copy is contained in the root of the FatJar, too.
        Hide
        jayunit100 jay vyas added a comment -

        +1 pending olaf's comment. can you update the patch?

        Show
        jayunit100 jay vyas added a comment - +1 pending olaf's comment. can you update the patch?
        Hide
        sekikn Kengo Seki added a comment -

        Thanks for the comments, I'll update the patch after a while.

        Show
        sekikn Kengo Seki added a comment - Thanks for the comments, I'll update the patch after a while.
        Hide
        sekikn Kengo Seki added a comment -

        Patch updated.

        Show
        sekikn Kengo Seki added a comment - Patch updated.
        Hide
        oflebbe Olaf Flebbe added a comment -

        Thanks Kengo Seki for proposing the patch ! Committed.

        Show
        oflebbe Olaf Flebbe added a comment - Thanks Kengo Seki for proposing the patch ! Committed.
        Hide
        cos Konstantin Boudnik added a comment -

        There's another undesirable effect of the shadowing: the resulting fat jar might contain versions of the class from different versions of the library and inadvertently introduced bugs into the system. Said issues are very difficult to trace and diagnose.

        Show
        cos Konstantin Boudnik added a comment - There's another undesirable effect of the shadowing: the resulting fat jar might contain versions of the class from different versions of the library and inadvertently introduced bugs into the system. Said issues are very difficult to trace and diagnose.

          People

          • Assignee:
            sekikn Kengo Seki
            Reporter:
            sekikn Kengo Seki
          • Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development