Uploaded image for project: 'Bigtop'
  1. Bigtop
  2. BIGTOP-2135

PigCSVCleaner fails due to the lack of the dependent jar

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 1.1.0
    • Fix Version/s: 1.2.0
    • Component/s: blueprints
    • Labels:

      Description

      I created fat-jar and ran PigCSVCleaner following bigpetstore-mapreduce/README.md and bigpetstore-mapreduce/arch.dot, but it failed as follows:

      [sekikn@mobile bigpetstore-mapreduce]$ hadoop jar build/libs/BigPetStore-1.1.0-SNAPSHOT-all.jar org.apache.bigtop.bigpetstore.etl.PigCSVCleaner bps/generated bps/cleaned
      Starting pig etl 2
      
      (snip)
      
      Exception in thread "main" java.lang.NoClassDefFoundError: dk/brics/automaton/Automaton
      	at org.apache.pig.impl.util.JarManager$DefaultPigPackages.<clinit>(JarManager.java:108)
      	at org.apache.pig.impl.util.JarManager.createJar(JarManager.java:148)
      	at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler.getJob(JobControlCompiler.java:517)
      	at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler.compile(JobControlCompiler.java:296)
      	at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher.launchPig(MapReduceLauncher.java:190)
      	at org.apache.pig.PigServer.launchPlan(PigServer.java:1322)
      	at org.apache.pig.PigServer.executeCompiledLogicalPlan(PigServer.java:1307)
      	at org.apache.pig.PigServer.storeEx(PigServer.java:978)
      	at org.apache.pig.PigServer.store(PigServer.java:942)
      	at org.apache.pig.PigServer.store(PigServer.java:910)
      	at org.apache.bigtop.bigpetstore.etl.PigCSVCleaner.<init>(PigCSVCleaner.java:85)
      	at org.apache.bigtop.bigpetstore.etl.PigCSVCleaner$1.run(PigCSVCleaner.java:146)
      	at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
      	at org.apache.bigtop.bigpetstore.etl.PigCSVCleaner.main(PigCSVCleaner.java:128)
      	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
      	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
      	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
      	at java.lang.reflect.Method.invoke(Method.java:497)
      	at org.apache.hadoop.util.RunJar.run(RunJar.java:221)
      	at org.apache.hadoop.util.RunJar.main(RunJar.java:136)
      Caused by: java.lang.ClassNotFoundException: dk.brics.automaton.Automaton
      	at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
      	at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
      	at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
      	... 20 more
      [sekikn@mobile bigpetstore-mapreduce]$ 
      

        Activity

        Hide
        sekikn Kengo Seki added a comment -

        Attaching a patch. After applying this patch and re-generating fat-jar, PigCSVCleaner succeeds:

        [sekikn@mobile bigpetstore-mapreduce]$ hadoop jar build/libs/BigPetStore-1.1.0-SNAPSHOT-all.jar org.apache.bigtop.bigpetstore.etl.PigCSVCleaner bps/generated bps/cleaned
        Starting pig etl 2
        
        (snip)
        
        15/11/13 15:30:39 INFO mapReduceLayer.MapReduceLauncher: Success!
        
        Show
        sekikn Kengo Seki added a comment - Attaching a patch. After applying this patch and re-generating fat-jar, PigCSVCleaner succeeds: [sekikn@mobile bigpetstore-mapreduce]$ hadoop jar build/libs/BigPetStore-1.1.0-SNAPSHOT-all.jar org.apache.bigtop.bigpetstore.etl.PigCSVCleaner bps/generated bps/cleaned Starting pig etl 2 (snip) 15/11/13 15:30:39 INFO mapReduceLayer.MapReduceLauncher: Success!
        Hide
        jayunit100 jay vyas added a comment -

        good fix. Maybe last time we updated the pig code we didn't test in a cluster.

        +1.

        I can try to commit these when I get home, am flying across the country.

        Otherwise maybe RJ Nowling can push these two patches in ( BIGTOP-2136 also).

        Thanks again.

        Show
        jayunit100 jay vyas added a comment - good fix. Maybe last time we updated the pig code we didn't test in a cluster. +1. I can try to commit these when I get home, am flying across the country. Otherwise maybe RJ Nowling can push these two patches in ( BIGTOP-2136 also). Thanks again.
        Hide
        rnowling RJ Nowling added a comment -

        I'm also traveling this weekend but I'd be happy to review and push these patches if I get a chance tonight or tomorrow.

        Show
        rnowling RJ Nowling added a comment - I'm also traveling this weekend but I'd be happy to review and push these patches if I get a chance tonight or tomorrow.
        Hide
        oflebbe Olaf Flebbe added a comment -

        Seems to fix the issue. Will commit it.

        Show
        oflebbe Olaf Flebbe added a comment - Seems to fix the issue. Will commit it.
        Hide
        oflebbe Olaf Flebbe added a comment -

        Thanks Kengo Seki !

        Show
        oflebbe Olaf Flebbe added a comment - Thanks Kengo Seki !

          People

          • Assignee:
            sekikn Kengo Seki
            Reporter:
            sekikn Kengo Seki
          • Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development