Nutch
  1. Nutch
  2. NUTCH-1205

Upgrade gora modules to 0.2 in ivy/ivy.xml

    Details

    • Type: Improvement Improvement
    • Status: Closed
    • Priority: Blocker Blocker
    • Resolution: Fixed
    • Affects Version/s: nutchgora
    • Fix Version/s: nutchgora
    • Component/s: storage
    • Labels:
      None
    • Patch Info:
      Patch Available

      Description

      Although gora trunk is unstable, work is ongoing to get this fixed. For the time being, I think Nutchgora should use gora trunk as this will identify more vulnerabilities. I'll get the trivial patch submitted shortly.

      1. NUTCH-1205-v6.patch
        7 kB
        Lewis John McGibbney
      2. NUTCH-1205-v5.patch
        6 kB
        Lewis John McGibbney
      3. NUTCH-1205-v5.patch
        6 kB
        Lewis John McGibbney
      4. NUTCH-1205-v4.patch
        6 kB
        Lewis John McGibbney
      5. NUTCH-1205-v3.patch
        4 kB
        Ferdy Galema
      6. NUTCH-1205-v2.patch
        3 kB
        Lewis John McGibbney
      7. NUTCH-1205-v11-addition.patch
        1 kB
        Ferdy Galema
      8. NUTCH-1205-v11.patch
        19 kB
        Ferdy Galema
      9. NUTCH-1205-v10.patch
        20 kB
        Ferdy Galema
      10. NUTCH-1205.patch
        4 kB
        Lewis John McGibbney

        Issue Links

          Activity

          Hide
          Hudson added a comment -

          Integrated in Nutch-nutchgora #244 (See https://builds.apache.org/job/Nutch-nutchgora/244/)
          NUTCH-1205 Upgrade gora modules to 0.2 in ivy/ivy.xml (addition) (Revision 1333551)
          NUTCH-1205 Upgrade gora modules to 0.2 in ivy/ivy.xml (addition) (Revision 1333547)
          NUTCH-1205 Upgrade gora modules to 0.2 in ivy/ivy.xml (Revision 1333435)

          Result = SUCCESS
          ferdy :
          Files :

          • /nutch/branches/nutchgora/ivy/ivy.xml

          ferdy :
          Files :

          • /nutch/branches/nutchgora/ivy/ivy.xml

          ferdy :
          Files :

          • /nutch/branches/nutchgora/CHANGES.txt
          • /nutch/branches/nutchgora/build.xml
          • /nutch/branches/nutchgora/conf/gora.properties
          • /nutch/branches/nutchgora/ivy/ivy.xml
          • /nutch/branches/nutchgora/src/java/org/apache/nutch/storage/StorageUtils.java
          • /nutch/branches/nutchgora/src/test/gora.properties
          • /nutch/branches/nutchgora/src/test/org/apache/nutch/storage/TestGoraStorage.java
          • /nutch/branches/nutchgora/src/test/org/apache/nutch/util/AbstractNutchTest.java
          • /nutch/branches/nutchgora/src/testprocess
          • /nutch/branches/nutchgora/src/testprocess/gora.properties
          Show
          Hudson added a comment - Integrated in Nutch-nutchgora #244 (See https://builds.apache.org/job/Nutch-nutchgora/244/ ) NUTCH-1205 Upgrade gora modules to 0.2 in ivy/ivy.xml (addition) (Revision 1333551) NUTCH-1205 Upgrade gora modules to 0.2 in ivy/ivy.xml (addition) (Revision 1333547) NUTCH-1205 Upgrade gora modules to 0.2 in ivy/ivy.xml (Revision 1333435) Result = SUCCESS ferdy : Files : /nutch/branches/nutchgora/ivy/ivy.xml ferdy : Files : /nutch/branches/nutchgora/ivy/ivy.xml ferdy : Files : /nutch/branches/nutchgora/CHANGES.txt /nutch/branches/nutchgora/build.xml /nutch/branches/nutchgora/conf/gora.properties /nutch/branches/nutchgora/ivy/ivy.xml /nutch/branches/nutchgora/src/java/org/apache/nutch/storage/StorageUtils.java /nutch/branches/nutchgora/src/test/gora.properties /nutch/branches/nutchgora/src/test/org/apache/nutch/storage/TestGoraStorage.java /nutch/branches/nutchgora/src/test/org/apache/nutch/util/AbstractNutchTest.java /nutch/branches/nutchgora/src/testprocess /nutch/branches/nutchgora/src/testprocess/gora.properties
          Lewis John McGibbney made changes -
          Status Resolved [ 5 ] Closed [ 6 ]
          Hide
          Lewis John McGibbney added a comment -

          Excellent Ferdy. I'll buy you some nice whisky if we get a chance to catch up at some stage in the future :0)
          Closing this one off.
          Good work.

          Show
          Lewis John McGibbney added a comment - Excellent Ferdy. I'll buy you some nice whisky if we get a chance to catch up at some stage in the future :0) Closing this one off. Good work.
          Ferdy Galema made changes -
          Attachment NUTCH-1205-v11-addition.patch [ 12525466 ]
          Hide
          Ferdy Galema added a comment -

          I committed a minor addition, that fixes the maven-plugins error when uncommenting another store.

          Show
          Ferdy Galema added a comment - I committed a minor addition, that fixes the maven-plugins error when uncommenting another store.
          Ferdy Galema made changes -
          Status In Progress [ 3 ] Resolved [ 5 ]
          Resolution Fixed [ 1 ]
          Hide
          Ferdy Galema added a comment -

          Attached new patch v11. Committed.

          -Fixed the jdom issue. (Added test dep again).
          -Added a single global exclusion for hsqldb. (The deps can have the exclusion removed).
          -Tests succeed.
          -Build a sqlstore runtime and played around doing some local crawls succesfully.

          I did not test a deployment for the other stores. (When there is something wrong with one of them dependency-wise, we can always create new issues).

          Show
          Ferdy Galema added a comment - Attached new patch v11. Committed. -Fixed the jdom issue. (Added test dep again). -Added a single global exclusion for hsqldb. (The deps can have the exclusion removed). -Tests succeed. -Build a sqlstore runtime and played around doing some local crawls succesfully. I did not test a deployment for the other stores. (When there is something wrong with one of them dependency-wise, we can always create new issues).
          Ferdy Galema made changes -
          Attachment NUTCH-1205-v11.patch [ 12525439 ]
          Hide
          Lewis John McGibbney added a comment -

          Hi Ferdy, I am very happy with this. It's been a bit of a sticky one as I'm sure you'll agree. Anyway, one comment I have is that, I removed the jdom deps because I thought somewhere down the line a transitive dependency was mucking things up. As we now know this is not the case, I notice that you added one of the jdom dependencies back in. If you look at the current ivy.xml file in our svn area, you'll see the presence of two jdom dependencies, one appears to be for tests and the other I assume for runtime configuration.

          I've tested this patch against our test suite e.g. ant test, and confirm that all tests pass successfully. I've also run the benchmark but will comment about this elsewhere where it is relevant.

          If we can clear up the jdom discrepancy I note above then I am +1 on this one. Great work Ferdy.

          Show
          Lewis John McGibbney added a comment - Hi Ferdy, I am very happy with this. It's been a bit of a sticky one as I'm sure you'll agree. Anyway, one comment I have is that, I removed the jdom deps because I thought somewhere down the line a transitive dependency was mucking things up. As we now know this is not the case, I notice that you added one of the jdom dependencies back in. If you look at the current ivy.xml file in our svn area, you'll see the presence of two jdom dependencies, one appears to be for tests and the other I assume for runtime configuration. I've tested this patch against our test suite e.g. ant test, and confirm that all tests pass successfully. I've also run the benchmark but will comment about this elsewhere where it is relevant. If we can clear up the jdom discrepancy I note above then I am +1 on this one. Great work Ferdy.
          Ferdy Galema made changes -
          Attachment NUTCH-1205-v9.patch [ 12524870 ]
          Ferdy Galema made changes -
          Attachment NUTCH-1205-v9.patch [ 12524871 ]
          Ferdy Galema made changes -
          Attachment NUTCH-1205-v8.patch [ 12524862 ]
          Ferdy Galema made changes -
          Attachment NUTCH-1205-v7.patch [ 12524852 ]
          Ferdy Galema made changes -
          Attachment NUTCH-1205-v10.patch [ 12525269 ]
          Hide
          Ferdy Galema added a comment -

          The tests now work and TestGoraStorage uses a proper standalone database. (Integrating issue NUTCH-902). I think it's good to do a final check of what's to be included as dependencies in the ivy.xml.

          Show
          Ferdy Galema added a comment - The tests now work and TestGoraStorage uses a proper standalone database. (Integrating issue NUTCH-902 ). I think it's good to do a final check of what's to be included as dependencies in the ivy.xml.
          Ferdy Galema made changes -
          Link This issue incorporates NUTCH-896 [ NUTCH-896 ]
          Ferdy Galema made changes -
          Attachment NUTCH-1205-v9.patch [ 12524870 ]
          Attachment NUTCH-1205-v9.patch [ 12524871 ]
          Hide
          Ferdy Galema added a comment -

          This is getting hilarious. The tests are working but TestGoraStorage multiproces actually just connects to local mem sql stores.

          So, to recap the current state of affairs:
          The tests extending AbstractNutchText work validly. (Because they simply need a mem sql store). TestGoraStorage works but actually does not test multi-process. Somehow setting the sql url properties does not seem to reflect on the sql store.

          Any suggestions?

          Show
          Ferdy Galema added a comment - This is getting hilarious. The tests are working but TestGoraStorage multiproces actually just connects to local mem sql stores. So, to recap the current state of affairs: The tests extending AbstractNutchText work validly. (Because they simply need a mem sql store). TestGoraStorage works but actually does not test multi-process. Somehow setting the sql url properties does not seem to reflect on the sql store. Any suggestions?
          Ferdy Galema made changes -
          Attachment NUTCH-1205-v8.patch [ 12524862 ]
          Hide
          Ferdy Galema added a comment -

          Oops there still was a failure in a test later on. (TestProtocolHttpClient). This was because of multiple ant jars. I noticed that this was caused by removing the global exclude but adding excludes to hadoop deps. (This was not sufficient obviously).

          New version of patch succeeds all tests.

          Show
          Ferdy Galema added a comment - Oops there still was a failure in a test later on. (TestProtocolHttpClient). This was because of multiple ant jars. I noticed that this was caused by removing the global exclude but adding excludes to hadoop deps. (This was not sufficient obviously). New version of patch succeeds all tests.
          Hide
          Ferdy Galema added a comment -

          (Also I reformatted the ivy.xml to only include spaces as indentation. This is the policy right? If so, could anyone editing xml files double check their editor settings.)

          Show
          Ferdy Galema added a comment - (Also I reformatted the ivy.xml to only include spaces as indentation. This is the policy right? If so, could anyone editing xml files double check their editor settings.)
          Ferdy Galema made changes -
          Attachment NUTCH-1205-v7.patch [ 12524852 ]
          Hide
          Ferdy Galema added a comment -

          OK I got the tests working now. The problem is the fact that properties object are not correctly handled throughout the tests. This is a problem currently in Gora. In short, it means that Properties are not properly loaded in GoraMapper from dynamic properties but ALWAYS from static gora.properties. (Will shortly open issue for that). A consequence to make tests work now with Gora 0.2 is that the default gora.properties now uses a hsqldb memstore instead of a standalone hsqldb server.

          Lewis, I noticed that you excluded jdom in the ivy.xml? Why is that? I included it again because the SqlStore needs it to reads it mapping.

          Show
          Ferdy Galema added a comment - OK I got the tests working now. The problem is the fact that properties object are not correctly handled throughout the tests. This is a problem currently in Gora. In short, it means that Properties are not properly loaded in GoraMapper from dynamic properties but ALWAYS from static gora.properties. (Will shortly open issue for that). A consequence to make tests work now with Gora 0.2 is that the default gora.properties now uses a hsqldb memstore instead of a standalone hsqldb server. Lewis, I noticed that you excluded jdom in the ivy.xml? Why is that? I included it again because the SqlStore needs it to reads it mapping.
          Lewis John McGibbney made changes -
          Attachment NUTCH-1205-v6.patch [ 12524778 ]
          Hide
          Lewis John McGibbney added a comment - - edited

          Hi Ferdy. If you would be so good to look at the patch. If you also observe the way in which we do the following in AbstractNutchTest

           
          52 DataStoreFactory.properties.setProperty("gora.sqlstore.jdbc.password","");
          53 webPageStore = StorageUtils.createWebStore(conf, String.class,
          54    WebPage.class);
          

          you will see that we are not passing the correct parameters to StorageUtils.createWebStore. This makes all tests that rely on AbstractNutchTest fail misserably complaining about connection refused errors. (I'm hoping that this is correct, because if not then I am pretty much stumped.)

          Show
          Lewis John McGibbney added a comment - - edited Hi Ferdy. If you would be so good to look at the patch. If you also observe the way in which we do the following in AbstractNutchTest 52 DataStoreFactory.properties.setProperty( "gora.sqlstore.jdbc.password" ,""); 53 webPageStore = StorageUtils.createWebStore(conf, String .class, 54 WebPage.class); you will see that we are not passing the correct parameters to StorageUtils.createWebStore. This makes all tests that rely on AbstractNutchTest fail misserably complaining about connection refused errors. (I'm hoping that this is correct, because if not then I am pretty much stumped.)
          Lewis John McGibbney made changes -
          Status Open [ 1 ] In Progress [ 3 ]
          Lewis John McGibbney made changes -
          Summary Upgrade gora modules to 0.2-SNAPSHOT in ivy/ivy.xml Upgrade gora modules to 0.2 in ivy/ivy.xml
          Hide
          Lewis John McGibbney added a comment -

          One last thing, I've unchecked all of the gora dependencies, not as this will be default configuration, but because we need to be 100% certain that we can gather them all, that everything compiles, tests and functions properly. Currently with this patch in place this doesn't seem to be the case.

          Show
          Lewis John McGibbney added a comment - One last thing, I've unchecked all of the gora dependencies, not as this will be default configuration, but because we need to be 100% certain that we can gather them all, that everything compiles, tests and functions properly. Currently with this patch in place this doesn't seem to be the case.
          Lewis John McGibbney made changes -
          Attachment NUTCH-1205-v5.patch [ 12515271 ]
          Attachment NUTCH-1205-v5.patch [ 12515272 ]
          Hide
          Lewis John McGibbney added a comment -

          This is getting laughable now. I've overcome god knows how many problems, but now there is a problem with the actual gora-core-0.2-SNAPSHOT jar which we pull from the nexus snapshot repository. For some reason we are pulling the test jar and not the functional one!!!
          When applying the patch to Nutchgora and running

          $ant compile > compile.txt
          

          I get

          compile-core:
              [javac] /home/lewis/ASF/nutchgora/build.xml:97: warning: 'includeantruntime' was not set, defaulting to build.sysclasspath=last; set to false for repeatable builds
              [javac] Compiling 170 source files to /home/lewis/ASF/nutchgora/build/classes
              [javac] warning: [path] bad path element "/home/lewis/ASF/nutchgora/build/lib/apache-cassandra-clientutil-1.0.2.jar": no such file or directory
              [javac] warning: [path] bad path element "/home/lewis/ASF/nutchgora/build/lib/apache-cassandra-thrift-1.0.2.jar": no such file or directory
              [javac] warning: [path] bad path element "/home/lewis/ASF/nutchgora/build/lib/bcel.jar": no such file or directory
              [javac] warning: [path] bad path element "/home/lewis/ASF/nutchgora/build/lib/dom4j-full.jar": no such file or directory
              [javac] warning: [path] bad path element "/home/lewis/ASF/nutchgora/build/lib/findbugs.jar": no such file or directory
              [javac] warning: [path] bad path element "/home/lewis/ASF/nutchgora/build/lib/plastic.jar": no such file or directory
              [javac] warning: [path] bad path element "/home/lewis/ASF/nutchgora/build/lib/jaxb-api.jar": no such file or directory
              [javac] warning: [path] bad path element "/home/lewis/ASF/nutchgora/build/lib/activation.jar": no such file or directory
              [javac] warning: [path] bad path element "/home/lewis/ASF/nutchgora/build/lib/jsr173_1.0_api.jar": no such file or directory
              [javac] warning: [path] bad path element "/home/lewis/ASF/nutchgora/build/lib/jaxb1-impl.jar": no such file or directory
              [javac] warning: [path] bad path element "/home/lewis/ASF/nutchgora/build/lib/xercesImpl.jar": no such file or directory
              [javac] warning: [path] bad path element "/home/lewis/ASF/nutchgora/build/lib/xml-apis.jar": no such file or directory
          

          When you look in your /build directory and open the gora-core-0.2-SNAPSHOT.jar you'll see that it's the test jar right enough.

          For tonight I'm giving it a by, but will try and pick this up tomorrow at some stage.

          Show
          Lewis John McGibbney added a comment - This is getting laughable now. I've overcome god knows how many problems, but now there is a problem with the actual gora-core-0.2-SNAPSHOT jar which we pull from the nexus snapshot repository. For some reason we are pulling the test jar and not the functional one!!! When applying the patch to Nutchgora and running $ant compile > compile.txt I get compile-core: [javac] /home/lewis/ASF/nutchgora/build.xml:97: warning: 'includeantruntime' was not set, defaulting to build.sysclasspath=last; set to false for repeatable builds [javac] Compiling 170 source files to /home/lewis/ASF/nutchgora/build/classes [javac] warning: [path] bad path element "/home/lewis/ASF/nutchgora/build/lib/apache-cassandra-clientutil-1.0.2.jar" : no such file or directory [javac] warning: [path] bad path element "/home/lewis/ASF/nutchgora/build/lib/apache-cassandra-thrift-1.0.2.jar" : no such file or directory [javac] warning: [path] bad path element "/home/lewis/ASF/nutchgora/build/lib/bcel.jar" : no such file or directory [javac] warning: [path] bad path element "/home/lewis/ASF/nutchgora/build/lib/dom4j-full.jar" : no such file or directory [javac] warning: [path] bad path element "/home/lewis/ASF/nutchgora/build/lib/findbugs.jar" : no such file or directory [javac] warning: [path] bad path element "/home/lewis/ASF/nutchgora/build/lib/plastic.jar" : no such file or directory [javac] warning: [path] bad path element "/home/lewis/ASF/nutchgora/build/lib/jaxb-api.jar" : no such file or directory [javac] warning: [path] bad path element "/home/lewis/ASF/nutchgora/build/lib/activation.jar" : no such file or directory [javac] warning: [path] bad path element "/home/lewis/ASF/nutchgora/build/lib/jsr173_1.0_api.jar" : no such file or directory [javac] warning: [path] bad path element "/home/lewis/ASF/nutchgora/build/lib/jaxb1-impl.jar" : no such file or directory [javac] warning: [path] bad path element "/home/lewis/ASF/nutchgora/build/lib/xercesImpl.jar" : no such file or directory [javac] warning: [path] bad path element "/home/lewis/ASF/nutchgora/build/lib/xml-apis.jar" : no such file or directory When you look in your /build directory and open the gora-core-0.2-SNAPSHOT.jar you'll see that it's the test jar right enough. For tonight I'm giving it a by, but will try and pick this up tomorrow at some stage.
          Lewis John McGibbney made changes -
          Priority Minor [ 4 ] Blocker [ 1 ]
          Hide
          Lewis John McGibbney added a comment -

          Yeah it's another kettle of fish altogether. I'll get on it and hopefully get it sorted out. I'll ensure that the final patch includes the hsqldb upgrage as well Ferdy. Thanks for now.

          Show
          Lewis John McGibbney added a comment - Yeah it's another kettle of fish altogether. I'll get on it and hopefully get it sorted out. I'll ensure that the final patch includes the hsqldb upgrage as well Ferdy. Thanks for now.
          Hide
          Ferdy Galema added a comment -

          Haven't got the slightest clue about why this error appears, sorry.. I simply do not understand how this ivy/maven repos stuff works.

          Show
          Ferdy Galema added a comment - Haven't got the slightest clue about why this error appears, sorry.. I simply do not understand how this ivy/maven repos stuff works.
          Hide
          Lewis John McGibbney added a comment -

          To add to this, I can confirm that we are pulling the most up to date maven artifacts from the apache snapshots repository, so at least we are using bleeding edge here.

          Show
          Lewis John McGibbney added a comment - To add to this, I can confirm that we are pulling the most up to date maven artifacts from the apache snapshots repository, so at least we are using bleeding edge here.
          Hide
          Lewis John McGibbney added a comment -

          OK when I apply the patch, I'm seeing

          [ivy:resolve] :: problems summary ::
          [ivy:resolve] :::: WARNINGS
          [ivy:resolve] 		[FAILED     ] maven-plugins#maven-cobertura-plugin;1.3!maven-cobertura-plugin.plugin:  (0ms)
          [ivy:resolve] 	==== local: tried
          [ivy:resolve] 	  /home/lewis/.ivy2/local/maven-plugins/maven-cobertura-plugin/1.3/plugins/maven-cobertura-plugin.plugin
          [ivy:resolve] 	==== maven2: tried
          [ivy:resolve] 	  http://repo1.maven.org/maven2/maven-plugins/maven-cobertura-plugin/1.3/maven-cobertura-plugin-1.3.plugin
          [ivy:resolve] 	==== apache-snapshot: tried
          [ivy:resolve] 	  http://repository.apache.org/content/groups/snapshots-group/maven-plugins/maven-cobertura-plugin/1.3/maven-cobertura-plugin-1.3.plugin
          [ivy:resolve] 		[FAILED     ] maven-plugins#maven-findbugs-plugin;1.3.1!maven-findbugs-plugin.plugin:  (0ms)
          [ivy:resolve] 	==== local: tried
          [ivy:resolve] 	  /home/lewis/.ivy2/local/maven-plugins/maven-findbugs-plugin/1.3.1/plugins/maven-findbugs-plugin.plugin
          [ivy:resolve] 	==== maven2: tried
          [ivy:resolve] 	  http://repo1.maven.org/maven2/maven-plugins/maven-findbugs-plugin/1.3.1/maven-findbugs-plugin-1.3.1.plugin
          [ivy:resolve] 	==== apache-snapshot: tried
          [ivy:resolve] 	  http://repository.apache.org/content/groups/snapshots-group/maven-plugins/maven-findbugs-plugin/1.3.1/maven-findbugs-plugin-1.3.1.plugin
          [ivy:resolve] 		::::::::::::::::::::::::::::::::::::::::::::::
          [ivy:resolve] 		::              FAILED DOWNLOADS            ::
          [ivy:resolve] 		:: ^ see resolution messages for details  ^ ::
          [ivy:resolve] 		::::::::::::::::::::::::::::::::::::::::::::::
          [ivy:resolve] 		:: maven-plugins#maven-cobertura-plugin;1.3!maven-cobertura-plugin.plugin
          [ivy:resolve] 		:: maven-plugins#maven-findbugs-plugin;1.3.1!maven-findbugs-plugin.plugin
          [ivy:resolve] 		::::::::::::::::::::::::::::::::::::::::::::::
          
          

          There is a really weird extension for the plugins e.g.

          maven-cobertura-plugin.plugin
          

          I've tried excluding these as both individual exclusions for the Gora atrifacts and as global exclusions for maven-plugins but noneof this works. Been doing some reading on ivysettings on the ant/ivy website but there is a bit fo documentation so it's taking a while.

          Show
          Lewis John McGibbney added a comment - OK when I apply the patch, I'm seeing [ivy:resolve] :: problems summary :: [ivy:resolve] :::: WARNINGS [ivy:resolve] [FAILED ] maven-plugins#maven-cobertura-plugin;1.3!maven-cobertura-plugin.plugin: (0ms) [ivy:resolve] ==== local: tried [ivy:resolve] /home/lewis/.ivy2/local/maven-plugins/maven-cobertura-plugin/1.3/plugins/maven-cobertura-plugin.plugin [ivy:resolve] ==== maven2: tried [ivy:resolve] http: //repo1.maven.org/maven2/maven-plugins/maven-cobertura-plugin/1.3/maven-cobertura-plugin-1.3.plugin [ivy:resolve] ==== apache-snapshot: tried [ivy:resolve] http: //repository.apache.org/content/groups/snapshots-group/maven-plugins/maven-cobertura-plugin/1.3/maven-cobertura-plugin-1.3.plugin [ivy:resolve] [FAILED ] maven-plugins#maven-findbugs-plugin;1.3.1!maven-findbugs-plugin.plugin: (0ms) [ivy:resolve] ==== local: tried [ivy:resolve] /home/lewis/.ivy2/local/maven-plugins/maven-findbugs-plugin/1.3.1/plugins/maven-findbugs-plugin.plugin [ivy:resolve] ==== maven2: tried [ivy:resolve] http: //repo1.maven.org/maven2/maven-plugins/maven-findbugs-plugin/1.3.1/maven-findbugs-plugin-1.3.1.plugin [ivy:resolve] ==== apache-snapshot: tried [ivy:resolve] http: //repository.apache.org/content/groups/snapshots-group/maven-plugins/maven-findbugs-plugin/1.3.1/maven-findbugs-plugin-1.3.1.plugin [ivy:resolve] :::::::::::::::::::::::::::::::::::::::::::::: [ivy:resolve] :: FAILED DOWNLOADS :: [ivy:resolve] :: ^ see resolution messages for details ^ :: [ivy:resolve] :::::::::::::::::::::::::::::::::::::::::::::: [ivy:resolve] :: maven-plugins#maven-cobertura-plugin;1.3!maven-cobertura-plugin.plugin [ivy:resolve] :: maven-plugins#maven-findbugs-plugin;1.3.1!maven-findbugs-plugin.plugin [ivy:resolve] :::::::::::::::::::::::::::::::::::::::::::::: There is a really weird extension for the plugins e.g. maven-cobertura-plugin.plugin I've tried excluding these as both individual exclusions for the Gora atrifacts and as global exclusions for maven-plugins but noneof this works. Been doing some reading on ivysettings on the ant/ivy website but there is a bit fo documentation so it's taking a while.
          Hide
          Ferdy Galema added a comment -

          Could you please elaborate on what exactly the problems are and how to reproduce them?

          Show
          Ferdy Galema added a comment - Could you please elaborate on what exactly the problems are and how to reproduce them?
          Lewis John McGibbney made changes -
          Summary Upgrade gora modules to 0.2-incubating in ivy/ivy.xml Upgrade gora modules to 0.2-SNAPSHOT in ivy/ivy.xml
          Lewis John McGibbney made changes -
          Attachment NUTCH-1205-v4.patch [ 12514223 ]
          Hide
          Lewis John McGibbney added a comment -

          This patch updates the deprecated ivy resolver enabling us to utilise the bleeding edge Gora stuff. There is a problem with two maven-plugin dependencies, I don't know where they are coming from so I thought I would put this patch up to see if anyone can resolve it themself! Thanks

          Show
          Lewis John McGibbney added a comment - This patch updates the deprecated ivy resolver enabling us to utilise the bleeding edge Gora stuff. There is a problem with two maven-plugin dependencies, I don't know where they are coming from so I thought I would put this patch up to see if anyone can resolve it themself! Thanks
          Hide
          Ferdy Galema added a comment -

          Just to clarify, after applying v3 patch (and making sure that the right 0.2-SNAPSHOT or 0.2-incubating is installed in your local ivy2 repo) the TestGoraStorage multithreaded test fails. This is because of SqlStore not being thread-safe, see details in GORA-84.

          Show
          Ferdy Galema added a comment - Just to clarify, after applying v3 patch (and making sure that the right 0.2-SNAPSHOT or 0.2-incubating is installed in your local ivy2 repo) the TestGoraStorage multithreaded test fails. This is because of SqlStore not being thread-safe, see details in GORA-84 .
          Ferdy Galema made changes -
          Link This issue depends upon GORA-84 [ GORA-84 ]
          Ferdy Galema made changes -
          Attachment NUTCH-1205-v3.patch [ 12511244 ]
          Ferdy Galema made changes -
          Attachment NUTCH-1205-v3.patch [ 12511246 ]
          Hide
          Ferdy Galema added a comment -

          Reattaching because of "grant license to ASF"..

          Show
          Ferdy Galema added a comment - Reattaching because of "grant license to ASF"..
          Ferdy Galema made changes -
          Attachment NUTCH-1205-v3.patch [ 12511244 ]
          Hide
          Ferdy Galema added a comment -

          There, this patch fixes the two "createDataStore" compile errors. (The existing changes of v2 are untouched.)

          Show
          Ferdy Galema added a comment - There, this patch fixes the two "createDataStore" compile errors. (The existing changes of v2 are untouched.)
          Lewis John McGibbney made changes -
          Link This issue is part of NUTCH-902 [ NUTCH-902 ]
          Show
          Lewis John McGibbney added a comment - for reference to fix the above problems http://stackoverflow.com/questions/197986/what-causes-javac-to-issue-the-uses-unchecked-or-unsafe-operations-warning
          Lewis John McGibbney made changes -
          Summary Upgrade gora modules to 0.2-SNAPSHOT Upgrade gora modules to 0.2-incubating in ivy/ivy.xml
          Lewis John McGibbney made changes -
          Attachment NUTCH-1205-v2.patch [ 12508688 ]
          Hide
          Lewis John McGibbney added a comment -

          new patch which acknowledges Juliens comments regarding other dependencies. This patch ONLY upgrades the gora-module dependencies to 0.2-incubating

          Show
          Lewis John McGibbney added a comment - new patch which acknowledges Juliens comments regarding other dependencies. This patch ONLY upgrades the gora-module dependencies to 0.2-incubating
          Hide
          Lewis John McGibbney added a comment -

          Patch aside... even if I only change ivy/ivy.xml to

          <dependency org="org.apache.gora" name="gora-core" rev="latest.integration" conf="*->compile"/>	
          <dependency org="org.apache.gora" name="gora-sql" rev="latest.integration" conf="*->compile"/>	
          

          the above errors still occur. Is anyone familiar with o.a.n.s.StorageUtils and I can attempt to walk my way through this one with them? My main target here is to get Nutchgora working with Gora 0.2-incubating.

          From reading the code, would it not make more sense to implement the getDataStore method rather than the createDataDtore method, this is the point of abstracting this stuff away from Nutch? If this is the case then there is some editing of methods in StorageUtils class. Something like this

            @SuppressWarnings("unchecked")
            public static <K, V extends Persistent> DataStore<K, V> createDataStore(Configuration conf,
                Class<K> keyClass, Class<V> persistentClass) throws ClassNotFoundException, GoraException {
              Class<? extends DataStore<K, V>> dataStoreClass =
                (Class<? extends DataStore<K, V>>) getDataStoreClass(conf);
              return DataStoreFactory.*getDataStore*(dataStoreClass, <<<<<
                      keyClass, persistentClass, *remaining parameters* );
            }
          
          Show
          Lewis John McGibbney added a comment - Patch aside... even if I only change ivy/ivy.xml to <dependency org= "org.apache.gora" name= "gora-core" rev= "latest.integration" conf= "*->compile" /> <dependency org= "org.apache.gora" name= "gora-sql" rev= "latest.integration" conf= "*->compile" /> the above errors still occur. Is anyone familiar with o.a.n.s.StorageUtils and I can attempt to walk my way through this one with them? My main target here is to get Nutchgora working with Gora 0.2-incubating. From reading the code, would it not make more sense to implement the getDataStore method rather than the createDataDtore method, this is the point of abstracting this stuff away from Nutch? If this is the case then there is some editing of methods in StorageUtils class. Something like this @SuppressWarnings( "unchecked" ) public static <K, V extends Persistent> DataStore<K, V> createDataStore(Configuration conf, Class <K> keyClass, Class <V> persistentClass) throws ClassNotFoundException, GoraException { Class <? extends DataStore<K, V>> dataStoreClass = ( Class <? extends DataStore<K, V>>) getDataStoreClass(conf); return DataStoreFactory.*getDataStore*(dataStoreClass, <<<<< keyClass, persistentClass, *remaining parameters* ); }
          Hide
          Lewis John McGibbney added a comment -

          Yeah they probably are Julien. How would I find this out for sure?
          Also did you have a look at the errors? I've not the time to do this today but I'll try to get on to it at a later stage in the week. Thanks

          Show
          Lewis John McGibbney added a comment - Yeah they probably are Julien. How would I find this out for sure? Also did you have a look at the errors? I've not the time to do this today but I'll try to get on to it at a later stage in the week. Thanks
          Hide
          Julien Nioche added a comment -

          Why upgrading the sub-dependencies such as commons-lang explicitely? Aren't they pulled transitively along with GORA?

          Show
          Julien Nioche added a comment - Why upgrading the sub-dependencies such as commons-lang explicitely? Aren't they pulled transitively along with GORA?
          Lewis John McGibbney made changes -
          Patch Info Patch Available [ 10042 ]
          Lewis John McGibbney made changes -
          Field Original Value New Value
          Attachment NUTCH-1205.patch [ 12504897 ]
          Hide
          Lewis John McGibbney added a comment -

          This patch is breaking by build but it is a work in progress. In short the patch,
          1) Upgrades all gora dependencies to 0.2-incubating
          2) Upgrades all other dependencies which were also upgraded in current gora trunk.

          The build fails as follows

          compile-core:
              [javac] /home/lewis/ASF/nutchgora/build.xml:97: warning: 'includeantruntime' was not set, defaulting to build.sysclasspath=last; set to false for repeatable builds
              [javac] Compiling 171 source files to /home/lewis/ASF/nutchgora/build/classes
              [javac] /home/lewis/ASF/nutchgora/src/java/org/apache/nutch/storage/StorageUtils.java:43: cannot find symbol
              [javac] symbol  : method createDataStore(java.lang.Class<capture#763 of ? extends org.apache.gora.store.DataStore<K,V>>,java.lang.Class<K>,java.lang.Class<V>)
              [javac] location: class org.apache.gora.store.DataStoreFactory
              [javac]     return DataStoreFactory.createDataStore(dataStoreClass,
              [javac]                            ^
              [javac] /home/lewis/ASF/nutchgora/src/java/org/apache/nutch/storage/StorageUtils.java:59: cannot find symbol
              [javac] symbol  : method createDataStore(java.lang.Class<capture#496 of ? extends org.apache.gora.store.DataStore<K,V>>,java.lang.Class<K>,java.lang.Class<V>,java.lang.String)
              [javac] location: class org.apache.gora.store.DataStoreFactory
              [javac]     return DataStoreFactory.createDataStore(dataStoreClass,
              [javac]                            ^
              [javac] Note: Some input files use or override a deprecated API.
              [javac] Note: Recompile with -Xlint:deprecation for details.
              [javac] Note: Some input files use unchecked or unsafe operations.
              [javac] Note: Recompile with -Xlint:unchecked for details.
              [javac] 2 errors
          
          BUILD FAILED
          
          
          Show
          Lewis John McGibbney added a comment - This patch is breaking by build but it is a work in progress. In short the patch, 1) Upgrades all gora dependencies to 0.2-incubating 2) Upgrades all other dependencies which were also upgraded in current gora trunk. The build fails as follows compile-core: [javac] /home/lewis/ASF/nutchgora/build.xml:97: warning: 'includeantruntime' was not set, defaulting to build.sysclasspath=last; set to false for repeatable builds [javac] Compiling 171 source files to /home/lewis/ASF/nutchgora/build/classes [javac] /home/lewis/ASF/nutchgora/src/java/org/apache/nutch/storage/StorageUtils.java:43: cannot find symbol [javac] symbol : method createDataStore(java.lang. Class <capture#763 of ? extends org.apache.gora.store.DataStore<K,V>>,java.lang. Class <K>,java.lang. Class <V>) [javac] location: class org.apache.gora.store.DataStoreFactory [javac] return DataStoreFactory.createDataStore(dataStoreClass, [javac] ^ [javac] /home/lewis/ASF/nutchgora/src/java/org/apache/nutch/storage/StorageUtils.java:59: cannot find symbol [javac] symbol : method createDataStore(java.lang. Class <capture#496 of ? extends org.apache.gora.store.DataStore<K,V>>,java.lang. Class <K>,java.lang. Class <V>,java.lang. String ) [javac] location: class org.apache.gora.store.DataStoreFactory [javac] return DataStoreFactory.createDataStore(dataStoreClass, [javac] ^ [javac] Note: Some input files use or override a deprecated API. [javac] Note: Recompile with -Xlint:deprecation for details. [javac] Note: Some input files use unchecked or unsafe operations. [javac] Note: Recompile with -Xlint:unchecked for details. [javac] 2 errors BUILD FAILED
          Hide
          Chris A. Mattmann added a comment -

          See my comment on the gora-dev list. I don't think this is explicitly needed or desired. SNAPSHOTS should go to http://repository.apache.org and then referenced according to the "How do I do this" section of:

          http://maven.apache.org/guides/development/guide-testing-development-plugins.html

          Show
          Chris A. Mattmann added a comment - See my comment on the gora-dev list. I don't think this is explicitly needed or desired. SNAPSHOTS should go to http://repository.apache.org and then referenced according to the "How do I do this" section of: http://maven.apache.org/guides/development/guide-testing-development-plugins.html
          Hide
          Lewis John McGibbney added a comment -

          This can't be progressed with until we get the Gora 0.2-SNAPSHOT's loaded to

          http://repo1.maven.org/maven2/org/apache/gora/

          I'll work on this

          Show
          Lewis John McGibbney added a comment - This can't be progressed with until we get the Gora 0.2-SNAPSHOT's loaded to http://repo1.maven.org/maven2/org/apache/gora/ I'll work on this
          Lewis John McGibbney created issue -

            People

            • Assignee:
              Lewis John McGibbney
              Reporter:
              Lewis John McGibbney
            • Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development