Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Critical Critical
    • Resolution: Fixed
    • Affects Version/s: nutchgora
    • Fix Version/s: nutchgora
    • Component/s: fetcher, generator, injector, storage
    • Labels:
      None
    • Environment:

      Ubuntu release 11.04 (natty)
      Kernerl Linux 2.6.38-10-generic
      GNOME 2.32.1

      Description

      The following tests fail when running ant test on trunk 2.0

          [junit] Running org.apache.nutch.api.TestAPI
          [junit] Tests run: 4, Failures: 1, Errors: 0, Time elapsed: 11.028 sec
          [junit] Test org.apache.nutch.api.TestAPI FAILED
          [junit] Running org.apache.nutch.crawl.TestGenerator
          [junit] Tests run: 4, Failures: 0, Errors: 4, Time elapsed: 0.478 sec
          [junit] Test org.apache.nutch.crawl.TestGenerator FAILED
          [junit] Running org.apache.nutch.crawl.TestInjector
          [junit] Tests run: 1, Failures: 0, Errors: 1, Time elapsed: 0.474 sec
          [junit] Test org.apache.nutch.crawl.TestInjector FAILED
          [junit] Running org.apache.nutch.fetcher.TestFetcher
          [junit] Tests run: 2, Failures: 0, Errors: 2, Time elapsed: 0.526 sec
          [junit] Test org.apache.nutch.fetcher.TestFetcher FAILED
          [junit] Running org.apache.nutch.storage.TestGoraStorage
          [junit] Tests run: 2, Failures: 0, Errors: 2, Time elapsed: 0.468 sec
          [junit] Test org.apache.nutch.storage.TestGoraStorage FAILED
      

        Issue Links

        There are no Sub-Tasks for this issue.

          Activity

          Hide
          Lewis John McGibbney added a comment -

          Over a number of issues this seems to have been phased out/addressed as testing has been stable for some time. Thanks guys.

          Show
          Lewis John McGibbney added a comment - Over a number of issues this seems to have been phased out/addressed as testing has been stable for some time. Thanks guys.
          Hide
          Ferdy Galema added a comment -

          Yes this one should be closed.

          Show
          Ferdy Galema added a comment - Yes this one should be closed.
          Hide
          Julien Nioche added a comment -

          The tests for nutchgora seem to work fine now. Close?

          Show
          Julien Nioche added a comment - The tests for nutchgora seem to work fine now. Close?
          Hide
          Lewis John McGibbney added a comment -

          Set and classify

          Show
          Lewis John McGibbney added a comment - Set and classify
          Hide
          Ferdy Galema added a comment -

          Everything works fine except for org.apache.nutch.api.TestAPI. That one still fails, sometimes. When I just ran these tests with "ant test" they all worked perfectly fine, but the runs after that simply fail. Cleaning the project with "ant clean" doesn't help. See corresponding mailing list discussion in link above.

          I have not yet looked into this test thouroughly, because it is part of Nutch that I am completely unfamiliar with. (The NutchServer API). I think it is best we close this issue, and start a new one that will deal with the this API and the test.

          Show
          Ferdy Galema added a comment - Everything works fine except for org.apache.nutch.api.TestAPI. That one still fails, sometimes. When I just ran these tests with "ant test" they all worked perfectly fine, but the runs after that simply fail. Cleaning the project with "ant clean" doesn't help. See corresponding mailing list discussion in link above. I have not yet looked into this test thouroughly, because it is part of Nutch that I am completely unfamiliar with. (The NutchServer API). I think it is best we close this issue, and start a new one that will deal with the this API and the test.
          Hide
          Lewis John McGibbney added a comment -

          Hi Ferdy. Have you noticed anything dodgy with this?

          Show
          Lewis John McGibbney added a comment - Hi Ferdy. Have you noticed anything dodgy with this?
          Hide
          Lewis John McGibbney added a comment -

          Hi Ferdy. There has been almost no problems within the CI testing environment for a number of weeks/months. Any failures seem to have been down to the project building on Ubuntu slaves as oppose to Solaris slaves, the failures are a result of incorrect envars being specified. I've added some more functionality to the nutchgora build characteristics e.g. Publish JUnit test result report and publish Javadoc. So as agreed we will keep an eye on this.

          Show
          Lewis John McGibbney added a comment - Hi Ferdy. There has been almost no problems within the CI testing environment for a number of weeks/months. Any failures seem to have been down to the project building on Ubuntu slaves as oppose to Solaris slaves, the failures are a result of incorrect envars being specified. I've added some more functionality to the nutchgora build characteristics e.g. Publish JUnit test result report and publish Javadoc. So as agreed we will keep an eye on this.
          Hide
          Lewis John McGibbney added a comment -

          Reopening this issue as per our concerns.

          For the record, the Jenkins build area has been cleaned up and we now only maintain 3 builds; trunk, Nutchgora and a maven trunk.

          Show
          Lewis John McGibbney added a comment - Reopening this issue as per our concerns. For the record, the Jenkins build area has been cleaned up and we now only maintain 3 builds; trunk, Nutchgora and a maven trunk.
          Hide
          Ferdy Galema added a comment -
          Show
          Ferdy Galema added a comment - TestAPI is troublesome: http://www.mail-archive.com/dev@nutch.apache.org/msg05417.html
          Hide
          Lewis John McGibbney added a comment -

          As of the recent NUTCH-1135 commit, this summary is being closed out.

          Show
          Lewis John McGibbney added a comment - As of the recent NUTCH-1135 commit, this summary is being closed out.
          Hide
          Ferdy Galema added a comment -

          My bad it was a local issue indeed.

          Show
          Ferdy Galema added a comment - My bad it was a local issue indeed.
          Hide
          Lewis John McGibbney added a comment -

          Hi Ferdy

          copy-generated-lib:
          
          test:
               [echo] Testing plugin: parse-tika
              [junit] Running org.apache.nutch.parse.tika.TestMSWordParser
              [junit] Tests run: 2, Failures: 0, Errors: 0, Time elapsed: 4.349 sec
              [junit] Running org.apache.nutch.parse.tika.TestOOParser
              [junit] Tests run: 1, Failures: 0, Errors: 0, Time elapsed: 2.409 sec
              [junit] Running org.apache.nutch.parse.tika.TestPdfParser
              [junit] Tests run: 1, Failures: 0, Errors: 0, Time elapsed: 3.674 sec
              [junit] Running org.apache.nutch.parse.tika.TestRSSParser
              [junit] Tests run: 1, Failures: 0, Errors: 0, Time elapsed: 2.698 sec
              [junit] Running org.apache.nutch.parse.tika.TestRTFParser
              [junit] Tests run: 1, Failures: 0, Errors: 0, Time elapsed: 0.013 sec
          
          init:
          
          init-plugin:
          
          deps-jar:
          ...
          BUILD SUCCESSFUL
          Total time: 2 minutes 20 seconds
          

          Ti make sure I checkedout our most recent nutchgora and applied your interim testGoraStorage patch. Is there maybe some config that you have changed at your end?

          Show
          Lewis John McGibbney added a comment - Hi Ferdy copy-generated-lib: test: [echo] Testing plugin: parse-tika [junit] Running org.apache.nutch.parse.tika.TestMSWordParser [junit] Tests run: 2, Failures: 0, Errors: 0, Time elapsed: 4.349 sec [junit] Running org.apache.nutch.parse.tika.TestOOParser [junit] Tests run: 1, Failures: 0, Errors: 0, Time elapsed: 2.409 sec [junit] Running org.apache.nutch.parse.tika.TestPdfParser [junit] Tests run: 1, Failures: 0, Errors: 0, Time elapsed: 3.674 sec [junit] Running org.apache.nutch.parse.tika.TestRSSParser [junit] Tests run: 1, Failures: 0, Errors: 0, Time elapsed: 2.698 sec [junit] Running org.apache.nutch.parse.tika.TestRTFParser [junit] Tests run: 1, Failures: 0, Errors: 0, Time elapsed: 0.013 sec init: init-plugin: deps-jar: ... BUILD SUCCESSFUL Total time: 2 minutes 20 seconds Ti make sure I checkedout our most recent nutchgora and applied your interim testGoraStorage patch. Is there maybe some config that you have changed at your end?
          Hide
          Ferdy Galema added a comment -

          Are you aware of the fact that TestMSWordParser currently fails too? Or am I missing something?
          [echo] Testing plugin: parse-tika
          [junit] Running org.apache.nutch.parse.tika.TestMSWordParser
          [junit] Tests run: 2, Failures: 2, Errors: 0, Time elapsed: 6.6 sec
          [junit] Test org.apache.nutch.parse.tika.TestMSWordParser FAILED

          If it is broken, could you make a subtask?

          Show
          Ferdy Galema added a comment - Are you aware of the fact that TestMSWordParser currently fails too? Or am I missing something? [echo] Testing plugin: parse-tika [junit] Running org.apache.nutch.parse.tika.TestMSWordParser [junit] Tests run: 2, Failures: 2, Errors: 0, Time elapsed: 6.6 sec [junit] Test org.apache.nutch.parse.tika.TestMSWordParser FAILED If it is broken, could you make a subtask?
          Hide
          Lewis John McGibbney added a comment -

          Thanks Ferdy. It was also my initial thoughts that this was maybe too simplistic a fix! If however you look here [1] you will see that Dogacan changed the import namespaces for the Gora imports. It would seem that over time we forgot to do this with hard-coded imports in Injector, Generator and Fetcher tests.

          Are there any objections to committing this as an interm fix before concentrating on NUTCH-896 ?

          [1] http://svn.apache.org/viewvc/nutch/branches/nutchgora/src/test/org/apache/nutch/storage/TestGoraStorage.java?view=log

          Show
          Lewis John McGibbney added a comment - Thanks Ferdy. It was also my initial thoughts that this was maybe too simplistic a fix! If however you look here [1] you will see that Dogacan changed the import namespaces for the Gora imports. It would seem that over time we forgot to do this with hard-coded imports in Injector, Generator and Fetcher tests. Are there any objections to committing this as an interm fix before concentrating on NUTCH-896 ? [1] http://svn.apache.org/viewvc/nutch/branches/nutchgora/src/test/org/apache/nutch/storage/TestGoraStorage.java?view=log
          Hide
          Ferdy Galema added a comment -

          It seems like your patch is fine, at least as a temporary solution. Tests run fine. (Please see my notes for NUTCH-1135 in the corresponding issue.)

          I see NUTCH-896 as a rather separate issue. Sure it would be nice have a configurable backend in tests, but as sql is currently the default backend (also for building) I see it as no problem to have it hard-coded in the tests for now.

          To summarize: +1

          Show
          Ferdy Galema added a comment - It seems like your patch is fine, at least as a temporary solution. Tests run fine. (Please see my notes for NUTCH-1135 in the corresponding issue.) I see NUTCH-896 as a rather separate issue. Sure it would be nice have a configurable backend in tests, but as sql is currently the default backend (also for building) I see it as no problem to have it hard-coded in the tests for now. To summarize: +1
          Hide
          Lewis John McGibbney added a comment -

          to update on this issue. All tests as above e.g. all that extend AbstractNutchTest now pass successfully. The daemon TestGoraStorage is still giving us bother, and please bear in mind that none of this takes into consideration NUTCH-896. Is the patch I submitted for these tests deemed appropriate as a temporary fix? Or should my efforts be concentrated towards NUTCH-896 ? Thanks

          Show
          Lewis John McGibbney added a comment - to update on this issue. All tests as above e.g. all that extend AbstractNutchTest now pass successfully. The daemon TestGoraStorage is still giving us bother, and please bear in mind that none of this takes into consideration NUTCH-896 . Is the patch I submitted for these tests deemed appropriate as a temporary fix? Or should my efforts be concentrated towards NUTCH-896 ? Thanks
          Hide
          Lewis John McGibbney added a comment -

          I have marked this as critical now as it is the 'only' thing preventing us from finally achieving a stable nightly build for the nutchgora branch. In an attempt to get this moving, I'm going to create subtasks for each test, this way we will be able to track reasonable progress on each potential blocker.

          Show
          Lewis John McGibbney added a comment - I have marked this as critical now as it is the 'only' thing preventing us from finally achieving a stable nightly build for the nutchgora branch. In an attempt to get this moving, I'm going to create subtasks for each test, this way we will be able to track reasonable progress on each potential blocker.

            People

            • Assignee:
              Lewis John McGibbney
              Reporter:
              Lewis John McGibbney
            • Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development