Nutch
  1. Nutch
  2. NUTCH-1356

ParseUtil use ExecutorService instead of manually thread handling.

    Details

    • Type: Improvement Improvement
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: nutchgora, 1.6
    • Component/s: None
    • Labels:
      None
    • Patch Info:
      Patch Available

      Description

      Because ParseUtil manages it's own parser threads by creating a thread for every parse it sometimes happens that specific parsers are very expensive. For example, parsers that have threadlocal fields will initialize them for every item to be parsed.

      By simply introducing a caching ExecutorService the ParseUtil will be able to cache threads therefore parsing more efficient. See attached patch.

      1. NUTCH-1356-trunk-v2.patch
        3 kB
        Ferdy Galema
      2. NUTCH-1356-trunk.patch
        3 kB
        Ferdy Galema
      3. NUTCH-1356.patch
        4 kB
        Ferdy Galema

        Activity

        Ferdy Galema created issue -
        Ferdy Galema made changes -
        Field Original Value New Value
        Attachment NUTCH-1356.patch [ 12525835 ]
        Hide
        Markus Jelsma added a comment -

        Nice! Do you have a 1.x patch as well?

        Show
        Markus Jelsma added a comment - Nice! Do you have a 1.x patch as well?
        Hide
        Ferdy Galema added a comment -

        Sure will create patch for 1.x too. (Seems not that different).

        Show
        Ferdy Galema added a comment - Sure will create patch for 1.x too. (Seems not that different).
        Ferdy Galema made changes -
        Fix Version/s 1.6 [ 12319941 ]
        Hide
        Ferdy Galema added a comment -

        Patch for trunk.

        Show
        Ferdy Galema added a comment - Patch for trunk.
        Ferdy Galema made changes -
        Attachment NUTCH-1356-trunk.patch [ 12525839 ]
        Hide
        Markus Jelsma added a comment -

        Guava is not listed as dependency on trunk.

        <dependency org="com.google.guava" name="guava" rev="12.0" />
        

        Great work!

        Show
        Markus Jelsma added a comment - Guava is not listed as dependency on trunk. <dependency org= "com.google.guava" name= "guava" rev= "12.0" /> Great work!
        Hide
        Ferdy Galema added a comment -

        It was working though, I guess that is because of a transitive dependancy. Anyway it's best to declare it as a direct dependancy too. Patch v2 does this. (11.0.2 --> the same as the already present jar).

        Show
        Ferdy Galema added a comment - It was working though, I guess that is because of a transitive dependancy. Anyway it's best to declare it as a direct dependancy too. Patch v2 does this. (11.0.2 --> the same as the already present jar).
        Ferdy Galema made changes -
        Attachment NUTCH-1356-trunk-v2.patch [ 12525847 ]
        Hide
        Ferdy Galema added a comment -

        committed at nutchgora

        Show
        Ferdy Galema added a comment - committed at nutchgora
        Hide
        Hudson added a comment -

        Integrated in Nutch-nutchgora #248 (See https://builds.apache.org/job/Nutch-nutchgora/248/)
        NUTCH-1356 ParseUtil use ExecutorService instead of manually thread handling. (Revision 1335065)

        Result = SUCCESS
        ferdy :
        Files :

        • /nutch/branches/nutchgora/CHANGES.txt
        • /nutch/branches/nutchgora/src/java/org/apache/nutch/parse/ParseUtil.java
        Show
        Hudson added a comment - Integrated in Nutch-nutchgora #248 (See https://builds.apache.org/job/Nutch-nutchgora/248/ ) NUTCH-1356 ParseUtil use ExecutorService instead of manually thread handling. (Revision 1335065) Result = SUCCESS ferdy : Files : /nutch/branches/nutchgora/CHANGES.txt /nutch/branches/nutchgora/src/java/org/apache/nutch/parse/ParseUtil.java
        Hide
        Markus Jelsma added a comment -

        I came across an NPE when inspecting the logs. We use the patch on 1.5 and i can't remember i've seen this error before. Is it related to this patch?

        It is, unfortunately, not reproducible with the parserchecker. I should also note that this fetcher is having a very tough job (all others finished a while ago) so something is going on, perhaps eating up memory due to a leak somewhere (Tika?).

        2012-05-30 07:03:39,560 WARN org.apache.nutch.parse.ParseUtil: Error parsing http://www.vvvroomshoopseboys.nl/e7-programma.html with org.apache.nutch.parse.tika.TikaParser@2bd2e84e
        java.util.concurrent.ExecutionException: java.lang.NullPointerException
        	at java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:232)
        	at java.util.concurrent.FutureTask.get(FutureTask.java:91)
        	at org.apache.nutch.parse.ParseUtil.runParser(ParseUtil.java:162)
        	at org.apache.nutch.parse.ParseUtil.parse(ParseUtil.java:93)
        
        Show
        Markus Jelsma added a comment - I came across an NPE when inspecting the logs. We use the patch on 1.5 and i can't remember i've seen this error before. Is it related to this patch? It is, unfortunately, not reproducible with the parserchecker. I should also note that this fetcher is having a very tough job (all others finished a while ago) so something is going on, perhaps eating up memory due to a leak somewhere (Tika?). 2012-05-30 07:03:39,560 WARN org.apache.nutch.parse.ParseUtil: Error parsing http: //www.vvvroomshoopseboys.nl/e7-programma.html with org.apache.nutch.parse.tika.TikaParser@2bd2e84e java.util.concurrent.ExecutionException: java.lang.NullPointerException at java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:232) at java.util.concurrent.FutureTask.get(FutureTask.java:91) at org.apache.nutch.parse.ParseUtil.runParser(ParseUtil.java:162) at org.apache.nutch.parse.ParseUtil.parse(ParseUtil.java:93)
        Hide
        Markus Jelsma added a comment -

        I kept checking the log and found some more of the exceptions above and several similar:

        java.util.concurrent.TimeoutException
        	at java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:228)
        	at java.util.concurrent.FutureTask.get(FutureTask.java:91)
        	at org.apache.nutch.parse.ParseUtil.runParser(ParseUtil.java:162)
        	at org.apache.nutch.parse.ParseUtil.parse(ParseUtil.java:93)
        

        More interesting is that i see additional stack traces elsewhere indicating a problem with a parse filter:

        Caused by: java.lang.NullPointerException
        	at org.apache.xerces.dom.ParentNode.nodeListItem(Unknown Source)
        	at org.apache.xerces.dom.ParentNode.item(Unknown Source)
        	at org.apache.nutch.util.NodeWalker.nextNode(NodeWalker.java:75)
        	at org.apache.nutch.parse.headings.HeadingsParseFilter.getElement(HeadingsParseFilter.java:79)
        	at org.apache.nutch.parse.headings.HeadingsParseFilter.filter(HeadingsParseFilter.java:48)
        	at org.apache.nutch.parse.HtmlParseFilters.filter(HtmlParseFilters.java:98)
        	at org.apache.nutch.parse.tika.TikaParser.getParse(TikaParser.java:208)
        	at org.apache.nutch.parse.ParseCallable.call(ParseCallable.java:35)
        	at org.apache.nutch.parse.ParseCallable.call(ParseCallable.java:24)
        	at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
        	at java.util.concurrent.FutureTask.run(FutureTask.java:138)
        	at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
        	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
        	at java.lang.Thread.run(Thread.java:662)
        
        Show
        Markus Jelsma added a comment - I kept checking the log and found some more of the exceptions above and several similar: java.util.concurrent.TimeoutException at java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:228) at java.util.concurrent.FutureTask.get(FutureTask.java:91) at org.apache.nutch.parse.ParseUtil.runParser(ParseUtil.java:162) at org.apache.nutch.parse.ParseUtil.parse(ParseUtil.java:93) More interesting is that i see additional stack traces elsewhere indicating a problem with a parse filter: Caused by: java.lang.NullPointerException at org.apache.xerces.dom.ParentNode.nodeListItem(Unknown Source) at org.apache.xerces.dom.ParentNode.item(Unknown Source) at org.apache.nutch.util.NodeWalker.nextNode(NodeWalker.java:75) at org.apache.nutch.parse.headings.HeadingsParseFilter.getElement(HeadingsParseFilter.java:79) at org.apache.nutch.parse.headings.HeadingsParseFilter.filter(HeadingsParseFilter.java:48) at org.apache.nutch.parse.HtmlParseFilters.filter(HtmlParseFilters.java:98) at org.apache.nutch.parse.tika.TikaParser.getParse(TikaParser.java:208) at org.apache.nutch.parse.ParseCallable.call(ParseCallable.java:35) at org.apache.nutch.parse.ParseCallable.call(ParseCallable.java:24) at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303) at java.util.concurrent.FutureTask.run(FutureTask.java:138) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang. Thread .run( Thread .java:662)
        Hide
        Ferdy Galema added a comment -

        I find it difficult to believe those exceptions are caused by this patch. It does not change the way exceptions/timeouts are handled, it only makes sure parser threads are reused.

        It seems you are suffering from two types of (unrelated) exceptions. The first is ExecutionException. This is caused whenever the execution inside the FutureTask.get() throws an exception that is not catched anywere but the FutureTask.get() itself. In your case this seems to be a NPE during the parse of the html page. Might be a bug but then again it is strange that it is not reproducible with the ParserChecker. (You sure about this?)

        The second is TimeoutException, caused whenever the FutureTask.get() cannot be completed within the specified timeout. The tricky part is that single urls might be perfectly able to complete within the timeout, but when there is a heavy concurrent load (a lot of semi-expensive parses) the parser load might stack up and cause many parses to timeout. This can be the case with parsing during fetch. But when using a separate parserjob this can also happen because Parser implementation do not necessarily have to respond to a thread interrupt. (Which is fired away with the task.cancel(true) call). If a parser does not check the Thread.interrupted state at regular intervals, it will just continue to run and eat up resources. I find it very helpful to debug stalling fetchers/parsers with the lazy men's profiler: kill -QUIT <process_id>. This will dump stacktraces, sometimes exposing the fact that hundreds of parser threads are still active in the background. (Of course many of them already timed out a long time ago).

        Show
        Ferdy Galema added a comment - I find it difficult to believe those exceptions are caused by this patch. It does not change the way exceptions/timeouts are handled, it only makes sure parser threads are reused. It seems you are suffering from two types of (unrelated) exceptions. The first is ExecutionException. This is caused whenever the execution inside the FutureTask.get() throws an exception that is not catched anywere but the FutureTask.get() itself. In your case this seems to be a NPE during the parse of the html page. Might be a bug but then again it is strange that it is not reproducible with the ParserChecker. (You sure about this?) The second is TimeoutException, caused whenever the FutureTask.get() cannot be completed within the specified timeout. The tricky part is that single urls might be perfectly able to complete within the timeout, but when there is a heavy concurrent load (a lot of semi-expensive parses) the parser load might stack up and cause many parses to timeout. This can be the case with parsing during fetch. But when using a separate parserjob this can also happen because Parser implementation do not necessarily have to respond to a thread interrupt. (Which is fired away with the task.cancel(true) call). If a parser does not check the Thread.interrupted state at regular intervals, it will just continue to run and eat up resources. I find it very helpful to debug stalling fetchers/parsers with the lazy men's profiler: kill -QUIT <process_id>. This will dump stacktraces, sometimes exposing the fact that hundreds of parser threads are still active in the background. (Of course many of them already timed out a long time ago).
        Hide
        Markus Jelsma added a comment -

        Thanks for clearing things up.

        These errors are indeed not reproducable. I collected about 12 of these error throwing URL's and manually fetched them again with success, the parsechecker also processes them without error.

        The parser threads you refer to, is that a known problem? Can we solve it? Once in a while a stalling parser eats up all remaining memory and dies.

        Show
        Markus Jelsma added a comment - Thanks for clearing things up. These errors are indeed not reproducable. I collected about 12 of these error throwing URL's and manually fetched them again with success, the parsechecker also processes them without error. The parser threads you refer to, is that a known problem? Can we solve it? Once in a while a stalling parser eats up all remaining memory and dies.
        Hide
        Markus Jelsma added a comment -

        Committed for 1.6 in rev. 1349230.
        Thanks Ferdy.

        Show
        Markus Jelsma added a comment - Committed for 1.6 in rev. 1349230. Thanks Ferdy.
        Markus Jelsma made changes -
        Status Open [ 1 ] Resolved [ 5 ]
        Resolution Fixed [ 1 ]
        Hide
        Ferdy Galema added a comment -

        Thanks.

        "The parser threads you refer to, is that a known problem? Can we solve it?"
        To solve it correctly every parser should check the interrupted state at regular intervals. This is pretty huge task considering the amount of parsers. For now it is something to keep in mind. I'll create an issue for reference.

        Show
        Ferdy Galema added a comment - Thanks. "The parser threads you refer to, is that a known problem? Can we solve it?" To solve it correctly every parser should check the interrupted state at regular intervals. This is pretty huge task considering the amount of parsers. For now it is something to keep in mind. I'll create an issue for reference.
        Hide
        Hudson added a comment -

        Integrated in nutch-trunk-maven #310 (See https://builds.apache.org/job/nutch-trunk-maven/310/)
        NUTCH-1356 ParseUtil use ExecutorService instead of manually thread handling (Revision 1349230)

        Result = SUCCESS
        markus :
        Files :

        • /nutch/trunk/CHANGES.txt
        • /nutch/trunk/ivy/ivy.xml
        • /nutch/trunk/src/java/org/apache/nutch/parse/ParseUtil.java
        Show
        Hudson added a comment - Integrated in nutch-trunk-maven #310 (See https://builds.apache.org/job/nutch-trunk-maven/310/ ) NUTCH-1356 ParseUtil use ExecutorService instead of manually thread handling (Revision 1349230) Result = SUCCESS markus : Files : /nutch/trunk/CHANGES.txt /nutch/trunk/ivy/ivy.xml /nutch/trunk/src/java/org/apache/nutch/parse/ParseUtil.java
        Hide
        Hudson added a comment -

        Integrated in Nutch-trunk #1869 (See https://builds.apache.org/job/Nutch-trunk/1869/)
        NUTCH-1356 ParseUtil use ExecutorService instead of manually thread handling (Revision 1349230)

        Result = SUCCESS
        markus : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1349230
        Files :

        • /nutch/trunk/CHANGES.txt
        • /nutch/trunk/ivy/ivy.xml
        • /nutch/trunk/src/java/org/apache/nutch/parse/ParseUtil.java
        Show
        Hudson added a comment - Integrated in Nutch-trunk #1869 (See https://builds.apache.org/job/Nutch-trunk/1869/ ) NUTCH-1356 ParseUtil use ExecutorService instead of manually thread handling (Revision 1349230) Result = SUCCESS markus : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1349230 Files : /nutch/trunk/CHANGES.txt /nutch/trunk/ivy/ivy.xml /nutch/trunk/src/java/org/apache/nutch/parse/ParseUtil.java
        Lewis John McGibbney made changes -
        Status Resolved [ 5 ] Closed [ 6 ]

          People

          • Assignee:
            Unassigned
            Reporter:
            Ferdy Galema
          • Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development