[GORA-210] thread safety: java.util.ConcurrentModificationException

• Type: Bug
• Status: Closed
• Priority: Critical
• Resolution: Fixed
• Affects Version/s: 0.2
• Fix Version/s: 0.3
• Component/s: gora-cassandra
• Labels: patch

Environment

nutch 2.1 / cassandra 1.2.1 / gora-cassandra 0.2 / gora-core 0.2.1
running fetch with parse=true
fetcher.threads.per.queue=2 / about 1 Exception per 100k URLs fetched

nutch on a 16 core AMD Opteron 2GHz.
Cassandra on 8 core Intel Xeon 3.3 GHz

• Assignee: Roland von Herget
• Reporter: Roland von Herget
• Created: 26/Feb/13 10:25
• Updated: 03/Mar/22 14:27
• Resolved: 12/Mar/13 11:16

Description

This is the result of debugging one of my issues described in NUTCH-1534.
I think there is a wrong assumpation about thread safety of LinkedHashMap, it is not enough to not iterate over the buffer (which is a LinkedHashMap).
My patch fixes this error for me:

java.util.ConcurrentModificationException
at java.util.LinkedHashMap$LinkedHashIterator.nextEntry(LinkedHashMap.java:394)
at java.util.LinkedHashMap$KeyIterator.next(LinkedHashMap.java:405)
at java.util.AbstractCollection.toArray(AbstractCollection.java:141)
at org.apache.gora.cassandra.store.CassandraStore.flush(CassandraStore.java:200)
at org.apache.gora.mapreduce.GoraRecordWriter.write(GoraRecordWriter.java:65)
at org.apache.hadoop.mapred.ReduceTask$NewTrackingRecordWriter.write(ReduceTask.java:587)
at org.apache.hadoop.mapreduce.TaskInputOutputContext.write(TaskInputOutputContext.java:80)
at org.apache.nutch.fetcher.FetcherReducer$FetcherThread.output(FetcherReducer.java:664)
at org.apache.nutch.fetcher.FetcherReducer$FetcherThread.run(FetcherReducer.java:534)

It may not be perfect from a performance point of view...

Issue Links

• is depended upon by: NUTCH-1534 cassandra/hector exception: InvalidRequestException(why:column name must not be empty)