[#SOLR-310] Deleting pending handling can cause OOM

Solr

Deleting pending handling can cause OOM

Created: 19/Jul/07 12:15 AM Updated: 04/Sep/07 11:06 PM

Return to search

Component/s:

None

Affects Version/s:

None

Fix Version/s:

1.3

Time Tracking:

Not Specified

File Attachments:

	File Name	Date Attached	Attached By	Size
	maxPendingDeletes.patch	2007-08-30 05:54 AM	Mike Klaas	4 kB

Resolution Date:

04/Sep/07 11:06 PM

Description

« Hide

> Mike:
>> Could happen when doDeleting the pending docs too. James: try
>> sending commit every 500k docs or so.

Yonik:
> Hmmm, right... some of the memory usage will be related to the treemap
> keeping track of deleted items, and some of it will be related to the
> reader itself (the term index, and the norms).

> Perhaps we should have some sort of limit on the number of deletions
> we cache so people don't have to worry about that part.

Good idea. Perhaps a solrconfig.xml setting, <maxPendingDocs>? Default to 100k?

All

Comments

Work Log

Change History

Subversion Commits

Sort Order:

Mike Klaas made changes - 30/Aug/07 05:53 AM

Field	Original Value	New Value
Status	Open [ 1 ]	In Progress [ 3 ]

[ Permlink | « Hide ]

Mike Klaas added a comment - 30/Aug/07 05:54 AM

q&d patch that implements the above. Called it maxPendingDeletes, since the complexity/memory being bounded is determined by that, not pending docs.

Mike Klaas made changes - 30/Aug/07 05:54 AM

Attachment

maxPendingDeletes.patch [ 12364813 ]

Repository	Revision	Date	User	Message
ASF	#572831	Tue Sep 04 23:05:39 UTC 2007	klaas	~~SOLR-310~~: bound pending deletes
				Files Changed
				MODIFY /lucene/solr/trunk/src/test/org/apache/solr/update/AutoCommitTest.java MODIFY /lucene/solr/trunk/example/solr/conf/solrconfig.xml MODIFY /lucene/solr/trunk/CHANGES.txt MODIFY /lucene/solr/trunk/src/java/org/apache/solr/update/DirectUpdateHandler2.java

[ Permlink | « Hide ]

Mike Klaas added a comment - 04/Sep/07 11:06 PM

committed in r572831

Mike Klaas made changes - 04/Sep/07 11:06 PM

Resolution		Fixed [ 1 ]
Status	In Progress [ 3 ]	Resolved [ 5 ]