[#DERBY-3354] Select from large lob table with embedded gives OutOfMemoryError

Derby

Select from large lob table with embedded gives OutOfMemoryError

Created: 28/Jan/08 07:50 PM Updated: 30/Jun/09 03:55 PM

Return to search

Component/s:

SQL

Affects Version/s:

10.3.1.4, 10.3.2.1, 10.4.1.3

Fix Version/s:

10.3.3.0, 10.4.1.3, 10.5.1.1

Time Tracking:

Not Specified

File Attachments:

File Name	Date Attached	Attached By	Size
derby-3354.diff	2008-02-12 02:03 PM	Øystein Grøvlen	0.9 kB
derby-3354_preview.diff	2008-04-02 11:41 AM	Anurag Shekhar	3 kB
derby-3354_v1.diff	2008-04-03 08:53 PM	Anurag Shekhar	5 kB
derby-3354_v2.diff	2008-04-04 08:58 AM	Anurag Shekhar	5 kB
derby-3354_v3.diff	2008-04-04 10:59 AM	Anurag Shekhar	5 kB
derby-3354_v4.diff	2008-04-04 03:20 PM	Anurag Shekhar	5 kB
LocLeak.java	2008-01-28 07:53 PM	Kathey Marsden	1 kB

Bug behavior facts:	Regression
Resolution Date:	07/Apr/08 08:50 AM

Description

« Hide

Retrieving from a large table with lobs gives an OutOfMemoryException, even if free() is explictly called on the lob. I believe this is because EmbedConnection.addLobMapping is called for every lob creation but is never cleared until commit or rollback, even if the lob is freed.

All

Comments

Work Log

Change History

Subversion Commits

Sort Order:

Repository	Revision	Date	User	Message
ASF	#644755	Fri Apr 04 16:19:19 UTC 2008	oysteing	~~DERBY-3354~~: Select from large lob table with embedded gives OutOfMemoryError Contributed by Anurag. This patch introduces a new WeakHashMap in EmbedConnection. EmbedBlob and EmbedClob objects references are stored in this map (objects as key and null as value). Adding entry to locater map is differed till the first call of getLocater. This ensures that there is entry of LOB objects in locater map if they are invoked in embedded mode. As the keys of WeakHashMap doesn't prevents the objects from being garbage collected, once the lob objects are unreferenced lob objects will be garbage collected releasing the memory. During commit/rollback or Connection.close, free is invoked on all the lob objects from WeakHashMap and the map is cleared. Modified files java/engine/org/apache/derby/impl/jdbc/EmbedConnection.java Added a new attribute lobRefrences of type WeakHashMap. Added a new method addLOBReference to make an entry in new hash map. Modified clearLOBMapping to use lobRefrences to fetch and invoke free on lob objects instead of lobHashMap. java/engine/org/apache/derby/impl/jdbc/EmbedBlob.java java/engine/org/apache/derby/impl/jdbc/EmbedClob.java Modified constructs to call connection.lobRefrences instead of conn.addLOBMapping. Modified getLocater method to check if the locater value is non zero before returning and if its zero calling conn.addLOBMapping to make entry of lob objects and getting locater value. Calling removeLOBMapping in free method. Cleanup of temporary file is already being taken care by the finalizer of LOBStreamControl so I haven't added any new cleanup code for finalizer.
				Files Changed
				MODIFY /db/derby/code/trunk/java/engine/org/apache/derby/impl/jdbc/EmbedBlob.java

Repository	Revision	Date	User	Message
ASF	#644764	Fri Apr 04 16:40:25 UTC 2008	oysteing	~~DERBY-3354~~: Select from large lob table with embedded gives OutOfMemoryError Contributed by Anurag. This patch introduces a new WeakHashMap in EmbedConnection. EmbedBlob and EmbedClob objects references are stored in this map (objects as key and null as value). Adding entry to locater map is differed till the first call of getLocater. This ensures that there is entry of LOB objects in locater map if they are invoked in embedded mode. As the keys of WeakHashMap doesn't prevents the objects from being garbage collected, once the lob objects are unreferenced lob objects will be garbage collected releasing the memory. During commit/rollback or Connection.close, free is invoked on all the lob objects from WeakHashMap and the map is cleared. Modified files java/engine/org/apache/derby/impl/jdbc/EmbedConnection.java Added a new attribute lobRefrences of type WeakHashMap. Added a new method addLOBReference to make an entry in new hash map. Modified clearLOBMapping to use lobRefrences to fetch and invoke free on lob objects instead of lobHashMap. java/engine/org/apache/derby/impl/jdbc/EmbedBlob.java java/engine/org/apache/derby/impl/jdbc/EmbedClob.java Modified constructs to call connection.lobRefrences instead of conn.addLOBMapping. Modified getLocater method to check if the locater value is non zero before returning and if its zero calling conn.addLOBMapping to make entry of lob objects and getting locater value. Calling removeLOBMapping in free method. Cleanup of temporary file is already being taken care by the finalizer of LOBStreamControl so I haven't added any new cleanup code for finalizer.
				Files Changed
				MODIFY /db/derby/code/trunk/java/engine/org/apache/derby/impl/jdbc/EmbedConnection.java MODIFY /db/derby/code/trunk/java/engine/org/apache/derby/impl/jdbc/EmbedBlob.java MODIFY /db/derby/code/trunk/java/engine/org/apache/derby/impl/jdbc/EmbedClob.java

Repository	Revision	Date	User	Message
ASF	#644916	Fri Apr 04 20:59:38 UTC 2008	oysteing	~~DERBY-3354~~: Select from large lob table with embedded gives OutOfMemoryError Contributed by Anurag. Merged to 10.4 branch with command: svn merge -r644763:644764 https://svn.apache.org/repos/asf/db/derby/code/trunk/ This patch introduces a new WeakHashMap in EmbedConnection. EmbedBlob and EmbedClob objects references are stored in this map (objects as key and null as value). Adding entry to locater map is differed till the first call of getLocater. This ensures that there is entry of LOB objects in locater map if they are invoked in embedded mode. As the keys of WeakHashMap doesn't prevents the objects from being garbage collected, once the lob objects are unreferenced lob objects will be garbage collected releasing the memory. During commit/rollback or Connection.close, free is invoked on all the lob objects from WeakHashMap and the map is cleared. Modified files java/engine/org/apache/derby/impl/jdbc/EmbedConnection.java Added a new attribute lobRefrences of type WeakHashMap. Added a new method addLOBReference to make an entry in new hash map. Modified clearLOBMapping to use lobRefrences to fetch and invoke free on lob objects instead of lobHashMap. java/engine/org/apache/derby/impl/jdbc/EmbedBlob.java java/engine/org/apache/derby/impl/jdbc/EmbedClob.java Modified constructs to call connection.lobRefrences instead of conn.addLOBMapping. Modified getLocater method to check if the locater value is non zero before returning and if its zero calling conn.addLOBMapping to make entry of lob objects and getting locater value. Calling removeLOBMapping in free method. Cleanup of temporary file is already being taken care by the finalizer of LOBStreamControl so I haven't added any new cleanup code for finalizer.
				Files Changed
				MODIFY /db/derby/code/branches/10.4/java/engine/org/apache/derby/impl/jdbc/EmbedConnection.java MODIFY /db/derby/code/branches/10.4/java/engine/org/apache/derby/impl/jdbc/EmbedBlob.java MODIFY /db/derby/code/branches/10.4/java/engine/org/apache/derby/impl/jdbc/EmbedClob.java

Repository	Revision	Date	User	Message
ASF	#652087	Tue Apr 29 20:25:00 UTC 2008	kmarsden	~~DERBY-2885~~ Cleanup of EmbedConnection.clearLOBMapping(): revision 552325 Contributed by Knut Anders Hatlen ~~DERBY-3576~~ Merge EngineBlob and EngineClob into a single interface revision 642707 Contributed by Kritian Waagan. ~~DERBY-3354~~ Select from large lob table with embedded gives OutOfMemoryError revision 644755, 644764 Contributed by Anurag Shekhar
				Files Changed
				ADD /db/derby/code/branches/10.3/java/engine/org/apache/derby/iapi/jdbc/EngineLOB.java MODIFY /db/derby/code/branches/10.3/java/engine/org/apache/derby/impl/jdbc/LOBStoredProcedure.java DEL /db/derby/code/branches/10.3/java/engine/org/apache/derby/iapi/jdbc/EngineBlob.java DEL /db/derby/code/branches/10.3/java/engine/org/apache/derby/iapi/jdbc/EngineClob.java MODIFY /db/derby/code/branches/10.3/java/engine/org/apache/derby/impl/jdbc/EmbedConnection.java MODIFY /db/derby/code/branches/10.3/java/engine/org/apache/derby/impl/jdbc/EmbedBlob.java MODIFY /db/derby/code/branches/10.3/java/engine/org/apache/derby/impl/jdbc/EmbedClob.java MODIFY /db/derby/code/branches/10.3/java/drda/org/apache/derby/impl/drda/DRDAConnThread.java