Details
-
Type:
Bug
-
Status: Closed
-
Priority:
Minor
-
Resolution: Fixed
-
Affects Version/s: None
-
Fix Version/s: 0.21.0
-
Component/s: None
-
Labels:None
-
Hadoop Flags:Reviewed
Description
DBInputFormat.getSplits() does not connection.commit() after the COUNT query. This can leave an open transaction against the database which interferes with other connections to the same table.
Activity
- All
- Comments
- Work Log
- History
- Activity
- Transitions
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12415893/MAPREDUCE-840.patch
against trunk revision 801959.
+1 @author. The patch does not contain any @author tags.
-1 tests included. The patch doesn't appear to include any new or modified tests.
Please justify why no new tests are needed for this patch.
Also please list what manual steps were performed to verify this patch.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
+1 findbugs. The patch does not introduce any new Findbugs warnings.
+1 release audit. The applied patch does not increase the total number of release audit warnings.
+1 core tests. The patch passed core unit tests.
-1 contrib tests. The patch failed contrib unit tests.
Test results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-vesta.apache.org/458/testReport/
Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-vesta.apache.org/458/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-vesta.apache.org/458/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-vesta.apache.org/458/console
This message is automatically generated.
At DBInputFormat.java, lines 359,360, the resultset and the statement are closed, which should release the resources as stated in the javadoc. Is the problem specific to the PostgreSQL driver?
Enis,
Closing the resultset and the statement do not commit the transaction. The postgresql server will continue to hold reader locks on all affected tables until that connection is closed, times out, etc. (Postgresql will begin a transaction for any SELECT statement; other drivers may or may not wrap reads in a transaction – JDBC doesn't mandate what happens here. Thus far this is an issue for me only with Postgresql; mysql and oracle don't seem to produce this same behavior.)
This is about cleaning up server-side resources, at least insofar as Postgresql implements the spec. I think that it's good hygiene to assume that any driver may open a transaction that should be committed when we're done reading.
Was just curious about the issue being Postgres-specific. Other than that, I'm +1.
I've just committed this. Thanks Aaron!
Integrated in Hadoop-Mapreduce-trunk #46 (See http://hudson.zones.apache.org/hudson/job/Hadoop-Mapreduce-trunk/46/)
. DBInputFormat leaves open transaction. Contributed by Aaron Kimball.
Attaching trivial patch for this issue. No new tests because I've only seen this issue manifest in interacting with postgresql. I've verified that with this fix in place, it works with postgresql. The TestDBJob unit test also works.