Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Minor Minor
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.21.0
    • Component/s: None
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      DBInputFormat.getSplits() does not connection.commit() after the COUNT query. This can leave an open transaction against the database which interferes with other connections to the same table.

      1. MAPREDUCE-840.patch
        0.5 kB
        Aaron Kimball

        Activity

        Hide
        Aaron Kimball added a comment -

        Attaching trivial patch for this issue. No new tests because I've only seen this issue manifest in interacting with postgresql. I've verified that with this fix in place, it works with postgresql. The TestDBJob unit test also works.

        Show
        Aaron Kimball added a comment - Attaching trivial patch for this issue. No new tests because I've only seen this issue manifest in interacting with postgresql. I've verified that with this fix in place, it works with postgresql. The TestDBJob unit test also works.
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12415893/MAPREDUCE-840.patch
        against trunk revision 801959.

        +1 @author. The patch does not contain any @author tags.

        -1 tests included. The patch doesn't appear to include any new or modified tests.
        Please justify why no new tests are needed for this patch.
        Also please list what manual steps were performed to verify this patch.

        +1 javadoc. The javadoc tool did not generate any warning messages.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 findbugs. The patch does not introduce any new Findbugs warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        +1 core tests. The patch passed core unit tests.

        -1 contrib tests. The patch failed contrib unit tests.

        Test results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-vesta.apache.org/458/testReport/
        Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-vesta.apache.org/458/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
        Checkstyle results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-vesta.apache.org/458/artifact/trunk/build/test/checkstyle-errors.html
        Console output: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-vesta.apache.org/458/console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12415893/MAPREDUCE-840.patch against trunk revision 801959. +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed core unit tests. -1 contrib tests. The patch failed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-vesta.apache.org/458/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-vesta.apache.org/458/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Checkstyle results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-vesta.apache.org/458/artifact/trunk/build/test/checkstyle-errors.html Console output: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-vesta.apache.org/458/console This message is automatically generated.
        Hide
        Enis Soztutar added a comment -

        At DBInputFormat.java, lines 359,360, the resultset and the statement are closed, which should release the resources as stated in the javadoc. Is the problem specific to the PostgreSQL driver?

        Show
        Enis Soztutar added a comment - At DBInputFormat.java, lines 359,360, the resultset and the statement are closed, which should release the resources as stated in the javadoc. Is the problem specific to the PostgreSQL driver?
        Hide
        Aaron Kimball added a comment -

        Enis,

        Closing the resultset and the statement do not commit the transaction. The postgresql server will continue to hold reader locks on all affected tables until that connection is closed, times out, etc. (Postgresql will begin a transaction for any SELECT statement; other drivers may or may not wrap reads in a transaction – JDBC doesn't mandate what happens here. Thus far this is an issue for me only with Postgresql; mysql and oracle don't seem to produce this same behavior.)

        This is about cleaning up server-side resources, at least insofar as Postgresql implements the spec. I think that it's good hygiene to assume that any driver may open a transaction that should be committed when we're done reading.

        Show
        Aaron Kimball added a comment - Enis, Closing the resultset and the statement do not commit the transaction. The postgresql server will continue to hold reader locks on all affected tables until that connection is closed, times out, etc. (Postgresql will begin a transaction for any SELECT statement; other drivers may or may not wrap reads in a transaction – JDBC doesn't mandate what happens here. Thus far this is an issue for me only with Postgresql; mysql and oracle don't seem to produce this same behavior.) This is about cleaning up server-side resources, at least insofar as Postgresql implements the spec. I think that it's good hygiene to assume that any driver may open a transaction that should be committed when we're done reading.
        Hide
        Enis Soztutar added a comment -

        Was just curious about the issue being Postgres-specific. Other than that, I'm +1.

        Show
        Enis Soztutar added a comment - Was just curious about the issue being Postgres-specific. Other than that, I'm +1.
        Hide
        Tom White added a comment -

        I've just committed this. Thanks Aaron!

        Show
        Tom White added a comment - I've just committed this. Thanks Aaron!
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Mapreduce-trunk #46 (See http://hudson.zones.apache.org/hudson/job/Hadoop-Mapreduce-trunk/46/)
        . DBInputFormat leaves open transaction. Contributed by Aaron Kimball.

        Show
        Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk #46 (See http://hudson.zones.apache.org/hudson/job/Hadoop-Mapreduce-trunk/46/ ) . DBInputFormat leaves open transaction. Contributed by Aaron Kimball.

          People

          • Assignee:
            Aaron Kimball
            Reporter:
            Aaron Kimball
          • Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development