Details

    • Type: Improvement Improvement
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.21.0
    • Component/s: None
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      DBInputFormat's DBRecordReader executes the user's SQL query in the constructor. If the query is long-running, this can cause task timeout. The user is unable to spawn a background thread (e.g., in a MapRunnable) to inform Hadoop of on-going progress.

      1. MAPREDUCE-875.patch
        1 kB
        Aaron Kimball
      2. MAPREDUCE-875.2.patch
        1 kB
        Aaron Kimball

        Activity

        Hide
        Aaron Kimball added a comment -

        Trivial patch that makes the query execute only when nextKeyValue() is actually run for the first time. This is inside the control of the MapRunnable.

        Show
        Aaron Kimball added a comment - Trivial patch that makes the query execute only when nextKeyValue() is actually run for the first time. This is inside the control of the MapRunnable.
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12416623/MAPREDUCE-875.patch
        against trunk revision 804284.

        +1 @author. The patch does not contain any @author tags.

        -1 tests included. The patch doesn't appear to include any new or modified tests.
        Please justify why no new tests are needed for this patch.
        Also please list what manual steps were performed to verify this patch.

        +1 javadoc. The javadoc tool did not generate any warning messages.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 findbugs. The patch does not introduce any new Findbugs warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        -1 core tests. The patch failed core unit tests.

        -1 contrib tests. The patch failed contrib unit tests.

        Test results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-vesta.apache.org/477/testReport/
        Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-vesta.apache.org/477/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
        Checkstyle results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-vesta.apache.org/477/artifact/trunk/build/test/checkstyle-errors.html
        Console output: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-vesta.apache.org/477/console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12416623/MAPREDUCE-875.patch against trunk revision 804284. +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed core unit tests. -1 contrib tests. The patch failed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-vesta.apache.org/477/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-vesta.apache.org/477/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Checkstyle results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-vesta.apache.org/477/artifact/trunk/build/test/checkstyle-errors.html Console output: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-vesta.apache.org/477/console This message is automatically generated.
        Hide
        Aaron Kimball added a comment -

        The failing Sqoop tests claim that it's failing because it can't find Avro. Not sure why this is happening – Sqoop doesn't make use of Avro anywhere. Recycling patch status in case this was transient. If not, do I have to put some more random libraries in ivy.xml?

        According to git-blame, this was added to the root ivy.xml earlier that day:

        9e58f6fc (Sharad Agarwal     2009-08-14 05:10:40 +0000 276)     <dependency org="org.apache.hadoop"
        9e58f6fc (Sharad Agarwal     2009-08-14 05:10:40 +0000 277)       name="avro"
        9e58f6fc (Sharad Agarwal     2009-08-14 05:10:40 +0000 278)       rev="1.0.0"
        9e58f6fc (Sharad Agarwal     2009-08-14 05:10:40 +0000 279)       conf="common->default"/>
        9e58f6fc (Sharad Agarwal     2009-08-14 05:10:40 +0000 280)     <dependency org="org.codehaus.jackso
        9e58f6fc (Sharad Agarwal     2009-08-14 05:10:40 +0000 281)       name="jackson-mapper-asl"
        9e58f6fc (Sharad Agarwal     2009-08-14 05:10:40 +0000 282)       rev="1.0.1"
        9e58f6fc (Sharad Agarwal     2009-08-14 05:10:40 +0000 283)       conf="common->default"/>
        9e58f6fc (Sharad Agarwal     2009-08-14 05:10:40 +0000 284)     <dependency org="com.thoughtworks.pa
        9e58f6fc (Sharad Agarwal     2009-08-14 05:10:40 +0000 285)       name="paranamer"
        9e58f6fc (Sharad Agarwal     2009-08-14 05:10:40 +0000 286)       rev="1.5"
        9e58f6fc (Sharad Agarwal     2009-08-14 05:10:40 +0000 287)       conf="common->default"/>
        
        Show
        Aaron Kimball added a comment - The failing Sqoop tests claim that it's failing because it can't find Avro. Not sure why this is happening – Sqoop doesn't make use of Avro anywhere. Recycling patch status in case this was transient. If not, do I have to put some more random libraries in ivy.xml? According to git-blame , this was added to the root ivy.xml earlier that day: 9e58f6fc (Sharad Agarwal 2009-08-14 05:10:40 +0000 276) <dependency org= "org.apache.hadoop" 9e58f6fc (Sharad Agarwal 2009-08-14 05:10:40 +0000 277) name= "avro" 9e58f6fc (Sharad Agarwal 2009-08-14 05:10:40 +0000 278) rev= "1.0.0" 9e58f6fc (Sharad Agarwal 2009-08-14 05:10:40 +0000 279) conf= "common-> default " /> 9e58f6fc (Sharad Agarwal 2009-08-14 05:10:40 +0000 280) <dependency org="org.codehaus.jackso 9e58f6fc (Sharad Agarwal 2009-08-14 05:10:40 +0000 281) name= "jackson-mapper-asl" 9e58f6fc (Sharad Agarwal 2009-08-14 05:10:40 +0000 282) rev= "1.0.1" 9e58f6fc (Sharad Agarwal 2009-08-14 05:10:40 +0000 283) conf= "common-> default " /> 9e58f6fc (Sharad Agarwal 2009-08-14 05:10:40 +0000 284) <dependency org="com.thoughtworks.pa 9e58f6fc (Sharad Agarwal 2009-08-14 05:10:40 +0000 285) name= "paranamer" 9e58f6fc (Sharad Agarwal 2009-08-14 05:10:40 +0000 286) rev= "1.5" 9e58f6fc (Sharad Agarwal 2009-08-14 05:10:40 +0000 287) conf= "common-> default " />
        Hide
        Aaron Kimball added a comment -

        Attaching new patch after resync'ing with trunk. Just realized that avro was already added to sqoop's ivy.xml

        Show
        Aaron Kimball added a comment - Attaching new patch after resync'ing with trunk. Just realized that avro was already added to sqoop's ivy.xml
        Hide
        Aaron Kimball added a comment -

        Recycling patch again.. seems to have been dropped from the queue.

        Show
        Aaron Kimball added a comment - Recycling patch again.. seems to have been dropped from the queue.
        Hide
        Enis Soztutar added a comment -

        +1

        Show
        Enis Soztutar added a comment - +1
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12416939/MAPREDUCE-875.2.patch
        against trunk revision 807165.

        +1 @author. The patch does not contain any @author tags.

        -1 tests included. The patch doesn't appear to include any new or modified tests.
        Please justify why no new tests are needed for this patch.
        Also please list what manual steps were performed to verify this patch.

        +1 javadoc. The javadoc tool did not generate any warning messages.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 findbugs. The patch does not introduce any new Findbugs warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        -1 core tests. The patch failed core unit tests.

        +1 contrib tests. The patch passed contrib unit tests.

        Test results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-vesta.apache.org/513/testReport/
        Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-vesta.apache.org/513/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
        Checkstyle results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-vesta.apache.org/513/artifact/trunk/build/test/checkstyle-errors.html
        Console output: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-vesta.apache.org/513/console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12416939/MAPREDUCE-875.2.patch against trunk revision 807165. +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed core unit tests. +1 contrib tests. The patch passed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-vesta.apache.org/513/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-vesta.apache.org/513/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Checkstyle results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-vesta.apache.org/513/artifact/trunk/build/test/checkstyle-errors.html Console output: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-vesta.apache.org/513/console This message is automatically generated.
        Hide
        Aaron Kimball added a comment -

        No new tests because it's a very small code change; existing Sqoop unit tests cover this and pass.

        Core test failure is unrelated.

        Show
        Aaron Kimball added a comment - No new tests because it's a very small code change; existing Sqoop unit tests cover this and pass. Core test failure is unrelated.
        Hide
        Enis Soztutar added a comment -

        I have committed this. Thanks Aaron!

        Show
        Enis Soztutar added a comment - I have committed this. Thanks Aaron!

          People

          • Assignee:
            Aaron Kimball
            Reporter:
            Aaron Kimball
          • Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development