Sqoop
  1. Sqoop
  2. SQOOP-357

To make debugging easier, Sqoop should print out all the exceptions

    Details

    • Type: Improvement Improvement
    • Status: Resolved
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 1.4.0-incubating
    • Component/s: None
    • Labels:
      None

      Description

      I am using sqoop version from https://github.com/apache/sqoop (trunk) . I am pasting some of the error message lines I get when I try to run an import command

      SELECT ID, FIRSTNAME, LASTNAME FROM TEST_TABLE AS TEST_TABLE WHERE ( ID >= 10 ) AND ( ID < 15 ). Details of the failure can be found in the exception chain that is accessible with getNextException.

      .....

      at com.cloudera.sqoop.mapreduce.db.DBRecordReader.executeQuery(DBRecordReader.java:110)
      at com.cloudera.sqoop.mapreduce.db.DBRecordReader.nextKeyValue(DBRecordReader.java:235)
      ... 11 more

      It does not print out the entire list of exceptions.

      1. SQOOP-357-2.patch
        4 kB
        Bilung Lee
      2. SQOOP-357-1.patch
        4 kB
        Bilung Lee
      3. SQOOP-357.patch
        4 kB
        Bilung Lee

        Activity

        Hide
        Arvind Prabhakar added a comment -

        Prashant, can you please attach the full console output with the --verbose flag.

        Show
        Arvind Prabhakar added a comment - Prashant, can you please attach the full console output with the --verbose flag.
        Hide
        jiraposter@reviews.apache.org added a comment -

        -----------------------------------------------------------
        This is an automatically generated e-mail. To reply, visit:
        https://reviews.apache.org/r/2265/
        -----------------------------------------------------------

        Review request for Sqoop and Arvind Prabhakar.

        Summary
        -------

        Currently, only the error message of the first exception in a chain will be logged.
        However, the root cause exception may be buried in the exception chain.
        To help diagnose the root cause, we need to log all error messages in the chain.

        This addresses bug SQOOP-357.
        https://issues.apache.org/jira/browse/SQOOP-357

        Diffs


        ./src/java/com/cloudera/sqoop/mapreduce/AsyncSqlRecordWriter.java 1179036
        ./src/java/com/cloudera/sqoop/mapreduce/db/DBRecordReader.java 1179036

        Diff: https://reviews.apache.org/r/2265/diff

        Testing
        -------

        Thanks,

        Bilung

        Show
        jiraposter@reviews.apache.org added a comment - ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/2265/ ----------------------------------------------------------- Review request for Sqoop and Arvind Prabhakar. Summary ------- Currently, only the error message of the first exception in a chain will be logged. However, the root cause exception may be buried in the exception chain. To help diagnose the root cause, we need to log all error messages in the chain. This addresses bug SQOOP-357 . https://issues.apache.org/jira/browse/SQOOP-357 Diffs ./src/java/com/cloudera/sqoop/mapreduce/AsyncSqlRecordWriter.java 1179036 ./src/java/com/cloudera/sqoop/mapreduce/db/DBRecordReader.java 1179036 Diff: https://reviews.apache.org/r/2265/diff Testing ------- Thanks, Bilung
        Hide
        jiraposter@reviews.apache.org added a comment -

        -----------------------------------------------------------
        This is an automatically generated e-mail. To reply, visit:
        https://reviews.apache.org/r/2265/#review2424
        -----------------------------------------------------------

        Overall the changes look good. One suggestion though - would it be possible to move the logging logic for SQLException in a Util class somewhere (perhaps a new util class called ExceptionUtils in org.apache.sqoop). That way, in other places where we catch SQLException, we could simply invoke this method.

        • Arvind

        On 2011-10-07 02:15:04, Bilung Lee wrote:

        -----------------------------------------------------------

        This is an automatically generated e-mail. To reply, visit:

        https://reviews.apache.org/r/2265/

        -----------------------------------------------------------

        (Updated 2011-10-07 02:15:04)

        Review request for Sqoop and Arvind Prabhakar.

        Summary

        -------

        Currently, only the error message of the first exception in a chain will be logged.

        However, the root cause exception may be buried in the exception chain.

        To help diagnose the root cause, we need to log all error messages in the chain.

        This addresses bug SQOOP-357.

        https://issues.apache.org/jira/browse/SQOOP-357

        Diffs

        -----

        ./src/java/com/cloudera/sqoop/mapreduce/AsyncSqlRecordWriter.java 1179036

        ./src/java/com/cloudera/sqoop/mapreduce/db/DBRecordReader.java 1179036

        Diff: https://reviews.apache.org/r/2265/diff

        Testing

        -------

        Thanks,

        Bilung

        Show
        jiraposter@reviews.apache.org added a comment - ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/2265/#review2424 ----------------------------------------------------------- Overall the changes look good. One suggestion though - would it be possible to move the logging logic for SQLException in a Util class somewhere (perhaps a new util class called ExceptionUtils in org.apache.sqoop). That way, in other places where we catch SQLException, we could simply invoke this method. Arvind On 2011-10-07 02:15:04, Bilung Lee wrote: ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/2265/ ----------------------------------------------------------- (Updated 2011-10-07 02:15:04) Review request for Sqoop and Arvind Prabhakar. Summary ------- Currently, only the error message of the first exception in a chain will be logged. However, the root cause exception may be buried in the exception chain. To help diagnose the root cause, we need to log all error messages in the chain. This addresses bug SQOOP-357 . https://issues.apache.org/jira/browse/SQOOP-357 Diffs ----- ./src/java/com/cloudera/sqoop/mapreduce/AsyncSqlRecordWriter.java 1179036 ./src/java/com/cloudera/sqoop/mapreduce/db/DBRecordReader.java 1179036 Diff: https://reviews.apache.org/r/2265/diff Testing ------- Thanks, Bilung
        Hide
        jiraposter@reviews.apache.org added a comment -

        -----------------------------------------------------------
        This is an automatically generated e-mail. To reply, visit:
        https://reviews.apache.org/r/2265/
        -----------------------------------------------------------

        (Updated 2011-10-07 16:06:41.332093)

        Review request for Sqoop and Arvind Prabhakar.

        Summary
        -------

        Currently, only the error message of the first exception in a chain will be logged.
        However, the root cause exception may be buried in the exception chain.
        To help diagnose the root cause, we need to log all error messages in the chain.

        This addresses bug SQOOP-357.
        https://issues.apache.org/jira/browse/SQOOP-357

        Diffs (updated)


        ./src/java/com/cloudera/sqoop/mapreduce/AsyncSqlRecordWriter.java 1179036
        ./src/java/com/cloudera/sqoop/mapreduce/db/DBRecordReader.java 1179036
        ./src/java/com/cloudera/sqoop/util/LoggingUtils.java PRE-CREATION

        Diff: https://reviews.apache.org/r/2265/diff

        Testing
        -------

        Thanks,

        Bilung

        Show
        jiraposter@reviews.apache.org added a comment - ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/2265/ ----------------------------------------------------------- (Updated 2011-10-07 16:06:41.332093) Review request for Sqoop and Arvind Prabhakar. Summary ------- Currently, only the error message of the first exception in a chain will be logged. However, the root cause exception may be buried in the exception chain. To help diagnose the root cause, we need to log all error messages in the chain. This addresses bug SQOOP-357 . https://issues.apache.org/jira/browse/SQOOP-357 Diffs (updated) ./src/java/com/cloudera/sqoop/mapreduce/AsyncSqlRecordWriter.java 1179036 ./src/java/com/cloudera/sqoop/mapreduce/db/DBRecordReader.java 1179036 ./src/java/com/cloudera/sqoop/util/LoggingUtils.java PRE-CREATION Diff: https://reviews.apache.org/r/2265/diff Testing ------- Thanks, Bilung
        Hide
        jiraposter@reviews.apache.org added a comment -

        On 2011-10-07 02:41:42, Arvind Prabhakar wrote:

        > Overall the changes look good. One suggestion though - would it be possible to move the logging logic for SQLException in a Util class somewhere (perhaps a new util class called ExceptionUtils in org.apache.sqoop). That way, in other places where we catch SQLException, we could simply invoke this method.

        Thanks for your feedback! A utility class is added in new patch.

        • Bilung

        -----------------------------------------------------------
        This is an automatically generated e-mail. To reply, visit:
        https://reviews.apache.org/r/2265/#review2424
        -----------------------------------------------------------

        On 2011-10-07 02:15:04, Bilung Lee wrote:

        -----------------------------------------------------------

        This is an automatically generated e-mail. To reply, visit:

        https://reviews.apache.org/r/2265/

        -----------------------------------------------------------

        (Updated 2011-10-07 02:15:04)

        Review request for Sqoop and Arvind Prabhakar.

        Summary

        -------

        Currently, only the error message of the first exception in a chain will be logged.

        However, the root cause exception may be buried in the exception chain.

        To help diagnose the root cause, we need to log all error messages in the chain.

        This addresses bug SQOOP-357.

        https://issues.apache.org/jira/browse/SQOOP-357

        Diffs

        -----

        ./src/java/com/cloudera/sqoop/mapreduce/AsyncSqlRecordWriter.java 1179036

        ./src/java/com/cloudera/sqoop/mapreduce/db/DBRecordReader.java 1179036

        Diff: https://reviews.apache.org/r/2265/diff

        Testing

        -------

        Thanks,

        Bilung

        Show
        jiraposter@reviews.apache.org added a comment - On 2011-10-07 02:41:42, Arvind Prabhakar wrote: > Overall the changes look good. One suggestion though - would it be possible to move the logging logic for SQLException in a Util class somewhere (perhaps a new util class called ExceptionUtils in org.apache.sqoop). That way, in other places where we catch SQLException, we could simply invoke this method. Thanks for your feedback! A utility class is added in new patch. Bilung ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/2265/#review2424 ----------------------------------------------------------- On 2011-10-07 02:15:04, Bilung Lee wrote: ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/2265/ ----------------------------------------------------------- (Updated 2011-10-07 02:15:04) Review request for Sqoop and Arvind Prabhakar. Summary ------- Currently, only the error message of the first exception in a chain will be logged. However, the root cause exception may be buried in the exception chain. To help diagnose the root cause, we need to log all error messages in the chain. This addresses bug SQOOP-357 . https://issues.apache.org/jira/browse/SQOOP-357 Diffs ----- ./src/java/com/cloudera/sqoop/mapreduce/AsyncSqlRecordWriter.java 1179036 ./src/java/com/cloudera/sqoop/mapreduce/db/DBRecordReader.java 1179036 Diff: https://reviews.apache.org/r/2265/diff Testing ------- Thanks, Bilung
        Hide
        jiraposter@reviews.apache.org added a comment -

        -----------------------------------------------------------
        This is an automatically generated e-mail. To reply, visit:
        https://reviews.apache.org/r/2265/#review2433
        -----------------------------------------------------------

        Ship it!

        +1

        • Arvind

        On 2011-10-07 16:06:41, Bilung Lee wrote:

        -----------------------------------------------------------

        This is an automatically generated e-mail. To reply, visit:

        https://reviews.apache.org/r/2265/

        -----------------------------------------------------------

        (Updated 2011-10-07 16:06:41)

        Review request for Sqoop and Arvind Prabhakar.

        Summary

        -------

        Currently, only the error message of the first exception in a chain will be logged.

        However, the root cause exception may be buried in the exception chain.

        To help diagnose the root cause, we need to log all error messages in the chain.

        This addresses bug SQOOP-357.

        https://issues.apache.org/jira/browse/SQOOP-357

        Diffs

        -----

        ./src/java/com/cloudera/sqoop/mapreduce/AsyncSqlRecordWriter.java 1179036

        ./src/java/com/cloudera/sqoop/mapreduce/db/DBRecordReader.java 1179036

        ./src/java/com/cloudera/sqoop/util/LoggingUtils.java PRE-CREATION

        Diff: https://reviews.apache.org/r/2265/diff

        Testing

        -------

        Thanks,

        Bilung

        Show
        jiraposter@reviews.apache.org added a comment - ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/2265/#review2433 ----------------------------------------------------------- Ship it! +1 Arvind On 2011-10-07 16:06:41, Bilung Lee wrote: ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/2265/ ----------------------------------------------------------- (Updated 2011-10-07 16:06:41) Review request for Sqoop and Arvind Prabhakar. Summary ------- Currently, only the error message of the first exception in a chain will be logged. However, the root cause exception may be buried in the exception chain. To help diagnose the root cause, we need to log all error messages in the chain. This addresses bug SQOOP-357 . https://issues.apache.org/jira/browse/SQOOP-357 Diffs ----- ./src/java/com/cloudera/sqoop/mapreduce/AsyncSqlRecordWriter.java 1179036 ./src/java/com/cloudera/sqoop/mapreduce/db/DBRecordReader.java 1179036 ./src/java/com/cloudera/sqoop/util/LoggingUtils.java PRE-CREATION Diff: https://reviews.apache.org/r/2265/diff Testing ------- Thanks, Bilung
        Hide
        Arvind Prabhakar added a comment -

        Thanks for the patch Bilung. It triggered a checkstyle warning:

        LoggingUtils: Utility classes should not have a public or default constructor.

        Can you please add a private constructor to this class to disable explicit object creation? Sorry I did not spot it during the review.

        Show
        Arvind Prabhakar added a comment - Thanks for the patch Bilung. It triggered a checkstyle warning: LoggingUtils: Utility classes should not have a public or default constructor. Can you please add a private constructor to this class to disable explicit object creation? Sorry I did not spot it during the review.
        Hide
        Arvind Prabhakar added a comment -

        Patch committed. Thanks Bilung!

        Show
        Arvind Prabhakar added a comment - Patch committed. Thanks Bilung!
        Hide
        Hudson added a comment -

        Integrated in Sqoop-jdk-1.6 #35 (See https://builds.apache.org/job/Sqoop-jdk-1.6/35/)
        SQOOP-357. Debug logs should include chained exception trace.

        (Bilung Lee via Arvind Prabhakar)

        arvind : http://svn.apache.org/viewvc/?view=rev&rev=1180162
        Files :

        • /incubator/sqoop/trunk/src/java/com/cloudera/sqoop/mapreduce/AsyncSqlRecordWriter.java
        • /incubator/sqoop/trunk/src/java/com/cloudera/sqoop/mapreduce/db/DBRecordReader.java
        • /incubator/sqoop/trunk/src/java/com/cloudera/sqoop/util/LoggingUtils.java
        Show
        Hudson added a comment - Integrated in Sqoop-jdk-1.6 #35 (See https://builds.apache.org/job/Sqoop-jdk-1.6/35/ ) SQOOP-357 . Debug logs should include chained exception trace. (Bilung Lee via Arvind Prabhakar) arvind : http://svn.apache.org/viewvc/?view=rev&rev=1180162 Files : /incubator/sqoop/trunk/src/java/com/cloudera/sqoop/mapreduce/AsyncSqlRecordWriter.java /incubator/sqoop/trunk/src/java/com/cloudera/sqoop/mapreduce/db/DBRecordReader.java /incubator/sqoop/trunk/src/java/com/cloudera/sqoop/util/LoggingUtils.java

          People

          • Assignee:
            Bilung Lee
            Reporter:
            Prashant Gokhale
          • Votes:
            0 Vote for this issue
            Watchers:
            0 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development