Sqoop
  1. Sqoop
  2. SQOOP-674

Sqoop2: Exceptions in special map reduce threads can cause mapreduce job to freeze

    Details

    • Type: Bug Bug
    • Status: Resolved
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 1.99.1
    • Component/s: None
    • Labels:
      None

      Description

      We need to properly handle exceptions in our map reduce threads as they can freeze running map reduce job.

      1. SQOOP-674.patch
        14 kB
        Hari Shreedharan
      2. bugSQOOP-674.patch
        0.9 kB
        Jarek Jarcec Cecho

        Issue Links

          Activity

          Hide
          Jarek Jarcec Cecho added a comment -

          I found one case when our threads will came to deadlock:

          1) Reader (=loader) gets data from shared queue.
          2) Writer (=extractor) will put additional data to shared queue.
          3) Writer (=extractor) will try to put additional data into shared queue, but will be blocked by call free.acquire() in SqoopRecordWriter::write()
          4) Reader (=loader) will throw an exception while processing data from 1)

          In this case we have a deadlock as writer will be waiting on dead reader.

          Show
          Jarek Jarcec Cecho added a comment - I found one case when our threads will came to deadlock: 1) Reader (=loader) gets data from shared queue. 2) Writer (=extractor) will put additional data to shared queue. 3) Writer (=extractor) will try to put additional data into shared queue, but will be blocked by call free.acquire() in SqoopRecordWriter::write() 4) Reader (=loader) will throw an exception while processing data from 1) In this case we have a deadlock as writer will be waiting on dead reader.
          Hide
          Hari Shreedharan added a comment -

          Per conversation on review board, I am submitting a patch which releases the semaphore on throwable being thrown before propagating the error. Added a bunch of unit tests too.

          Show
          Hari Shreedharan added a comment - Per conversation on review board, I am submitting a patch which releases the semaphore on throwable being thrown before propagating the error. Added a bunch of unit tests too.
          Hide
          Jarek Jarcec Cecho added a comment -

          The patch is in: https://git-wip-us.apache.org/repos/asf?p=sqoop.git;a=commit;h=ac362377853581eb3c5347da3c1d4404915c98d8

          Thank you very much for your hard work Hari!

          Jarcec

          Show
          Jarek Jarcec Cecho added a comment - The patch is in: https://git-wip-us.apache.org/repos/asf?p=sqoop.git;a=commit;h=ac362377853581eb3c5347da3c1d4404915c98d8 Thank you very much for your hard work Hari! Jarcec
          Hide
          Hudson added a comment -

          Integrated in Sqoop2-hadoop200 #11 (See https://builds.apache.org/job/Sqoop2-hadoop200/11/)
          SQOOP-674: Sqoop2: Exceptions in special map reduce threads can cause mapreduce job to freeze (Revision ac362377853581eb3c5347da3c1d4404915c98d8)

          Result = SUCCESS
          jarcec : https://git-wip-us.apache.org/repos/asf?p=sqoop.git&a=commit&h=ac362377853581eb3c5347da3c1d4404915c98d8
          Files :

          • execution/mapreduce/src/main/java/org/apache/sqoop/job/mr/SqoopOutputFormatLoadExecutor.java
          • execution/mapreduce/src/test/java/org/apache/sqoop/job/mr/TestSqoopOutputFormatLoadExecutor.java
          Show
          Hudson added a comment - Integrated in Sqoop2-hadoop200 #11 (See https://builds.apache.org/job/Sqoop2-hadoop200/11/ ) SQOOP-674 : Sqoop2: Exceptions in special map reduce threads can cause mapreduce job to freeze (Revision ac362377853581eb3c5347da3c1d4404915c98d8) Result = SUCCESS jarcec : https://git-wip-us.apache.org/repos/asf?p=sqoop.git&a=commit&h=ac362377853581eb3c5347da3c1d4404915c98d8 Files : execution/mapreduce/src/main/java/org/apache/sqoop/job/mr/SqoopOutputFormatLoadExecutor.java execution/mapreduce/src/test/java/org/apache/sqoop/job/mr/TestSqoopOutputFormatLoadExecutor.java
          Hide
          Hudson added a comment -

          Integrated in Sqoop2-hadoop100 #12 (See https://builds.apache.org/job/Sqoop2-hadoop100/12/)
          SQOOP-674: Sqoop2: Exceptions in special map reduce threads can cause mapreduce job to freeze (Revision ac362377853581eb3c5347da3c1d4404915c98d8)

          Result = FAILURE
          jarcec : https://git-wip-us.apache.org/repos/asf?p=sqoop.git&a=commit&h=ac362377853581eb3c5347da3c1d4404915c98d8
          Files :

          • execution/mapreduce/src/test/java/org/apache/sqoop/job/mr/TestSqoopOutputFormatLoadExecutor.java
          • execution/mapreduce/src/main/java/org/apache/sqoop/job/mr/SqoopOutputFormatLoadExecutor.java
          Show
          Hudson added a comment - Integrated in Sqoop2-hadoop100 #12 (See https://builds.apache.org/job/Sqoop2-hadoop100/12/ ) SQOOP-674 : Sqoop2: Exceptions in special map reduce threads can cause mapreduce job to freeze (Revision ac362377853581eb3c5347da3c1d4404915c98d8) Result = FAILURE jarcec : https://git-wip-us.apache.org/repos/asf?p=sqoop.git&a=commit&h=ac362377853581eb3c5347da3c1d4404915c98d8 Files : execution/mapreduce/src/test/java/org/apache/sqoop/job/mr/TestSqoopOutputFormatLoadExecutor.java execution/mapreduce/src/main/java/org/apache/sqoop/job/mr/SqoopOutputFormatLoadExecutor.java

            People

            • Assignee:
              Hari Shreedharan
              Reporter:
              Jarek Jarcec Cecho
            • Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development