Sqoop
  1. Sqoop
  2. SQOOP-863

Sqoop2: Introduce ProgressThread into Extractor and Loader

    Details

    • Type: Improvement Improvement
    • Status: Resolved
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 1.99.1
    • Fix Version/s: 1.99.2
    • Component/s: None
    • Labels:
      None

      Description

      We need to introduce extractor and loader ProgressThread so that we can report progress. This is very critical for long running tasks as we need to propagate the progress up to the mapreduce engine so that our tasks won't get killed.

      1. bugSQOOP-863.patch
        6 kB
        Jarek Jarcec Cecho
      2. bugSQOOP-863.patch
        6 kB
        Jarek Jarcec Cecho
      3. bugSQOOP-863.patch
        9 kB
        Jarek Jarcec Cecho

        Issue Links

          Activity

          Hide
          Kathleen Ting added a comment -
          Show
          Kathleen Ting added a comment - Thanks Jarcec for your contribution. The patch is in: https://git-wip-us.apache.org/repos/asf?p=sqoop.git;a=commit;h=3f574f9813edaa049bc833b8843c3039e75f2dbc
          Hide
          Hudson added a comment -

          Integrated in Sqoop2-hadoop100 #250 (See https://builds.apache.org/job/Sqoop2-hadoop100/250/)
          SQOOP-863: Sqoop2: Introduce ProgressThread into Extractor and Loader (Revision 3f574f9813edaa049bc833b8843c3039e75f2dbc)

          Result = UNSTABLE
          kathleen : https://git-wip-us.apache.org/repos/asf?p=sqoop.git&a=commit&h=3f574f9813edaa049bc833b8843c3039e75f2dbc
          Files :

          • execution/mapreduce/src/main/java/org/apache/sqoop/job/mr/SqoopReducer.java
          • execution/mapreduce/src/main/java/org/apache/sqoop/job/mr/ProgressRunnable.java
          • execution/mapreduce/src/main/java/org/apache/sqoop/job/mr/SqoopMapper.java
          Show
          Hudson added a comment - Integrated in Sqoop2-hadoop100 #250 (See https://builds.apache.org/job/Sqoop2-hadoop100/250/ ) SQOOP-863 : Sqoop2: Introduce ProgressThread into Extractor and Loader (Revision 3f574f9813edaa049bc833b8843c3039e75f2dbc) Result = UNSTABLE kathleen : https://git-wip-us.apache.org/repos/asf?p=sqoop.git&a=commit&h=3f574f9813edaa049bc833b8843c3039e75f2dbc Files : execution/mapreduce/src/main/java/org/apache/sqoop/job/mr/SqoopReducer.java execution/mapreduce/src/main/java/org/apache/sqoop/job/mr/ProgressRunnable.java execution/mapreduce/src/main/java/org/apache/sqoop/job/mr/SqoopMapper.java
          Hide
          Hudson added a comment -

          Integrated in Sqoop2-hadoop200 #251 (See https://builds.apache.org/job/Sqoop2-hadoop200/251/)
          SQOOP-863: Sqoop2: Introduce ProgressThread into Extractor and Loader (Revision 3f574f9813edaa049bc833b8843c3039e75f2dbc)

          Result = SUCCESS
          kathleen : https://git-wip-us.apache.org/repos/asf?p=sqoop.git&a=commit&h=3f574f9813edaa049bc833b8843c3039e75f2dbc
          Files :

          • execution/mapreduce/src/main/java/org/apache/sqoop/job/mr/SqoopMapper.java
          • execution/mapreduce/src/main/java/org/apache/sqoop/job/mr/SqoopReducer.java
          • execution/mapreduce/src/main/java/org/apache/sqoop/job/mr/ProgressRunnable.java
          Show
          Hudson added a comment - Integrated in Sqoop2-hadoop200 #251 (See https://builds.apache.org/job/Sqoop2-hadoop200/251/ ) SQOOP-863 : Sqoop2: Introduce ProgressThread into Extractor and Loader (Revision 3f574f9813edaa049bc833b8843c3039e75f2dbc) Result = SUCCESS kathleen : https://git-wip-us.apache.org/repos/asf?p=sqoop.git&a=commit&h=3f574f9813edaa049bc833b8843c3039e75f2dbc Files : execution/mapreduce/src/main/java/org/apache/sqoop/job/mr/SqoopMapper.java execution/mapreduce/src/main/java/org/apache/sqoop/job/mr/SqoopReducer.java execution/mapreduce/src/main/java/org/apache/sqoop/job/mr/ProgressRunnable.java

            People

            • Assignee:
              Jarek Jarcec Cecho
              Reporter:
              Jarek Jarcec Cecho
            • Votes:
              1 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development