Sqoop
  1. Sqoop
  2. SQOOP-678

Sqoop2: Add counters handling to map reduce submission engine

    Details

    • Type: Improvement Improvement
    • Status: Resolved
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 1.99.2
    • Component/s: None
    • Labels:

      Description

      We should implement counter handling in map reduce submission engine.

      1. bugSQOOP-678.patch
        4 kB
        Jarek Jarcec Cecho

        Issue Links

          Activity

          Jarek Jarcec Cecho created issue -
          Hide
          Hari Shreedharan added a comment -

          From what I see here, this basically is meant to expose the Map Reduce counters through the stats() method in the submission engine(through the Sqoop counters, of course)? I am planning to take a look at this one. For now, I am just planning to hard code a set of stats to be pulled and then later make it configurable. How does that sound?

          Show
          Hari Shreedharan added a comment - From what I see here, this basically is meant to expose the Map Reduce counters through the stats() method in the submission engine(through the Sqoop counters, of course)? I am planning to take a look at this one. For now, I am just planning to hard code a set of stats to be pulled and then later make it configurable. How does that sound?
          Jarek Jarcec Cecho made changes -
          Field Original Value New Value
          Link This issue is blocked by SQOOP-671 [ SQOOP-671 ]
          Hide
          Jarek Jarcec Cecho added a comment -

          Hi Hary,
          thank you very much for looking into this issue. I've just linked it with SQOOP-671 that I believe needs to be solved first.

          Otherwise you're right. I believe that we need to take important counters and propagate them throw Sqoop counters to framework (and further serialize them in Sqoop's metastore). I think that we do not need to transfer all mapreduce counters. I'm thinking that starting with "Number of transferred items" and "Number of transferred bytes" should be reasonable at the moment.

          Jarcec

          Show
          Jarek Jarcec Cecho added a comment - Hi Hary, thank you very much for looking into this issue. I've just linked it with SQOOP-671 that I believe needs to be solved first. Otherwise you're right. I believe that we need to take important counters and propagate them throw Sqoop counters to framework (and further serialize them in Sqoop's metastore). I think that we do not need to transfer all mapreduce counters. I'm thinking that starting with "Number of transferred items" and "Number of transferred bytes" should be reasonable at the moment. Jarcec
          Hide
          Hari Shreedharan added a comment -

          You mean serialize and store them after job completion, right?

          Show
          Hari Shreedharan added a comment - You mean serialize and store them after job completion, right?
          Hide
          Jarek Jarcec Cecho added a comment -

          Yes, I believe that the counters make sense only after the submission successfully finish. I wouldn't worry about serialization in metastore itself – there is already another JIRA (SQOOP-667) that is currently assigned to me that should cover that.

          Jarcec

          Show
          Jarek Jarcec Cecho added a comment - Yes, I believe that the counters make sense only after the submission successfully finish. I wouldn't worry about serialization in metastore itself – there is already another JIRA ( SQOOP-667 ) that is currently assigned to me that should cover that. Jarcec
          Jarek Jarcec Cecho made changes -
          Labels newbie
          Jarek Jarcec Cecho made changes -
          Assignee Jarek Jarcec Cecho [ jarcec ]
          Jarek Jarcec Cecho made changes -
          Summary Add counters handling to map reduce submission engine Sqoop2: Add counters handling to map reduce submission engine
          Jarek Jarcec Cecho made changes -
          Affects Version/s 2.0.0 [ 12319272 ]
          Fix Version/s 1.99.2 [ 12323640 ]
          Fix Version/s 2.0.0 [ 12319272 ]
          Jarek Jarcec Cecho made changes -
          Remote Link This issue links to "Review board (Web Link)" [ 11940 ]
          Jarek Jarcec Cecho made changes -
          Attachment bugSQOOP-678.patch [ 12562678 ]
          Jarek Jarcec Cecho made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          Hide
          Cheolsoo Park added a comment -

          +1. will commit it after running tests.

          Show
          Cheolsoo Park added a comment - +1. will commit it after running tests.
          Hide
          Cheolsoo Park added a comment -

          Committed to sqoop2. Thanks Jarcec!

          Show
          Cheolsoo Park added a comment - Committed to sqoop2. Thanks Jarcec!
          Cheolsoo Park made changes -
          Status Patch Available [ 10002 ] Resolved [ 5 ]
          Resolution Fixed [ 1 ]
          Hide
          Hudson added a comment -

          Integrated in Sqoop2-hadoop200 #91 (See https://builds.apache.org/job/Sqoop2-hadoop200/91/)
          SQOOP-678: Add counters handling to map reduce submission engine (Revision 4c2a343d96da9547c7c4c7f83782103310b3c0fe)

          Result = SUCCESS
          cheolsoo : https://git-wip-us.apache.org/repos/asf?p=sqoop.git&a=commit&h=4c2a343d96da9547c7c4c7f83782103310b3c0fe
          Files :

          • core/src/main/java/org/apache/sqoop/framework/SubmissionEngine.java
          • core/src/main/java/org/apache/sqoop/framework/FrameworkManager.java
          • submission/mapreduce/src/main/java/org/apache/sqoop/submission/mapreduce/MapreduceSubmissionEngine.java
          Show
          Hudson added a comment - Integrated in Sqoop2-hadoop200 #91 (See https://builds.apache.org/job/Sqoop2-hadoop200/91/ ) SQOOP-678 : Add counters handling to map reduce submission engine (Revision 4c2a343d96da9547c7c4c7f83782103310b3c0fe) Result = SUCCESS cheolsoo : https://git-wip-us.apache.org/repos/asf?p=sqoop.git&a=commit&h=4c2a343d96da9547c7c4c7f83782103310b3c0fe Files : core/src/main/java/org/apache/sqoop/framework/SubmissionEngine.java core/src/main/java/org/apache/sqoop/framework/FrameworkManager.java submission/mapreduce/src/main/java/org/apache/sqoop/submission/mapreduce/MapreduceSubmissionEngine.java
          Hide
          Hudson added a comment -

          Integrated in Sqoop2-hadoop100 #91 (See https://builds.apache.org/job/Sqoop2-hadoop100/91/)
          SQOOP-678: Add counters handling to map reduce submission engine (Revision 4c2a343d96da9547c7c4c7f83782103310b3c0fe)

          Result = SUCCESS
          cheolsoo : https://git-wip-us.apache.org/repos/asf?p=sqoop.git&a=commit&h=4c2a343d96da9547c7c4c7f83782103310b3c0fe
          Files :

          • core/src/main/java/org/apache/sqoop/framework/FrameworkManager.java
          • core/src/main/java/org/apache/sqoop/framework/SubmissionEngine.java
          • submission/mapreduce/src/main/java/org/apache/sqoop/submission/mapreduce/MapreduceSubmissionEngine.java
          Show
          Hudson added a comment - Integrated in Sqoop2-hadoop100 #91 (See https://builds.apache.org/job/Sqoop2-hadoop100/91/ ) SQOOP-678 : Add counters handling to map reduce submission engine (Revision 4c2a343d96da9547c7c4c7f83782103310b3c0fe) Result = SUCCESS cheolsoo : https://git-wip-us.apache.org/repos/asf?p=sqoop.git&a=commit&h=4c2a343d96da9547c7c4c7f83782103310b3c0fe Files : core/src/main/java/org/apache/sqoop/framework/FrameworkManager.java core/src/main/java/org/apache/sqoop/framework/SubmissionEngine.java submission/mapreduce/src/main/java/org/apache/sqoop/submission/mapreduce/MapreduceSubmissionEngine.java

            People

            • Assignee:
              Jarek Jarcec Cecho
              Reporter:
              Jarek Jarcec Cecho
            • Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development