Sqoop
  1. Sqoop
  2. SQOOP-678

Sqoop2: Add counters handling to map reduce submission engine

    Details

    • Type: Improvement Improvement
    • Status: Resolved
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 1.99.2
    • Component/s: None
    • Labels:

      Description

      We should implement counter handling in map reduce submission engine.

      1. bugSQOOP-678.patch
        4 kB
        Jarek Jarcec Cecho

        Issue Links

          Activity

          Hide
          Hari Shreedharan added a comment -

          From what I see here, this basically is meant to expose the Map Reduce counters through the stats() method in the submission engine(through the Sqoop counters, of course)? I am planning to take a look at this one. For now, I am just planning to hard code a set of stats to be pulled and then later make it configurable. How does that sound?

          Show
          Hari Shreedharan added a comment - From what I see here, this basically is meant to expose the Map Reduce counters through the stats() method in the submission engine(through the Sqoop counters, of course)? I am planning to take a look at this one. For now, I am just planning to hard code a set of stats to be pulled and then later make it configurable. How does that sound?
          Hide
          Jarek Jarcec Cecho added a comment -

          Hi Hary,
          thank you very much for looking into this issue. I've just linked it with SQOOP-671 that I believe needs to be solved first.

          Otherwise you're right. I believe that we need to take important counters and propagate them throw Sqoop counters to framework (and further serialize them in Sqoop's metastore). I think that we do not need to transfer all mapreduce counters. I'm thinking that starting with "Number of transferred items" and "Number of transferred bytes" should be reasonable at the moment.

          Jarcec

          Show
          Jarek Jarcec Cecho added a comment - Hi Hary, thank you very much for looking into this issue. I've just linked it with SQOOP-671 that I believe needs to be solved first. Otherwise you're right. I believe that we need to take important counters and propagate them throw Sqoop counters to framework (and further serialize them in Sqoop's metastore). I think that we do not need to transfer all mapreduce counters. I'm thinking that starting with "Number of transferred items" and "Number of transferred bytes" should be reasonable at the moment. Jarcec
          Hide
          Hari Shreedharan added a comment -

          You mean serialize and store them after job completion, right?

          Show
          Hari Shreedharan added a comment - You mean serialize and store them after job completion, right?
          Hide
          Jarek Jarcec Cecho added a comment -

          Yes, I believe that the counters make sense only after the submission successfully finish. I wouldn't worry about serialization in metastore itself – there is already another JIRA (SQOOP-667) that is currently assigned to me that should cover that.

          Jarcec

          Show
          Jarek Jarcec Cecho added a comment - Yes, I believe that the counters make sense only after the submission successfully finish. I wouldn't worry about serialization in metastore itself – there is already another JIRA ( SQOOP-667 ) that is currently assigned to me that should cover that. Jarcec
          Hide
          Cheolsoo Park added a comment -

          +1. will commit it after running tests.

          Show
          Cheolsoo Park added a comment - +1. will commit it after running tests.
          Hide
          Cheolsoo Park added a comment -

          Committed to sqoop2. Thanks Jarcec!

          Show
          Cheolsoo Park added a comment - Committed to sqoop2. Thanks Jarcec!
          Hide
          Hudson added a comment -

          Integrated in Sqoop2-hadoop200 #91 (See https://builds.apache.org/job/Sqoop2-hadoop200/91/)
          SQOOP-678: Add counters handling to map reduce submission engine (Revision 4c2a343d96da9547c7c4c7f83782103310b3c0fe)

          Result = SUCCESS
          cheolsoo : https://git-wip-us.apache.org/repos/asf?p=sqoop.git&a=commit&h=4c2a343d96da9547c7c4c7f83782103310b3c0fe
          Files :

          • core/src/main/java/org/apache/sqoop/framework/SubmissionEngine.java
          • core/src/main/java/org/apache/sqoop/framework/FrameworkManager.java
          • submission/mapreduce/src/main/java/org/apache/sqoop/submission/mapreduce/MapreduceSubmissionEngine.java
          Show
          Hudson added a comment - Integrated in Sqoop2-hadoop200 #91 (See https://builds.apache.org/job/Sqoop2-hadoop200/91/ ) SQOOP-678 : Add counters handling to map reduce submission engine (Revision 4c2a343d96da9547c7c4c7f83782103310b3c0fe) Result = SUCCESS cheolsoo : https://git-wip-us.apache.org/repos/asf?p=sqoop.git&a=commit&h=4c2a343d96da9547c7c4c7f83782103310b3c0fe Files : core/src/main/java/org/apache/sqoop/framework/SubmissionEngine.java core/src/main/java/org/apache/sqoop/framework/FrameworkManager.java submission/mapreduce/src/main/java/org/apache/sqoop/submission/mapreduce/MapreduceSubmissionEngine.java
          Hide
          Hudson added a comment -

          Integrated in Sqoop2-hadoop100 #91 (See https://builds.apache.org/job/Sqoop2-hadoop100/91/)
          SQOOP-678: Add counters handling to map reduce submission engine (Revision 4c2a343d96da9547c7c4c7f83782103310b3c0fe)

          Result = SUCCESS
          cheolsoo : https://git-wip-us.apache.org/repos/asf?p=sqoop.git&a=commit&h=4c2a343d96da9547c7c4c7f83782103310b3c0fe
          Files :

          • core/src/main/java/org/apache/sqoop/framework/FrameworkManager.java
          • core/src/main/java/org/apache/sqoop/framework/SubmissionEngine.java
          • submission/mapreduce/src/main/java/org/apache/sqoop/submission/mapreduce/MapreduceSubmissionEngine.java
          Show
          Hudson added a comment - Integrated in Sqoop2-hadoop100 #91 (See https://builds.apache.org/job/Sqoop2-hadoop100/91/ ) SQOOP-678 : Add counters handling to map reduce submission engine (Revision 4c2a343d96da9547c7c4c7f83782103310b3c0fe) Result = SUCCESS cheolsoo : https://git-wip-us.apache.org/repos/asf?p=sqoop.git&a=commit&h=4c2a343d96da9547c7c4c7f83782103310b3c0fe Files : core/src/main/java/org/apache/sqoop/framework/FrameworkManager.java core/src/main/java/org/apache/sqoop/framework/SubmissionEngine.java submission/mapreduce/src/main/java/org/apache/sqoop/submission/mapreduce/MapreduceSubmissionEngine.java

            People

            • Assignee:
              Jarek Jarcec Cecho
              Reporter:
              Jarek Jarcec Cecho
            • Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development