Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Blocker Blocker
    • Resolution: Fixed
    • Affects Version/s: 0.6.0
    • Fix Version/s: 0.6.0
    • Component/s: Tests
    • Labels:
      None

      Description

      We added Sqoop2 (1.99 something) to Bigtop recently. We should add some integration tests for it as well.

        Issue Links

          Activity

          Hide
          Mark Grover added a comment -

          FYI, while working on this I ran into a problem related to retrieving secrets in Credentials from Hadoop. Here is the exception trace I got:

          Job commit failed: java.lang.NullPointerException
          at java.lang.String.(String.java:593)
          at org.apache.sqoop.job.mr.ConfigurationUtils.loadConfiguration(ConfigurationUtils.java:77)
          at org.apache.sqoop.job.mr.ConfigurationUtils.getConnectorConnection(ConfigurationUtils.java:38)
          at org.apache.sqoop.job.mr.SqoopDestroyerExecutor.executeDestroyer(SqoopDestroyerExecutor.java:55)
          at org.apache.sqoop.job.mr.SqoopFileOutputFormat$DestroyerFileOutputCommitter.commitJob(SqoopFileOutputFormat.java:89)
          at org.apache.hadoop.mapreduce.v2.app.commit.CommitterEventHandler$EventProcessor.handleJobCommit(CommitterEventHandler.java:249)
          at org.apache.hadoop.mapreduce.v2.app.commit.CommitterEventHandler$EventProcessor.run(CommitterEventHandler.java:212)
          at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
          at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
          at java.lang.Thread.run(Thread.java:662)
          

          I am going to poke more into it...

          Show
          Mark Grover added a comment - FYI, while working on this I ran into a problem related to retrieving secrets in Credentials from Hadoop. Here is the exception trace I got: Job commit failed: java.lang.NullPointerException at java.lang. String .( String .java:593) at org.apache.sqoop.job.mr.ConfigurationUtils.loadConfiguration(ConfigurationUtils.java:77) at org.apache.sqoop.job.mr.ConfigurationUtils.getConnectorConnection(ConfigurationUtils.java:38) at org.apache.sqoop.job.mr.SqoopDestroyerExecutor.executeDestroyer(SqoopDestroyerExecutor.java:55) at org.apache.sqoop.job.mr.SqoopFileOutputFormat$DestroyerFileOutputCommitter.commitJob(SqoopFileOutputFormat.java:89) at org.apache.hadoop.mapreduce.v2.app.commit.CommitterEventHandler$EventProcessor.handleJobCommit(CommitterEventHandler.java:249) at org.apache.hadoop.mapreduce.v2.app.commit.CommitterEventHandler$EventProcessor.run(CommitterEventHandler.java:212) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang. Thread .run( Thread .java:662) I am going to poke more into it...
          Hide
          Jarek Jarcec Cecho added a comment -

          I've took a deeper look into the Sqoop part and interestingly everything is working when running on Hadoop 1. The same code sniplet is even working in map task, however is failing in OutputCommitter. Thus I'm guessing that YARN is not propagating the Credentials object into the OutputCommitter. I might be wrong though.

          Show
          Jarek Jarcec Cecho added a comment - I've took a deeper look into the Sqoop part and interestingly everything is working when running on Hadoop 1. The same code sniplet is even working in map task, however is failing in OutputCommitter . Thus I'm guessing that YARN is not propagating the Credentials object into the OutputCommitter . I might be wrong though.
          Hide
          Mark Grover added a comment -

          Thanks for looking into this, Jarcec! I appreciate your insight and thoughts. What you are saying makes sense to me.

          I also tested Sqoop against Bigtop 0.5 (which has Hadoop 2.0.2-alpha) and Sqoop import job failed there as well with the following error:

          Error: java.lang.NullPointerException 
          at java.lang.String.<init>(String.java:593) 
          at org.apache.sqoop.job.mr.ConfigurationUtils.loadConfiguration(ConfigurationUtils.java:77) 
          at org.apache.sqoop.job.mr.ConfigurationUtils.getConnectorConnection(ConfigurationUtils.java:38) 
          at org.apache.sqoop.job.mr.SqoopMapper.run(SqoopMapper.java:69) 
          at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:726) 
          at org.apache.hadoop.mapred.MapTask.run(MapTask.java:333) 
          at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:157) 
          at java.security.AccessController.doPrivileged(Native Method) 
          at javax.security.auth.Subject.doAs(Subject.java:396) 
          at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1367) 
          at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:152)
          

          Note that this error even though looks similar (related to retrieving secrets in credentials) at the first glance is a little different. The error on Bigtop 0.6 (hence Hadoop 2.0.4-alpha) happens in Job Commit phase (aka output committer) while the error on Bigtop 0.5 (hence Hadoop 2.0.2-alpha) happens in the map task.

          My hunch is this related to HADOOP-8726. If I were to guess, I would say while HADOOP-8726 made the credentials available to MR jobs, the OutputCommitter committer was missed, like Jarcec mentioned.

          Show
          Mark Grover added a comment - Thanks for looking into this, Jarcec! I appreciate your insight and thoughts. What you are saying makes sense to me. I also tested Sqoop against Bigtop 0.5 (which has Hadoop 2.0.2-alpha) and Sqoop import job failed there as well with the following error: Error: java.lang.NullPointerException at java.lang. String .<init>( String .java:593) at org.apache.sqoop.job.mr.ConfigurationUtils.loadConfiguration(ConfigurationUtils.java:77) at org.apache.sqoop.job.mr.ConfigurationUtils.getConnectorConnection(ConfigurationUtils.java:38) at org.apache.sqoop.job.mr.SqoopMapper.run(SqoopMapper.java:69) at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:726) at org.apache.hadoop.mapred.MapTask.run(MapTask.java:333) at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:157) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:396) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1367) at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:152) Note that this error even though looks similar (related to retrieving secrets in credentials) at the first glance is a little different. The error on Bigtop 0.6 (hence Hadoop 2.0.4-alpha) happens in Job Commit phase (aka output committer) while the error on Bigtop 0.5 (hence Hadoop 2.0.2-alpha) happens in the map task. My hunch is this related to HADOOP-8726 . If I were to guess , I would say while HADOOP-8726 made the credentials available to MR jobs, the OutputCommitter committer was missed, like Jarcec mentioned.
          Hide
          Roman Shaposhnik added a comment -

          Created MAPREDUCE-5240 lets see whether there's any kind of workaround for us here.

          Show
          Roman Shaposhnik added a comment - Created MAPREDUCE-5240 lets see whether there's any kind of workaround for us here.
          Hide
          Mark Grover added a comment -

          Thanks Roman! And, thanks for attaching the example on that JIRA, it really helps!

          Show
          Mark Grover added a comment - Thanks Roman! And, thanks for attaching the example on that JIRA, it really helps!
          Hide
          Anatoli Fomenko added a comment -

          Patch submitted.
          Many thanks to Roman, Jarcec and Mark without whose help creation of these tests would not be possible.

          Show
          Anatoli Fomenko added a comment - Patch submitted. Many thanks to Roman, Jarcec and Mark without whose help creation of these tests would not be possible.
          Hide
          Roman Shaposhnik added a comment -

          +1 and committed!

          Show
          Roman Shaposhnik added a comment - +1 and committed!

            People

            • Assignee:
              Anatoli Fomenko
              Reporter:
              Mark Grover
            • Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development