Uploaded image for project: 'Sqoop (Retired)'
  1. Sqoop (Retired)
  2. SQOOP-2496

Sqoop2: Provide a way to inject external connectors

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Open
    • Major
    • Resolution: Unresolved
    • 1.99.6
    • 1.99.8
    • None
    • None

    Description

      On internal hackathon we we're hacking Sqoop 2 connector with singhashish and we went through few troubles that we should address.

      We have a a configuration property for extra directory from which we will load jar files. We were able to use this configuration property to load our hacked connector to Server, but we were not able to get it working through job submission. Here is the exception that we hit:

       
      java.lang.IllegalArgumentException: java.net.URISyntaxException: Relative path in absolute URI: jar:file://var/lib/sqoop/connectors/Connector-1.0-SNAPSHOT.jar!
              at org.apache.hadoop.fs.Path.initialize(Path.java:206)
              at org.apache.hadoop.fs.Path.<init>(Path.java:172)
              at org.apache.hadoop.mapreduce.JobSubmitter.copyAndConfigureFiles(JobSubmitter.java:215)
              at org.apache.hadoop.mapreduce.JobSubmitter.copyAndConfigureFiles(JobSubmitter.java:390)
              at org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:483)
              at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1306)
              at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1303)
              at java.security.AccessController.doPrivileged(Native Method)
              at javax.security.auth.Subject.doAs(Subject.java:415)
              at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1671)
              at org.apache.hadoop.mapreduce.Job.submit(Job.java:1303)
              at org.apache.sqoop.submission.mapreduce.MapreduceSubmissionEngine.submitToCluster(MapreduceSubmissionEngine.java:274)
              at org.apache.sqoop.submission.mapreduce.MapreduceSubmissionEngine.submit(MapreduceSubmissionEngine.java:255)
              at org.apache.sqoop.driver.JobManager.start(JobManager.java:288)
              at org.apache.sqoop.handler.JobRequestHandler.startJob(JobRequestHandler.java:380)
              at org.apache.sqoop.handler.JobRequestHandler.handleEvent(JobRequestHandler.java:116)
              at org.apache.sqoop.server.v1.JobServlet.handlePutRequest(JobServlet.java:96)
              at org.apache.sqoop.server.SqoopProtocolServlet.doPut(SqoopProtocolServlet.java:79)
              at javax.servlet.http.HttpServlet.service(HttpServlet.java:646)
              at javax.servlet.http.HttpServlet.service(HttpServlet.java:723)
      

      To put into nutshell, ClassUtils.jarForClass returns for external connectors path starting with prefix jar:file and suffix ! that breaks mapreduce code.

      Attachments

        Activity

          People

            Unassigned Unassigned
            jarcec Jarek Jarcec Cecho
            Votes:
            1 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

              Created:
              Updated: