[SQOOP-2496] Sqoop2: Provide a way to inject external connectors

• Type: Bug
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 1.99.6
• Fix Version/s: 1.99.8
• Assignee: Unassigned
• Reporter: Jarek Jarcec Cecho
• Created: 15/Aug/15 15:41
• Updated: 28/Jul/16 00:13

Description

On internal hackathon we we're hacking Sqoop 2 connector with singhashish and we went through few troubles that we should address.

We have a a configuration property for extra directory from which we will load jar files. We were able to use this configuration property to load our hacked connector to Server, but we were not able to get it working through job submission. Here is the exception that we hit:

 
java.lang.IllegalArgumentException: java.net.URISyntaxException: Relative path in absolute URI: jar:file://var/lib/sqoop/connectors/Connector-1.0-SNAPSHOT.jar!
        at org.apache.hadoop.fs.Path.initialize(Path.java:206)
        at org.apache.hadoop.fs.Path.<init>(Path.java:172)
        at org.apache.hadoop.mapreduce.JobSubmitter.copyAndConfigureFiles(JobSubmitter.java:215)
        at org.apache.hadoop.mapreduce.JobSubmitter.copyAndConfigureFiles(JobSubmitter.java:390)
        at org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:483)
        at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1306)
        at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1303)
        at java.security.AccessController.doPrivileged(Native Method)
        at javax.security.auth.Subject.doAs(Subject.java:415)
        at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1671)
        at org.apache.hadoop.mapreduce.Job.submit(Job.java:1303)
        at org.apache.sqoop.submission.mapreduce.MapreduceSubmissionEngine.submitToCluster(MapreduceSubmissionEngine.java:274)
        at org.apache.sqoop.submission.mapreduce.MapreduceSubmissionEngine.submit(MapreduceSubmissionEngine.java:255)
        at org.apache.sqoop.driver.JobManager.start(JobManager.java:288)
        at org.apache.sqoop.handler.JobRequestHandler.startJob(JobRequestHandler.java:380)
        at org.apache.sqoop.handler.JobRequestHandler.handleEvent(JobRequestHandler.java:116)
        at org.apache.sqoop.server.v1.JobServlet.handlePutRequest(JobServlet.java:96)
        at org.apache.sqoop.server.SqoopProtocolServlet.doPut(SqoopProtocolServlet.java:79)
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:646)
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:723)

To put into nutshell, ClassUtils.jarForClass returns for external connectors path starting with prefix jar:file and suffix ! that breaks mapreduce code.