[SQOOP-1762] Unit tests in Sqoop should use MRUnit for testing the execution engine

• Type: Bug
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Fix Version/s: 2.0.0
• Component/s: test
• Labels: tech-debt-test unit-test
• Assignee: Unassigned
• Reporter: Veena Basavaraj
• Created: 18/Nov/14 22:55
• Updated: 15/Dec/14 17:59

Description

Running unit tests

TestMatching

and

TestMapreduce

tries to run the local hadoop runner environment.

Unit tests should not rely on external dependencies such as MR as well, Either we mock or use a

 MRUnit 

to test the MR execution code

The following exception was generated when trying to run unit tests from outside of the cloudera VPN.

499  [main] DEBUG org.apache.hadoop.mapreduce.Cluster  - Picked org.apache.hadoop.mapred.LocalClientProtocolProvider as the ClientProtocolProvider

499  [main] DEBUG org.apache.hadoop.security.UserGroupInformation  - PrivilegedAction as:vbasavaraj (auth:SIMPLE) from:org.apache.hadoop.mapreduce.Cluster.getFileSystem(Cluster.java:161)

500  [main] DEBUG org.apache.hadoop.security.UserGroupInformation  - PrivilegedAction as:vbasavaraj (auth:SIMPLE) from:org.apache.hadoop.mapreduce.Job.submit(Job.java:1282)

508  [main] DEBUG org.apache.hadoop.security.UserGroupInformation  - PrivilegedActionException as:vbasavaraj (auth:SIMPLE) cause:java.net.UnknownHostException: vbasavaraj.att.net: vbasavaraj.att.net: nodename nor servname provided, or not known

Testing with Schemas

FROM: Schema{name=FROM-1,columns=[

FixedPoint{name=1,nullable=true,type=FIXED_POINT,byteSize=null,unsigned=null},

FloatingPoint{name=2,nullable=true,type=FLOATING_POINT,byteSize=null},

Text{name=3,nullable=true,type=TEXT,charSize=null}]}

TO: Schema{name=TO-2,columns=[

FixedPoint{name=1,nullable=true,type=FIXED_POINT,byteSize=null,unsigned=null},

FloatingPoint{name=2,nullable=true,type=FLOATING_POINT,byteSize=null}]}

532  [main] DEBUG org.apache.hadoop.security.UserGroupInformation  - PrivilegedAction as:vbasavaraj (auth:SIMPLE) from:org.apache.hadoop.mapreduce.Job.connect(Job.java:1250)

533  [main] DEBUG org.apache.hadoop.mapreduce.Cluster  - Trying ClientProtocolProvider : org.apache.hadoop.mapred.YarnClientProtocolProvider

533  [main] DEBUG org.apache.hadoop.mapreduce.Cluster  - Cannot pick org.apache.hadoop.mapred.YarnClientProtocolProvider as the ClientProtocolProvider - returned null protocol

533  [main] DEBUG org.apache.hadoop.mapreduce.Cluster  - Trying ClientProtocolProvider : org.apache.hadoop.mapred.LocalClientProtocolProvider

534  [main] INFO  org.apache.hadoop.metrics.jvm.JvmMetrics  - Cannot initialize JVM Metrics with processName=JobTracker, sessionId= - already initialized

535  [main] INFO  org.apache.hadoop.metrics.MetricsUtil  - Unable to obtain hostName

java.net.UnknownHostException: vbasavaraj.att.net: vbasavaraj.att.net: nodename nor servname provided, or not known

at java.net.InetAddress.getLocalHost(InetAddress.java:1473)

at org.apache.hadoop.metrics.MetricsUtil.getHostName(MetricsUtil.java:95)

at org.apache.hadoop.metrics.MetricsUtil.createRecord(MetricsUtil.java:84)

at org.apache.hadoop.mapred.LocalJobRunnerMetrics.<init>(LocalJobRunnerMetrics.java:44)

at org.apache.hadoop.mapred.LocalJobRunner.<init>(LocalJobRunner.java:714)

at org.apache.hadoop.mapred.LocalJobRunner.<init>(LocalJobRunner.java:707)

at org.apache.hadoop.mapred.LocalClientProtocolProvider.create(LocalClientProtocolProvider.java:42)

at org.apache.hadoop.mapreduce.Cluster.initialize(Cluster.java:95)

at org.apache.hadoop.mapreduce.Cluster.<init>(Cluster.java:82)

at org.apache.hadoop.mapreduce.Cluster.<init>(Cluster.java:75)

at org.apache.hadoop.mapreduce.Job$9.run(Job.java:1255)

at org.apache.hadoop.mapreduce.Job$9.run(Job.java:1251)

at java.security.AccessController.doPrivileged(Native Method)

at javax.security.auth.Subject.doAs(Subject.java:415)

at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1614)

at org.apache.hadoop.mapreduce.Job.connect(Job.java:1250)

at org.apache.hadoop.mapreduce.Job.submit(Job.java:1279)

at org.apache.hadoop.mapreduce.Job.waitForCompletion(Job.java:1303)

at org.apache.sqoop.job.util.MRJobTestUtil.runJob(MRJobTestUtil.java:60)

at org.apache.sqoop.job.TestMatching.testSchemaMatching(TestMatching.java:137)

at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)

at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

at java.lang.reflect.Method.invoke(Method.java:606)

at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)

at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)

at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)

at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)

at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:271)

at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:70)

at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:50)

at org.junit.runners.ParentRunner$3.run(ParentRunner.java:238)

at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:63)

at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:236)

at org.junit.runners.ParentRunner.access$000(ParentRunner.java:53)

at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:229)

at org.junit.runners.ParentRunner.run(ParentRunner.java:309)

at org.junit.runners.Suite.runChild(Suite.java:127)

at org.junit.runners.Suite.runChild(Suite.java:26)

at org.junit.runners.ParentRunner$3.run(ParentRunner.java:238)

at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:63)

at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:236)

at org.junit.runners.ParentRunner.access$000(ParentRunner.java:53)

at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:229)

at org.junit.runners.ParentRunner.run(ParentRunner.java:309)

at org.eclipse.jdt.internal.junit4.runner.JUnit4TestReference.run(JUnit4TestReference.java:50)

at org.eclipse.jdt.internal.junit.runner.TestExecution.run(TestExecution.java:38)

at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:459)

at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:675)

at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.run(RemoteTestRunner.java:382)

at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.main(RemoteTestRunner.java:192)

Caused by: java.net.UnknownHostException: vbasavaraj.att.net: nodename nor servname provided, or not known

at java.net.Inet6AddressImpl.lookupAllHostAddr(Native Method)

at java.net.InetAddress$1.lookupAllHostAddr(InetAddress.java:901)

at java.net.InetAddress.getAddressesFromNameService(InetAddress.java:1293)

at java.net.InetAddress.getLocalHost(InetAddress.java:1469)

... 50 more

536  [main] DEBUG org.apache.hadoop.mapreduce.Cluster  - Picked org.apache.hadoop.mapred.LocalClientProtocolProvider as the ClientProtocolProvider

536  [main] DEBUG org.apache.hadoop.security.UserGroupInformation  - PrivilegedAction as:vbasavaraj (auth:SIMPLE) from:org.apache.hadoop.mapreduce.Cluster.getFileSystem(Cluster.java:161)

537  [main] DEBUG org.apache.hadoop.security.UserGroupInformation  - PrivilegedAction as:vbasavaraj (auth:SIMPLE) from:org.apache.hadoop.mapreduce.Job.submit(Job.java:1282)

Issue Links

• is a clone of: SQOOP-1761 Sqoop2: Unit tests for different Column sub classes Array/Set and Map types