[SPARK-3638] Commons HTTP client dependency conflict in extras/kinesis-asl module

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 1.1.0
• Fix Version/s: 1.1.1, 1.2.0
• Component/s: DStreams, Examples
• Labels: dependencies
• Assignee: Unassigned
• Reporter: Aniket Bhatnagar
• Created: 22/Sep/14 11:00
• Updated: 01/Apr/15 18:46
• Resolved: 02/Oct/14 01:31

Description

Followed instructions as mentioned @ https://github.com/apache/spark/blob/master/docs/streaming-kinesis-integration.md and when running the example, I get the following error:

Caused by: java.lang.NoSuchMethodError: org.apache.http.impl.conn.DefaultClientConnectionOperator.<init>(Lorg/apache/http/conn/scheme/SchemeRegistry;Lorg/apache/http/conn/DnsResolver;)V
        at org.apache.http.impl.conn.PoolingClientConnectionManager.createConnectionOperator(PoolingClientConnectionManager.java:140)
        at org.apache.http.impl.conn.PoolingClientConnectionManager.<init>(PoolingClientConnectionManager.java:114)
        at org.apache.http.impl.conn.PoolingClientConnectionManager.<init>(PoolingClientConnectionManager.java:99)
        at com.amazonaws.http.ConnectionManagerFactory.createPoolingClientConnManager(ConnectionManagerFactory.java:29)
        at com.amazonaws.http.HttpClientFactory.createHttpClient(HttpClientFactory.java:97)
        at com.amazonaws.http.AmazonHttpClient.<init>(AmazonHttpClient.java:181)
        at com.amazonaws.AmazonWebServiceClient.<init>(AmazonWebServiceClient.java:119)
        at com.amazonaws.AmazonWebServiceClient.<init>(AmazonWebServiceClient.java:103)
        at com.amazonaws.services.kinesis.AmazonKinesisClient.<init>(AmazonKinesisClient.java:136)
        at com.amazonaws.services.kinesis.AmazonKinesisClient.<init>(AmazonKinesisClient.java:117)
        at com.amazonaws.services.kinesis.AmazonKinesisAsyncClient.<init>(AmazonKinesisAsyncClient.java:132)

I believe this is due to the dependency conflict as described @ http://mail-archives.apache.org/mod_mbox/spark-dev/201409.mbox/[e-mail]%3E

Issue Links

• is related to: SPARK-6599 Improve usability and reliability of Kinesis stream
• links to: [Github] Pull Request #2489 (aniketbhatnagar)
• links to: [Github] Pull Request #2535 (aniketbhatnagar)