[FLINK-3373] Using a newer library of Apache HttpClient than 4.2.6 will get class loading problems

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 1.0.0

Environment

Latest Flink snapshot 1.0

• Assignee: Stephan Ewen
• Reporter: Jakob Sultan Ericsson
• Created: 09/Feb/16 08:40
• Updated: 02/Oct/19 17:44
• Resolved: 10/Feb/16 14:50

Description

When I trying to use Apache HTTP client 4.5.1 in my flink job it will crash with NoClassDefFound.
This has to do that it load some classes from provided httpclient 4.2.5/6 in core flink.

17:05:56,193 INFO  org.apache.flink.runtime.taskmanager.Task                     - DuplicateFilter -> InstallKeyLookup (11/16) switched to FAILED with exception.
java.lang.NoSuchFieldError: INSTANCE
        at org.apache.http.conn.ssl.SSLConnectionSocketFactory.<clinit>(SSLConnectionSocketFactory.java:144)
        at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.getDefaultRegistry(PoolingHttpClientConnectionManager.java:109)
        at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.<init>(PoolingHttpClientConnectionManager.java:116)
        ...<internal classes>
        at org.apache.flink.api.common.functions.util.FunctionUtils.openFunction(FunctionUtils.java:36)
        at org.apache.flink.streaming.api.operators.AbstractUdfStreamOperator.open(AbstractUdfStreamOperator.java:89)
        at org.apache.flink.streaming.runtime.tasks.StreamTask.openAllOperators(StreamTask.java:305)
        at org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:227)
        at org.apache.flink.runtime.taskmanager.Task.run(Task.java:566)
        at java.lang.Thread.run(Thread.java:745)

SSLConnectionSocketFactory and finds an earlier version of the AllowAllHostnameVerifier that does have the INSTANCE variable (instance variable was probably added in 4.3).

jar tvf lib/flink-dist-1.0-SNAPSHOT.jar |grep AllowAllHostnameVerifier  
   791 Thu Dec 17 09:55:46 CET 2015 org/apache/http/conn/ssl/AllowAllHostnameVerifier.class

Solutions would be:

• Fix the classloader so that my custom job does not conflict with internal flink-core classes... pretty hard
• Remove the dependency somehow.

Issue Links

• relates to: FLINK-4587 Yet another java.lang.NoSuchFieldError: INSTANCE