[HIVE-21374] Dependency conflicts on org.apache.httpcomponents:httpcore:jar, leading to invoking unexpected methods

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Duplicate
• Affects Version/s: 2.3.4
• Component/s: Hive
• Target Version/s: 2.3.4
• Assignee: Unassigned
• Reporter: Hello CoCooo
• Created: 02/Mar/19 07:50
• Updated: 21/May/19 12:39
• Resolved: 21/May/19 12:39

Description

Hi! In hive-rel-release-2.3.4\service-rpc,  there are multiple versions of org.apache.httpcomponents:httpcore:jar. As shown in the following dependency tree, according to Maven's dependency management strategy, only org.apache.httpcomponents:httpcore:jar:4.4 can be loaded, and org.apache.httpcomponents:httpcore:jar:4.4.1 will be shadowed.

Your project references the method <org.apache.http.impl.conn.CPool.validate(org.apache.http.pool.PoolEntry)> via the following invocation path, which is included in the shadowed version org.apache.httpcomponents:httpcore:jar:4.4.1. However, this method is missing in the actual loaded version org.apache.httpcomponents:httpcore:jar:4.4. Surprisingly, it will not cause NoSuchMethodError at rumtime.

Invocation path:

// code placeholder
<org.apache.thrift.server.TThreadedSelectorServer: boolean requestInvoke(org.apache.thrift.server.AbstractNonblockingServer$FrameBuffer)> C:\Users\Flipped\.m2\repository\org\apache\thrift\libthrift\0.9.3\libthrift-0.9.3.jar

<org.junit.internal.runners.MethodRoadie$1: void run()> C:\Users\Flipped\.m2\repository\junit\junit\4.11\junit-4.11.jar

<org.apache.http.pool.PoolEntryFuture: java.lang.Object get(long,java.util.concurrent.TimeUnit)> C:\Users\Flipped\.m2\repository\org\apache\httpcomponents\httpcore\4.4.1\httpcore-4.4.1.jar

<org.apache.http.pool.AbstractConnPool$2: java.lang.Object getPoolEntry(long,java.util.concurrent.TimeUnit)> C:\Users\Flipped\.m2\repository\org\apache\httpcomponents\httpcore\4.4.1\httpcore-4.4.1.jar

<org.apache.http.pool.AbstractConnPool$2: org.apache.http.pool.PoolEntry getPoolEntry(long,java.util.concurrent.TimeUnit)> C:\Users\Flipped\.m2\repository\org\apache\httpcomponents\httpcore\4.4.1\httpcore-4.4.1.jar

<org.apache.http.pool.AbstractConnPool: org.apache.http.pool.PoolEntry access$000(org.apache.http.pool.AbstractConnPool,java.lang.Object,java.lang.Object,long,java.util.concurrent.TimeUnit,org.apache.http.pool.PoolEntryFuture)> C:\Users\Flipped\.m2\repository\org\apache\httpcomponents\httpcore\4.4.1\httpcore-4.4.1.jar

<org.apache.http.pool.AbstractConnPool: org.apache.http.pool.PoolEntry getPoolEntryBlocking(java.lang.Object,java.lang.Object,long,java.util.concurrent.TimeUnit,org.apache.http.pool.PoolEntryFuture)> C:\Users\Flipped\.m2\repository\org\apache\httpcomponents\httpcore\4.4.1\httpcore-4.4.1.jar

<org.apache.http.impl.conn.CPool: boolean validate(org.apache.http.pool.PoolEntry)>

By further analyzing, I found that the caller org.apache.thrift.server.TThreadedSelectorServer.requestInvoke(AbstractNonblockingServer$FrameBuffer) would invoke the method AbstractConnPool.validate(PoolEntry) defined in the superclass of org.apache.http.impl.conn.CPool (CPool extends AbstractConnPool) with the same signature of the expected callee, due to dynamic binding mechanism.

Although the actual invoked method belonging to AbstractConnPool has the same method name, same parameter types and return type as the expected method defined in its subclass CPool, but it has different control flows and different behaviors. Maybe it is buggy behavior.

 

Solution:
Use the newer version org.apache.httpcomponents:httpcore:jar:4.4.1 in parent pom file to keep the version consistency.

 

Dependency tree----

[INFO] org.apache.hive:hive-service-rpc:jar:2.3.4

[INFO] +- commons-codec:commons-codec:jar:1.4:compile

[INFO] +- commons-cli:commons-cli:jar:1.2:compile

[INFO] +- tomcat:jasper-compiler:jar:5.5.23:compile

[INFO] |  +- javax.servlet:jsp-api:jar:2.0:compile

[INFO] |  |  - (javax.servlet:servlet-api:jar:2.4:compile - omitted for duplicate)

[INFO] |  - ant:ant:jar:1.6.5:compile

[INFO] +- tomcat:jasper-runtime:jar:5.5.23:compile

[INFO] |  +- javax.servlet:servlet-api:jar:2.4:compile

[INFO] |  - commons-el:commons-el:jar:1.0:compile

[INFO] |     - commons-logging:commons-logging:jar:1.0.3:compile

[INFO] +- org.apache.thrift:libfb303:jar:0.9.3:compile

[INFO] |  - (org.apache.thrift:libthrift:jar:0.9.3:compile - omitted for duplicate)

[INFO] +- org.apache.thrift:libthrift:jar:0.9.3:compile

[INFO] |  +- (org.slf4j:slf4j-api:jar:1.7.10:compile - version managed from 1.7.12; omitted for duplicate)

[INFO] |  +- org.apache.httpcomponents:httpclient:jar:4.4:compile (version managed from 4.4.1)

[INFO] |  |  +- (org.apache.httpcomponents:httpcore:jar:4.4:compile - version managed from 4.4.1; omitted for duplicate)

[INFO] |  |  +- (commons-logging:commons-logging:jar:1.2:compile - omitted for conflict with 1.0.3)

[INFO] |  |  - (commons-codec:commons-codec:jar:1.4:compile - version managed from 1.9; omitted for duplicate)

[INFO] |  - org.apache.httpcomponents:httpcore:jar:4.4:compile

[INFO] +- junit:junit:jar:4.11:test

[INFO] |  - org.hamcrest:hamcrest-core:jar:1.3:test

[INFO] +- org.slf4j:slf4j-api:jar:1.7.10:compile

[INFO] - org.mockito:mockito-all:jar:1.9.5:test

 

 

Thanks!
Best regards,
Coco

Issue Links

• duplicates: HIVE-14455 upgrade httpclient, httpcore to match updated hadoop dependency