[SOLR-10951] Update Hadoop dependencies to 2.8.1, so Solr works with Java 9

• Type: Task
• Status: Closed
• Priority: Major
• Resolution: Won't Fix
• Affects Version/s: 6.6, 7.0
• Fix Version/s: 7.0
• Component/s: Hadoop Integration, hdfs
• Labels: Java9
• Assignee: Uwe Schindler
• Reporter: Uwe Schindler
• Created: 24/Jun/17 13:05
• Updated: 08/Jun/19 15:22
• Resolved: 20/Aug/17 11:11

Description

See issue: HADOOP-14586

Since Java 9 build 175 (the first Java 9 relaese candidate), Hadoop integration fails in Java 9:

   [junit4]   2> 129956 ERROR (jetty-launcher-232-thread-2) [    ] o.a.s.c.SolrCore null:java.lang.ExceptionInInitializerError
   [junit4]   2> 	at org.apache.hadoop.util.StringUtils.<clinit>(StringUtils.java:80)
   [junit4]   2> 	at org.apache.hadoop.conf.Configuration.getBoolean(Configuration.java:1437)
   [junit4]   2> 	at org.apache.hadoop.security.token.delegation.web.DelegationTokenManager.<init>(DelegationTokenManager.java:115)
   [junit4]   2> 	at org.apache.hadoop.security.token.delegation.web.DelegationTokenAuthenticationHandler.initTokenManager(DelegationTokenAuthenticationHandler.java:148)
   [junit4]   2> 	at org.apache.hadoop.security.token.delegation.web.DelegationTokenAuthenticationHandler.init(DelegationTokenAuthenticationHandler.java:118)
   [junit4]   2> 	at org.apache.hadoop.security.authentication.server.AuthenticationFilter.initializeAuthHandler(AuthenticationFilter.java:238)
   [junit4]   2> 	at org.apache.hadoop.security.token.delegation.web.DelegationTokenAuthenticationFilter.initializeAuthHandler(DelegationTokenAuthenticationFilter.java:209)
   [junit4]   2> 	at org.apache.solr.security.HadoopAuthFilter.initializeAuthHandler(HadoopAuthFilter.java:120)
   [junit4]   2> 	at org.apache.hadoop.security.authentication.server.AuthenticationFilter.init(AuthenticationFilter.java:227)
   [junit4]   2> 	at org.apache.hadoop.security.token.delegation.web.DelegationTokenAuthenticationFilter.init(DelegationTokenAuthenticationFilter.java:175)
   [junit4]   2> 	at org.apache.solr.security.HadoopAuthFilter.init(HadoopAuthFilter.java:68)
   [junit4]   2> 	at org.apache.solr.security.HadoopAuthPlugin.init(HadoopAuthPlugin.java:142)
   [junit4]   2> 	at org.apache.solr.core.CoreContainer.initializeAuthenticationPlugin(CoreContainer.java:360)
   [junit4]   2> 	at org.apache.solr.core.CoreContainer.reloadSecurityProperties(CoreContainer.java:684)
   [junit4]   2> 	at org.apache.solr.core.CoreContainer.load(CoreContainer.java:522)
   [junit4]   2> 	at org.apache.solr.servlet.SolrDispatchFilter.createCoreContainer(SolrDispatchFilter.java:257)
   [junit4]   2> 	at org.apache.solr.servlet.SolrDispatchFilter.init(SolrDispatchFilter.java:177)
   [junit4]   2> 	at org.eclipse.jetty.servlet.FilterHolder.initialize(FilterHolder.java:137)
   [junit4]   2> 	at org.eclipse.jetty.servlet.ServletHandler.initialize(ServletHandler.java:873)
   [junit4]   2> 	at org.eclipse.jetty.servlet.ServletHandler.updateMappings(ServletHandler.java:1565)
   [junit4]   2> 	at org.eclipse.jetty.servlet.ServletHandler.setFilterMappings(ServletHandler.java:1599)
   [junit4]   2> 	at org.eclipse.jetty.servlet.ServletHandler.addFilterMapping(ServletHandler.java:1285)
   [junit4]   2> 	at org.eclipse.jetty.servlet.ServletHandler.addFilterWithMapping(ServletHandler.java:1130)
   [junit4]   2> 	at org.eclipse.jetty.servlet.ServletContextHandler.addFilter(ServletContextHandler.java:448)
   [junit4]   2> 	at org.apache.solr.client.solrj.embedded.JettySolrRunner$1.lifeCycleStarted(JettySolrRunner.java:306)
   [junit4]   2> 	at org.eclipse.jetty.util.component.AbstractLifeCycle.setStarted(AbstractLifeCycle.java:179)
   [junit4]   2> 	at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:69)
   [junit4]   2> 	at org.apache.solr.client.solrj.embedded.JettySolrRunner.start(JettySolrRunner.java:394)
   [junit4]   2> 	at org.apache.solr.client.solrj.embedded.JettySolrRunner.start(JettySolrRunner.java:367)
   [junit4]   2> 	at org.apache.solr.cloud.MiniSolrCloudCluster.startJettySolrRunner(MiniSolrCloudCluster.java:382)
   [junit4]   2> 	at org.apache.solr.cloud.MiniSolrCloudCluster.lambda$new$0(MiniSolrCloudCluster.java:245)
   [junit4]   2> 	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
   [junit4]   2> 	at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:188)
   [junit4]   2> 	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1161)
   [junit4]   2> 	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
   [junit4]   2> 	at java.base/java.lang.Thread.run(Thread.java:844)
   [junit4]   2> Caused by: java.lang.StringIndexOutOfBoundsException: begin 0, end 3, length 1
   [junit4]   2> 	at java.base/java.lang.String.checkBoundsBeginEnd(String.java:3116)
   [junit4]   2> 	at java.base/java.lang.String.substring(String.java:1885)
   [junit4]   2> 	at org.apache.hadoop.util.Shell.<clinit>(Shell.java:51)
   [junit4]   2> 	... 36 more
   [junit4]   2> 

This means we can for now only disable all tests using Hadoop, if Java 9 is detected.

On Policeman Jenkins the HDFS integration is already disabled, but there are 2 more tests that fail:

• org.apache.solr.security.hadoop.TestDelegationWithHadoopAuth
• org.apache.solr.cloud.TestSolrCloudWithDelegationTokens

I will add an assumeFalse for a while, until Hadoop fixed the issue.

Issue Links

• is broken by: HADOOP-14586 StringIndexOutOfBoundsException breaks org.apache.hadoop.util.Shell on 2.7.x with Java 9
• supercedes: SOLR-10966 Add workaround for Hadoop-Common 2.7.2 incompatibility with Java 9