Details
-
Sub-task
-
Status: Resolved
-
Major
-
Resolution: Works for Me
-
None
-
None
-
None
Description
Instantly reproducible with ./compose/ozone docker-compose.yml with JDK 17 image (follow HDDS-8247):
2023-03-22 16:00:35 ************************************************************/ 2023-03-22 16:00:35 2023-03-22 23:00:35,102 [main] INFO recon.ReconServer: registered UNIX signal handlers for [TERM, HUP, INT] 2023-03-22 16:00:35 Exception in thread "main" java.lang.ExceptionInInitializerError 2023-03-22 16:00:35 at com.google.inject.internal.cglib.reflect.$FastClassEmitter.<init>(FastClassEmitter.java:67) 2023-03-22 16:00:35 at com.google.inject.internal.cglib.reflect.$FastClass$Generator.generateClass(FastClass.java:72) 2023-03-22 16:00:35 at com.google.inject.internal.cglib.core.$DefaultGeneratorStrategy.generate(DefaultGeneratorStrategy.java:25) 2023-03-22 16:00:35 at com.google.inject.internal.cglib.core.$AbstractClassGenerator.create(AbstractClassGenerator.java:216) 2023-03-22 16:00:35 at com.google.inject.internal.cglib.reflect.$FastClass$Generator.create(FastClass.java:64) 2023-03-22 16:00:35 at com.google.inject.internal.BytecodeGen.newFastClass(BytecodeGen.java:204) 2023-03-22 16:00:35 at com.google.inject.internal.ProviderMethod$FastClassProviderMethod.<init>(ProviderMethod.java:256) 2023-03-22 16:00:35 at com.google.inject.internal.ProviderMethod.create(ProviderMethod.java:71) 2023-03-22 16:00:35 at com.google.inject.internal.ProviderMethodsModule.createProviderMethod(ProviderMethodsModule.java:275) 2023-03-22 16:00:35 at com.google.inject.internal.ProviderMethodsModule.getProviderMethods(ProviderMethodsModule.java:144) 2023-03-22 16:00:35 at com.google.inject.internal.ProviderMethodsModule.configure(ProviderMethodsModule.java:123) 2023-03-22 16:00:35 at com.google.inject.spi.Elements$RecordingBinder.install(Elements.java:340) 2023-03-22 16:00:35 at com.google.inject.spi.Elements$RecordingBinder.install(Elements.java:349) 2023-03-22 16:00:35 at com.google.inject.AbstractModule.install(AbstractModule.java:122) 2023-03-22 16:00:35 at org.apache.hadoop.ozone.recon.ReconControllerModule.configure(ReconControllerModule.java:106) 2023-03-22 16:00:35 at com.google.inject.AbstractModule.configure(AbstractModule.java:62) 2023-03-22 16:00:35 at com.google.inject.spi.Elements$RecordingBinder.install(Elements.java:340) 2023-03-22 16:00:35 at com.google.inject.spi.Elements.getElements(Elements.java:110) 2023-03-22 16:00:35 at com.google.inject.internal.InjectorShell$Builder.build(InjectorShell.java:138) 2023-03-22 16:00:35 at com.google.inject.internal.InternalInjectorCreator.build(InternalInjectorCreator.java:104) 2023-03-22 16:00:35 at com.google.inject.Guice.createInjector(Guice.java:96) 2023-03-22 16:00:35 at com.google.inject.Guice.createInjector(Guice.java:73) 2023-03-22 16:00:35 at com.google.inject.Guice.createInjector(Guice.java:62) 2023-03-22 16:00:35 at org.apache.hadoop.ozone.recon.ReconServer.call(ReconServer.java:100) 2023-03-22 16:00:35 at org.apache.hadoop.ozone.recon.ReconServer.call(ReconServer.java:65) 2023-03-22 16:00:35 at picocli.CommandLine.executeUserObject(CommandLine.java:1953) 2023-03-22 16:00:35 at picocli.CommandLine.access$1300(CommandLine.java:145) 2023-03-22 16:00:35 at picocli.CommandLine$RunLast.executeUserObjectOfLastSubcommandWithSameParent(CommandLine.java:2352) 2023-03-22 16:00:35 at picocli.CommandLine$RunLast.handle(CommandLine.java:2346) 2023-03-22 16:00:35 at picocli.CommandLine$RunLast.handle(CommandLine.java:2311) 2023-03-22 16:00:35 at picocli.CommandLine$AbstractParseResultHandler.execute(CommandLine.java:2179) 2023-03-22 16:00:35 at picocli.CommandLine.execute(CommandLine.java:2078) 2023-03-22 16:00:35 at org.apache.hadoop.hdds.cli.GenericCli.execute(GenericCli.java:100) 2023-03-22 16:00:35 at org.apache.hadoop.hdds.cli.GenericCli.run(GenericCli.java:91) 2023-03-22 16:00:35 at org.apache.hadoop.ozone.recon.ReconServer.main(ReconServer.java:87) 2023-03-22 16:00:35 Caused by: java.lang.reflect.InaccessibleObjectException: Unable to make protected final java.lang.Class java.lang.ClassLoader.defineClass(java.lang.String,byte[],int,int,java.security.ProtectionDomain) throws java.lang.ClassFormatError accessible: module java.base does not "opens java.lang" to unnamed module @6e20b53a 2023-03-22 16:00:35 at java.base/java.lang.reflect.AccessibleObject.checkCanSetAccessible(AccessibleObject.java:354) 2023-03-22 16:00:35 at java.base/java.lang.reflect.AccessibleObject.checkCanSetAccessible(AccessibleObject.java:297) 2023-03-22 16:00:35 at java.base/java.lang.reflect.Method.checkCanSetAccessible(Method.java:199) 2023-03-22 16:00:35 at java.base/java.lang.reflect.Method.setAccessible(Method.java:193) 2023-03-22 16:00:35 at com.google.inject.internal.cglib.core.$ReflectUtils$2.run(ReflectUtils.java:56) 2023-03-22 16:00:35 at java.base/java.security.AccessController.doPrivileged(AccessController.java:318) 2023-03-22 16:00:35 at com.google.inject.internal.cglib.core.$ReflectUtils.<clinit>(ReflectUtils.java:46) 2023-03-22 16:00:35 ... 35 more 2023-03-22 16:00:35 2023-03-22 23:00:35,365 [shutdown-hook-0] INFO recon.ReconServer: SHUTDOWN_MSG: 2023-03-22 16:00:35 /************************************************************ 2023-03-22 16:00:35 SHUTDOWN_MSG: Shutting down ReconServer at db96dd3198fc/172.22.0.4 2023-03-22 16:00:35 ************************************************************/
Might have something to do with the dependency injection library?
https://github.com/google/guice/issues/1536#issuecomment-963621665
Try bumping up Guice (HDDS-8263).
Update 1:
With HDDS-8263 done and guice bumped, Recon is crashing with a different exception on startup (still using the same dev docker with JDK 17):
2023-03-25 01:05:35,892 [Listener at 0.0.0.0/9891] INFO recon.ReconServer: Recon server initialized successfully! 2023-03-25 01:05:35,892 [Listener at 0.0.0.0/9891] INFO recon.ReconServer: Starting Recon server 2023-03-25 01:05:35,928 [Listener at 0.0.0.0/9891] INFO impl.MetricsConfig: Loaded properties from hadoop-metrics2.properties 2023-03-25 01:05:35,934 [Listener at 0.0.0.0/9891] INFO impl.MetricsSystemImpl: Scheduled Metric snapshot period at 10 second(s). 2023-03-25 01:05:35,934 [Listener at 0.0.0.0/9891] INFO impl.MetricsSystemImpl: Recon metrics system started 2023-03-25 01:05:36,056 [Listener at 0.0.0.0/9891] INFO http.HttpServer2: Jetty bound to port 9888 2023-03-25 01:05:36,057 [Listener at 0.0.0.0/9891] INFO server.Server: jetty-9.4.49.v20220914; built: 2022-09-14T01:07:36.601Z; git: 4231a3b2e4cb8548a412a789936d640a97b1aa0a; jvm 17.0.6+10-LTS 2023-03-25 01:05:36,071 [Listener at 0.0.0.0/9891] INFO server.session: DefaultSessionIdManager workerName=node0 2023-03-25 01:05:36,071 [Listener at 0.0.0.0/9891] INFO server.session: No SessionScavenger set, using defaults 2023-03-25 01:05:36,072 [Listener at 0.0.0.0/9891] INFO server.session: node0 Scavenging every 660000ms 2023-03-25 01:05:36,080 [Listener at 0.0.0.0/9891] INFO handler.ContextHandler: Started o.e.j.s.ServletContextHandler@37e7c4cc{logs,/logs,file:///var/log/hadoop/,AVAILABLE} 2023-03-25 01:05:36,080 [Listener at 0.0.0.0/9891] INFO handler.ContextHandler: Started o.e.j.s.ServletContextHandler@6aea99e7{static,/static,jar:file:/opt/hadoop/share/ozone/lib/ozone-recon-1.4.0-SNAPSHOT.jar!/webapps/static,AVAILABLE} Mar 25, 2023 1:05:37 AM com.sun.xml.bind.v2.runtime.reflect.opt.Injector <clinit> SEVERE: null java.security.PrivilegedActionException: java.lang.NoSuchMethodException: sun.misc.Unsafe.defineClass(java.lang.String,[B,int,int,java.lang.ClassLoader,java.security.ProtectionDomain) at java.base/java.security.AccessController.doPrivileged(AccessController.java:573) at com.sun.xml.bind.v2.runtime.reflect.opt.Injector.<clinit>(Injector.java:197) at com.sun.xml.bind.v2.runtime.reflect.opt.AccessorInjector.prepare(AccessorInjector.java:81) at com.sun.xml.bind.v2.runtime.reflect.opt.OptimizedAccessorFactory.get(OptimizedAccessorFactory.java:179) at com.sun.xml.bind.v2.runtime.reflect.Accessor$FieldReflection.optimize(Accessor.java:285) at com.sun.xml.bind.v2.runtime.property.ArrayProperty.<init>(ArrayProperty.java:68) at com.sun.xml.bind.v2.runtime.property.ArrayERProperty.<init>(ArrayERProperty.java:88) at com.sun.xml.bind.v2.runtime.property.ArrayElementProperty.<init>(ArrayElementProperty.java:100) at com.sun.xml.bind.v2.runtime.property.ArrayElementNodeProperty.<init>(ArrayElementNodeProperty.java:62) at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:77) at java.base/jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.base/java.lang.reflect.Constructor.newInstanceWithCaller(Constructor.java:499) at java.base/java.lang.reflect.Constructor.newInstance(Constructor.java:480) at com.sun.xml.bind.v2.runtime.property.PropertyFactory.create(PropertyFactory.java:128) at com.sun.xml.bind.v2.runtime.ClassBeanInfoImpl.<init>(ClassBeanInfoImpl.java:181) at com.sun.xml.bind.v2.runtime.JAXBContextImpl.getOrCreate(JAXBContextImpl.java:514) at com.sun.xml.bind.v2.runtime.JAXBContextImpl.<init>(JAXBContextImpl.java:331) at com.sun.xml.bind.v2.runtime.JAXBContextImpl.<init>(JAXBContextImpl.java:139) at com.sun.xml.bind.v2.runtime.JAXBContextImpl$JAXBContextBuilder.build(JAXBContextImpl.java:1156) at com.sun.xml.bind.v2.ContextFactory.createContext(ContextFactory.java:165) at com.sun.xml.bind.v2.ContextFactory.createContext(ContextFactory.java:289) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77) at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.base/java.lang.reflect.Method.invoke(Method.java:568) at javax.xml.bind.ContextFinder.newInstance(ContextFinder.java:217) at javax.xml.bind.ContextFinder.newInstance(ContextFinder.java:175) at javax.xml.bind.ContextFinder.find(ContextFinder.java:353) at javax.xml.bind.JAXBContext.newInstance(JAXBContext.java:508) at javax.xml.bind.JAXBContext.newInstance(JAXBContext.java:465) at org.glassfish.jersey.server.wadl.internal.WadlApplicationContextImpl.getJAXBContextFromWadlGenerator(WadlApplicationContextImpl.java:121) at org.glassfish.jersey.server.wadl.internal.WadlApplicationContextImpl.isJaxbImplAvailable(WadlApplicationContextImpl.java:270) at org.glassfish.jersey.server.wadl.WadlFeature.configure(WadlFeature.java:65) at org.glassfish.jersey.model.internal.CommonConfig.configureFeatures(CommonConfig.java:728) at org.glassfish.jersey.model.internal.CommonConfig.configureMetaProviders(CommonConfig.java:647) at org.glassfish.jersey.server.ResourceConfig.configureMetaProviders(ResourceConfig.java:823) at org.glassfish.jersey.server.ApplicationHandler.initialize(ApplicationHandler.java:328) at org.glassfish.jersey.server.ApplicationHandler.lambda$initialize$1(ApplicationHandler.java:293) at org.glassfish.jersey.internal.Errors.process(Errors.java:292) at org.glassfish.jersey.internal.Errors.process(Errors.java:274) at org.glassfish.jersey.internal.Errors.processWithException(Errors.java:232) at org.glassfish.jersey.server.ApplicationHandler.initialize(ApplicationHandler.java:292) at org.glassfish.jersey.server.ApplicationHandler.<init>(ApplicationHandler.java:259) at org.glassfish.jersey.servlet.WebComponent.<init>(WebComponent.java:311) at org.glassfish.jersey.servlet.ServletContainer.init(ServletContainer.java:154) at org.glassfish.jersey.servlet.ServletContainer.init(ServletContainer.java:347) at javax.servlet.GenericServlet.init(GenericServlet.java:244) at com.google.inject.servlet.ServletDefinition.init(ServletDefinition.java:119) at com.google.inject.servlet.ManagedServletPipeline.init(ManagedServletPipeline.java:80) at com.google.inject.servlet.ManagedFilterPipeline.initPipeline(ManagedFilterPipeline.java:102) at com.google.inject.servlet.GuiceFilter.init(GuiceFilter.java:232) at org.eclipse.jetty.servlet.FilterHolder.initialize(FilterHolder.java:140) at org.eclipse.jetty.servlet.ServletHandler.lambda$initialize$0(ServletHandler.java:750) at java.base/java.util.Spliterators$ArraySpliterator.forEachRemaining(Spliterators.java:992) at java.base/java.util.stream.Streams$ConcatSpliterator.forEachRemaining(Streams.java:734) at java.base/java.util.stream.ReferencePipeline$Head.forEach(ReferencePipeline.java:762) at org.eclipse.jetty.servlet.ServletHandler.initialize(ServletHandler.java:774) at org.eclipse.jetty.servlet.ServletContextHandler.startContext(ServletContextHandler.java:379) at org.eclipse.jetty.webapp.WebAppContext.startWebapp(WebAppContext.java:1449) at org.eclipse.jetty.webapp.WebAppContext.startContext(WebAppContext.java:1414) at org.eclipse.jetty.server.handler.ContextHandler.doStart(ContextHandler.java:916) at org.eclipse.jetty.servlet.ServletContextHandler.doStart(ServletContextHandler.java:288) at org.eclipse.jetty.webapp.WebAppContext.doStart(WebAppContext.java:524) at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:73) at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:169) at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:117) at org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:97) at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:73) at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:169) at org.eclipse.jetty.server.Server.start(Server.java:423) at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:110) at org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:97) at org.eclipse.jetty.server.Server.doStart(Server.java:387) at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:73) at org.apache.hadoop.hdds.server.http.HttpServer2.start(HttpServer2.java:1198) at org.apache.hadoop.hdds.server.http.BaseHttpServer.start(BaseHttpServer.java:312) at org.apache.hadoop.ozone.recon.ReconServer.start(ReconServer.java:238) at org.apache.hadoop.ozone.recon.ReconServer.call(ReconServer.java:151) at org.apache.hadoop.ozone.recon.ReconServer.call(ReconServer.java:65) at picocli.CommandLine.executeUserObject(CommandLine.java:1953) at picocli.CommandLine.access$1300(CommandLine.java:145) at picocli.CommandLine$RunLast.executeUserObjectOfLastSubcommandWithSameParent(CommandLine.java:2352) at picocli.CommandLine$RunLast.handle(CommandLine.java:2346) at picocli.CommandLine$RunLast.handle(CommandLine.java:2311) at picocli.CommandLine$AbstractParseResultHandler.execute(CommandLine.java:2179) at picocli.CommandLine.execute(CommandLine.java:2078) at org.apache.hadoop.hdds.cli.GenericCli.execute(GenericCli.java:100) at org.apache.hadoop.hdds.cli.GenericCli.run(GenericCli.java:91) at org.apache.hadoop.ozone.recon.ReconServer.main(ReconServer.java:87) Caused by: java.lang.NoSuchMethodException: sun.misc.Unsafe.defineClass(java.lang.String,[B,int,int,java.lang.ClassLoader,java.security.ProtectionDomain) at java.base/java.lang.Class.getMethod(Class.java:2227) at com.sun.xml.bind.v2.runtime.reflect.opt.Injector$3.run(Injector.java:201) at com.sun.xml.bind.v2.runtime.reflect.opt.Injector$3.run(Injector.java:197) at java.base/java.security.AccessController.doPrivileged(AccessController.java:569) ... 89 more 2023-03-25 01:05:37,126 [Listener at 0.0.0.0/9891] WARN webapp.WebAppContext: Failed startup of context o.e.j.w.WebAppContext@76bdfc19{recon,/,file:///data/metadata/webserver/jetty-0_0_0_0-9888-ozone-recon-1_4_0-SNAPSHOT_jar-_-any-4970389891519408937/webapp/,UNAVAILABLE}{jar:file:/opt/hadoop/share/ozone/lib/ozone-recon-1.4.0-SNAPSHOT.jar!/webapps/recon} java.lang.NullPointerException: Cannot invoke "java.lang.reflect.Method.invoke(Object, Object[])" because "com.sun.xml.bind.v2.runtime.reflect.opt.Injector.defineClass" is null at com.sun.xml.bind.v2.runtime.reflect.opt.Injector.inject(Injector.java:311) at com.sun.xml.bind.v2.runtime.reflect.opt.Injector.inject(Injector.java:97) at com.sun.xml.bind.v2.runtime.reflect.opt.AccessorInjector.prepare(AccessorInjector.java:87) at com.sun.xml.bind.v2.runtime.reflect.opt.OptimizedAccessorFactory.get(OptimizedAccessorFactory.java:179) at com.sun.xml.bind.v2.runtime.reflect.Accessor$FieldReflection.optimize(Accessor.java:285) at com.sun.xml.bind.v2.runtime.property.ArrayProperty.<init>(ArrayProperty.java:68) at com.sun.xml.bind.v2.runtime.property.ArrayERProperty.<init>(ArrayERProperty.java:88) at com.sun.xml.bind.v2.runtime.property.ArrayElementProperty.<init>(ArrayElementProperty.java:100) at com.sun.xml.bind.v2.runtime.property.ArrayElementNodeProperty.<init>(ArrayElementNodeProperty.java:62) at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:77) at java.base/jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.base/java.lang.reflect.Constructor.newInstanceWithCaller(Constructor.java:499) at java.base/java.lang.reflect.Constructor.newInstance(Constructor.java:480) at com.sun.xml.bind.v2.runtime.property.PropertyFactory.create(PropertyFactory.java:128) at com.sun.xml.bind.v2.runtime.ClassBeanInfoImpl.<init>(ClassBeanInfoImpl.java:181) at com.sun.xml.bind.v2.runtime.JAXBContextImpl.getOrCreate(JAXBContextImpl.java:514) at com.sun.xml.bind.v2.runtime.JAXBContextImpl.<init>(JAXBContextImpl.java:331) at com.sun.xml.bind.v2.runtime.JAXBContextImpl.<init>(JAXBContextImpl.java:139) at com.sun.xml.bind.v2.runtime.JAXBContextImpl$JAXBContextBuilder.build(JAXBContextImpl.java:1156) at com.sun.xml.bind.v2.ContextFactory.createContext(ContextFactory.java:165) at com.sun.xml.bind.v2.ContextFactory.createContext(ContextFactory.java:289) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77) at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.base/java.lang.reflect.Method.invoke(Method.java:568) at javax.xml.bind.ContextFinder.newInstance(ContextFinder.java:217) at javax.xml.bind.ContextFinder.newInstance(ContextFinder.java:175) at javax.xml.bind.ContextFinder.find(ContextFinder.java:353) at javax.xml.bind.JAXBContext.newInstance(JAXBContext.java:508) at javax.xml.bind.JAXBContext.newInstance(JAXBContext.java:465) at org.glassfish.jersey.server.wadl.internal.WadlApplicationContextImpl.getJAXBContextFromWadlGenerator(WadlApplicationContextImpl.java:121) at org.glassfish.jersey.server.wadl.internal.WadlApplicationContextImpl.isJaxbImplAvailable(WadlApplicationContextImpl.java:270) at org.glassfish.jersey.server.wadl.WadlFeature.configure(WadlFeature.java:65) at org.glassfish.jersey.model.internal.CommonConfig.configureFeatures(CommonConfig.java:728) at org.glassfish.jersey.model.internal.CommonConfig.configureMetaProviders(CommonConfig.java:647) at org.glassfish.jersey.server.ResourceConfig.configureMetaProviders(ResourceConfig.java:823) at org.glassfish.jersey.server.ApplicationHandler.initialize(ApplicationHandler.java:328) at org.glassfish.jersey.server.ApplicationHandler.lambda$initialize$1(ApplicationHandler.java:293) at org.glassfish.jersey.internal.Errors.process(Errors.java:292) at org.glassfish.jersey.internal.Errors.process(Errors.java:274) at org.glassfish.jersey.internal.Errors.processWithException(Errors.java:232) at org.glassfish.jersey.server.ApplicationHandler.initialize(ApplicationHandler.java:292) at org.glassfish.jersey.server.ApplicationHandler.<init>(ApplicationHandler.java:259) at org.glassfish.jersey.servlet.WebComponent.<init>(WebComponent.java:311) at org.glassfish.jersey.servlet.ServletContainer.init(ServletContainer.java:154) at org.glassfish.jersey.servlet.ServletContainer.init(ServletContainer.java:347) at javax.servlet.GenericServlet.init(GenericServlet.java:244) at com.google.inject.servlet.ServletDefinition.init(ServletDefinition.java:119) at com.google.inject.servlet.ManagedServletPipeline.init(ManagedServletPipeline.java:80) at com.google.inject.servlet.ManagedFilterPipeline.initPipeline(ManagedFilterPipeline.java:102) at com.google.inject.servlet.GuiceFilter.init(GuiceFilter.java:232) at org.eclipse.jetty.servlet.FilterHolder.initialize(FilterHolder.java:140) at org.eclipse.jetty.servlet.ServletHandler.lambda$initialize$0(ServletHandler.java:750) at java.base/java.util.Spliterators$ArraySpliterator.forEachRemaining(Spliterators.java:992) at java.base/java.util.stream.Streams$ConcatSpliterator.forEachRemaining(Streams.java:734) at java.base/java.util.stream.ReferencePipeline$Head.forEach(ReferencePipeline.java:762) at org.eclipse.jetty.servlet.ServletHandler.initialize(ServletHandler.java:774) at org.eclipse.jetty.servlet.ServletContextHandler.startContext(ServletContextHandler.java:379) at org.eclipse.jetty.webapp.WebAppContext.startWebapp(WebAppContext.java:1449) at org.eclipse.jetty.webapp.WebAppContext.startContext(WebAppContext.java:1414) at org.eclipse.jetty.server.handler.ContextHandler.doStart(ContextHandler.java:916) at org.eclipse.jetty.servlet.ServletContextHandler.doStart(ServletContextHandler.java:288) at org.eclipse.jetty.webapp.WebAppContext.doStart(WebAppContext.java:524) at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:73) at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:169) at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:117) at org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:97) at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:73) at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:169) at org.eclipse.jetty.server.Server.start(Server.java:423) at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:110) at org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:97) at org.eclipse.jetty.server.Server.doStart(Server.java:387) at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:73) at org.apache.hadoop.hdds.server.http.HttpServer2.start(HttpServer2.java:1198) at org.apache.hadoop.hdds.server.http.BaseHttpServer.start(BaseHttpServer.java:312) at org.apache.hadoop.ozone.recon.ReconServer.start(ReconServer.java:238) at org.apache.hadoop.ozone.recon.ReconServer.call(ReconServer.java:151) at org.apache.hadoop.ozone.recon.ReconServer.call(ReconServer.java:65) at picocli.CommandLine.executeUserObject(CommandLine.java:1953) at picocli.CommandLine.access$1300(CommandLine.java:145) at picocli.CommandLine$RunLast.executeUserObjectOfLastSubcommandWithSameParent(CommandLine.java:2352) at picocli.CommandLine$RunLast.handle(CommandLine.java:2346) at picocli.CommandLine$RunLast.handle(CommandLine.java:2311) at picocli.CommandLine$AbstractParseResultHandler.execute(CommandLine.java:2179) at picocli.CommandLine.execute(CommandLine.java:2078) at org.apache.hadoop.hdds.cli.GenericCli.execute(GenericCli.java:100) at org.apache.hadoop.hdds.cli.GenericCli.run(GenericCli.java:91) at org.apache.hadoop.ozone.recon.ReconServer.main(ReconServer.java:87)
This looks to be same source of problem as S3g: HDDS-8249. Might need to bump jetty to at least 10.x (Tried 9.4.51.v20230217, same issue). However, jetty 10.x/11.x requires the use of JDK 11 to compile.