Uploaded image for project: 'Apache Storm'
  1. Apache Storm
  2. STORM-3728

Workers are not able to connect to Pacemaker if pacemaker.auth.method is KERBEROS

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • 2.0.0, 2.1.0, 2.2.0
    • 2.3.0
    • None
    • None

    Description

      When pacemaker.auth.method is KERBEROS,  worker will fail to connect to KERBEROS because of exceptions like the following:
       

      2020-12-21 20:07:00.786 o.a.s.c.PaceMakerStateStorage executor-heartbeat-timer [ERROR] Timed out waiting for channel ready. Failed to set_worker_hb. Will make 2 more attempts.
      2020-12-21 20:07:00.902 o.a.s.m.n.KerberosSaslClientHandler openstorm3blue-n10.blue.ygrid.yahoo.com-pm-1 [INFO] Connection established from /10.215.73.209:45548 to openstorm3blue-n10.blue.ygrid.yahoo.com/10.215.79.152:6699
      2020-12-21 20:07:00.903 o.a.s.m.n.KerberosSaslNettyClient openstorm3blue-n10.blue.ygrid.yahoo.com-pm-1 [INFO] Creating Kerberos Client.
      2020-12-21 20:07:00.906 o.a.s.m.n.KerberosSaslNettyClient openstorm3blue-n10.blue.ygrid.yahoo.com-pm-1 [INFO] Kerberos Client Callback Handler got callback: class javax.security.auth.callback.PasswordCallback
      2020-12-21 20:07:00.906 o.a.s.m.n.Login openstorm3blue-n10.blue.ygrid.yahoo.com-pm-1 [ERROR] Login using jaas conf /home/y/lib/storm/current/conf/storm_jaas.conf failed
      2020-12-21 20:07:00.906 o.a.s.m.n.KerberosSaslNettyClient openstorm3blue-n10.blue.ygrid.yahoo.com-pm-1 [ERROR] Client failed to login in principal:javax.security.auth.login.LoginException: No password provided
      javax.security.auth.login.LoginException: No password provided
              at com.sun.security.auth.module.Krb5LoginModule.promptForPass(Krb5LoginModule.java:923) ~[?:1.8.0_262]
              at com.sun.security.auth.module.Krb5LoginModule.attemptAuthentication(Krb5LoginModule.java:764) ~[?:1.8.0_262]
              at com.sun.security.auth.module.Krb5LoginModule.login(Krb5LoginModule.java:618) ~[?:1.8.0_262]
              at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.8.0_262]
              at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:1.8.0_262]
              at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_262]
              at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_262]
              at javax.security.auth.login.LoginContext.invoke(LoginContext.java:755) ~[?:1.8.0_262]
              at javax.security.auth.login.LoginContext.access$000(LoginContext.java:195) ~[?:1.8.0_262]
              at javax.security.auth.login.LoginContext$4.run(LoginContext.java:682) ~[?:1.8.0_262]
              at javax.security.auth.login.LoginContext$4.run(LoginContext.java:680) ~[?:1.8.0_262]
              at java.security.AccessController.doPrivileged(Native Method) ~[?:1.8.0_262]
              at javax.security.auth.login.LoginContext.invokePriv(LoginContext.java:680) ~[?:1.8.0_262]
              at javax.security.auth.login.LoginContext.login(LoginContext.java:587) ~[?:1.8.0_262]
              at org.apache.storm.messaging.netty.Login.login(Login.java:301) ~[storm-client-2.3.0.y.jar:2.3.0-SNAPSHOT]
              at org.apache.storm.messaging.netty.Login.<init>(Login.java:83) ~[storm-client-2.3.0.y.jar:2.3.0-SNAPSHOT]
              at org.apache.storm.messaging.netty.KerberosSaslNettyClient.<init>(KerberosSaslNettyClient.java:66) [storm-client-2.3.0.y.jar:2.3.0-SNAPSHOT]
              at org.apache.storm.messaging.netty.KerberosSaslClientHandler.channelActive(KerberosSaslClientHandler.java:59) [storm-client-2.3.0.y.jar:2.3.0-SNAPSHOT]
              at org.apache.storm.shade.io.netty.channel.AbstractChannelHandlerContext.invokeChannelActive(AbstractChannelHandlerContext.java:213) [storm-shaded-deps-2.3.0.y.jar:2.3.0-SNAPSHOT]
              at org.apache.storm.shade.io.netty.channel.AbstractChannelHandlerContext.invokeChannelActive(AbstractChannelHandlerContext.java:199) [storm-shaded-deps-2.3.0.y.jar:2.3.0-SNAPSHOT]
              at org.apache.storm.shade.io.netty.channel.AbstractChannelHandlerContext.fireChannelActive(AbstractChannelHandlerContext.java:192) [storm-shaded-deps-2.3.0.y.jar:2.3.0-SNAPSHOT]
              at org.apache.storm.shade.io.netty.channel.ChannelInboundHandlerAdapter.channelActive(ChannelInboundHandlerAdapter.java:64) [storm-shaded-deps-2.3.0.y.jar:2.3.0-SNAPSHOT]
              at org.apache.storm.shade.io.netty.channel.AbstractChannelHandlerContext.invokeChannelActive(AbstractChannelHandlerContext.java:213) [storm-shaded-deps-2.3.0.y.jar:2.3.0-SNAPSHOT]
              at org.apache.storm.shade.io.netty.channel.AbstractChannelHandlerContext.invokeChannelActive(AbstractChannelHandlerContext.java:199) [storm-shaded-deps-2.3.0.y.jar:2.3.0-SNAPSHOT]
              at org.apache.storm.shade.io.netty.channel.AbstractChannelHandlerContext.fireChannelActive(AbstractChannelHandlerContext.java:192) [storm-shaded-deps-2.3.0.y.jar:2.3.0-SNAPSHOT]
              at org.apache.storm.shade.io.netty.channel.DefaultChannelPipeline$HeadContext.channelActive(DefaultChannelPipeline.java:1422) [storm-shaded-deps-2.3.0.y.jar:2.3.0-SNAPSHOT]
              at org.apache.storm.shade.io.netty.channel.AbstractChannelHandlerContext.invokeChannelActive(AbstractChannelHandlerContext.java:213) [storm-shaded-deps-2.3.0.y.jar:2.3.0-SNAPSHOT]
              at org.apache.storm.shade.io.netty.channel.AbstractChannelHandlerContext.invokeChannelActive(AbstractChannelHandlerContext.java:199) [storm-shaded-deps-2.3.0.y.jar:2.3.0-SNAPSHOT]
              at org.apache.storm.shade.io.netty.channel.DefaultChannelPipeline.fireChannelActive(DefaultChannelPipeline.java:941) [storm-shaded-deps-2.3.0.y.jar:2.3.0-SNAPSHOT]
              at org.apache.storm.shade.io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.fulfillConnectPromise(AbstractNioChannel.java:311) [storm-shaded-deps-2.3.0.y.jar:2.3.0-SNAPSHOT]
              at org.apache.storm.shade.io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:341) [storm-shaded-deps-2.3.0.y.jar:2.3.0-SNAPSHOT]
              at org.apache.storm.shade.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:632) [storm-shaded-deps-2.3.0.y.jar:2.3.0-SNAPSHOT]
              at org.apache.storm.shade.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:579) [storm-shaded-deps-2.3.0.y.jar:2.3.0-SNAPSHOT]
              at org.apache.storm.shade.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:496) [storm-shaded-deps-2.3.0.y.jar:2.3.0-SNAPSHOT]
              at org.apache.storm.shade.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:458) [storm-shaded-deps-2.3.0.y.jar:2.3.0-SNAPSHOT]
              at org.apache.storm.shade.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:897) [storm-shaded-deps-2.3.0.y.jar:2.3.0-SNAPSHOT]
              at java.lang.Thread.run(Thread.java:748) [?:1.8.0_262]
      2020-12-21 20:07:00.907 o.a.s.m.n.KerberosSaslClientHandler openstorm3blue-n10.blue.ygrid.yahoo.com-pm-1 [ERROR] Failed to authenticate with server due to error:
      java.lang.RuntimeException: javax.security.auth.login.LoginException: No password provided
              at org.apache.storm.messaging.netty.KerberosSaslNettyClient.<init>(KerberosSaslNettyClient.java:71) ~[storm-client-2.3.0.y.jar:2.3.0-SNAPSHOT]
              at org.apache.storm.messaging.netty.KerberosSaslClientHandler.channelActive(KerberosSaslClientHandler.java:59) [storm-client-2.3.0.y.jar:2.3.0-SNAPSHOT]
              at org.apache.storm.shade.io.netty.channel.AbstractChannelHandlerContext.invokeChannelActive(AbstractChannelHandlerContext.java:213) [storm-shaded-deps-2.3.0.y.jar:2.3.0-SNAPSHOT]
              at org.apache.storm.shade.io.netty.channel.AbstractChannelHandlerContext.invokeChannelActive(AbstractChannelHandlerContext.java:199) [storm-shaded-deps-2.3.0.y.jar:2.3.0-SNAPSHOT]
              at org.apache.storm.shade.io.netty.channel.AbstractChannelHandlerContext.fireChannelActive(AbstractChannelHandlerContext.java:192) [storm-shaded-deps-2.3.0.y.jar:2.3.0-SNAPSHOT]
              at org.apache.storm.shade.io.netty.channel.ChannelInboundHandlerAdapter.channelActive(ChannelInboundHandlerAdapter.java:64) [storm-shaded-deps-2.3.0.y.jar:2.3.0-SNAPSHOT]
              at org.apache.storm.shade.io.netty.channel.AbstractChannelHandlerContext.invokeChannelActive(AbstractChannelHandlerContext.java:213) [storm-shaded-deps-2.3.0.y.jar:2.3.0-SNAPSHOT]
              at org.apache.storm.shade.io.netty.channel.AbstractChannelHandlerContext.invokeChannelActive(AbstractChannelHandlerContext.java:199) [storm-shaded-deps-2.3.0.y.jar:2.3.0-SNAPSHOT]
              at org.apache.storm.shade.io.netty.channel.AbstractChannelHandlerContext.fireChannelActive(AbstractChannelHandlerContext.java:192) [storm-shaded-deps-2.3.0.y.jar:2.3.0-SNAPSHOT]
              at org.apache.storm.shade.io.netty.channel.DefaultChannelPipeline$HeadContext.channelActive(DefaultChannelPipeline.java:1422) [storm-shaded-deps-2.3.0.y.jar:2.3.0-SNAPSHOT]
              at org.apache.storm.shade.io.netty.channel.AbstractChannelHandlerContext.invokeChannelActive(AbstractChannelHandlerContext.java:213) [storm-shaded-deps-2.3.0.y.jar:2.3.0-SNAPSHOT]
              at org.apache.storm.shade.io.netty.channel.AbstractChannelHandlerContext.invokeChannelActive(AbstractChannelHandlerContext.java:199) [storm-shaded-deps-2.3.0.y.jar:2.3.0-SNAPSHOT]
              at org.apache.storm.shade.io.netty.channel.DefaultChannelPipeline.fireChannelActive(DefaultChannelPipeline.java:941) [storm-shaded-deps-2.3.0.y.jar:2.3.0-SNAPSHOT]
              at org.apache.storm.shade.io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.fulfillConnectPromise(AbstractNioChannel.java:311) [storm-shaded-deps-2.3.0.y.jar:2.3.0-SNAPSHOT]
              at org.apache.storm.shade.io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:341) [storm-shaded-deps-2.3.0.y.jar:2.3.0-SNAPSHOT]
              at org.apache.storm.shade.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:632) [storm-shaded-deps-2.3.0.y.jar:2.3.0-SNAPSHOT]
              at org.apache.storm.shade.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:579) [storm-shaded-deps-2.3.0.y.jar:2.3.0-SNAPSHOT]
              at org.apache.storm.shade.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:496) [storm-shaded-deps-2.3.0.y.jar:2.3.0-SNAPSHOT]
              at org.apache.storm.shade.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:458) [storm-shaded-deps-2.3.0.y.jar:2.3.0-SNAPSHOT]
              at org.apache.storm.shade.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:897) [storm-shaded-deps-2.3.0.y.jar:2.3.0-SNAPSHOT]
              at java.lang.Thread.run(Thread.java:748) [?:1.8.0_262]
      Caused by: javax.security.auth.login.LoginException: No password provided
              at com.sun.security.auth.module.Krb5LoginModule.promptForPass(Krb5LoginModule.java:923) ~[?:1.8.0_262]
              at com.sun.security.auth.module.Krb5LoginModule.attemptAuthentication(Krb5LoginModule.java:764) ~[?:1.8.0_262]
              at com.sun.security.auth.module.Krb5LoginModule.login(Krb5LoginModule.java:618) ~[?:1.8.0_262]
              at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.8.0_262]
              at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:1.8.0_262]
              at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_262]
              at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_262]
              at javax.security.auth.login.LoginContext.invoke(LoginContext.java:755) ~[?:1.8.0_262]
              at javax.security.auth.login.LoginContext.access$000(LoginContext.java:195) ~[?:1.8.0_262]
              at javax.security.auth.login.LoginContext$4.run(LoginContext.java:682) ~[?:1.8.0_262]
              at javax.security.auth.login.LoginContext$4.run(LoginContext.java:680) ~[?:1.8.0_262]
              at java.security.AccessController.doPrivileged(Native Method) ~[?:1.8.0_262]
              at javax.security.auth.login.LoginContext.invokePriv(LoginContext.java:680) ~[?:1.8.0_262]
              at javax.security.auth.login.LoginContext.login(LoginContext.java:587) ~[?:1.8.0_262]
              at org.apache.storm.messaging.netty.Login.login(Login.java:301) ~[storm-client-2.3.0.y.jar:2.3.0-SNAPSHOT]
              at org.apache.storm.messaging.netty.Login.<init>(Login.java:83) ~[storm-client-2.3.0.y.jar:2.3.0-SNAPSHOT]
              at org.apache.storm.messaging.netty.KerberosSaslNettyClient.<init>(KerberosSaslNettyClient.java:66) ~[storm-client-2.3.0.y.jar:2.3.0-SNAPSHOT]
              ... 20 more
      2020-12-21 20:07:01.802 o.a.s.p.PacemakerClient executor-heartbeat-timer [ERROR] Error attempting to write to a channel to host openstorm3blue-n10.blue.ygrid.yahoo.com - Timed out waiting for channel ready.
      2020-12-21 20:07:01.803 o.a.s.p.PacemakerClient executor-heartbeat-timer [WARN] Not getting response or getting null response. Making 9 more attempts for openstorm3blue-n10.blue.ygrid.yahoo.com.
      

      Currently by design https://github.com/apache/storm/blob/master/docs/Pacemaker.md#security pacemaker allows writes by anyone (which should be improved in the future).

      So a quick work around is to make sure worker always has pacemaker.auth.method set to NONE

       

      Attachments

        Activity

          People

            ethanli Ethan Li
            ethanli Ethan Li
            Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Time Tracking

                Estimated:
                Original Estimate - Not Specified
                Not Specified
                Remaining:
                Remaining Estimate - 0h
                0h
                Logged:
                Time Spent - 1h 10m
                1h 10m