Uploaded image for project: 'Sentry'
  1. Sentry
  2. SENTRY-2506

Sentry starts but dysfunctional when Web Port is unavailable

    Details

    • Type: Bug
    • Status: Open
    • Priority: Major
    • Resolution: Unresolved
    • Affects Version/s: None
    • Fix Version/s: None
    • Component/s: Sentry
    • Labels:
      None

      Description

      Sentry clients (e.g., HMS) show errors like:

      2019-02-26 14:04:13,451 ERROR org.apache.sentry.core.common.transport.SentryTransportPool: [pool-9-thread-131]: Failed to obtain transport for XXXX.net:8038: null
      2019-02-26 14:04:40,815 ERROR org.apache.sentry.core.common.transport.SentryTransportPool: [pool-9-thread-130]: Failed to obtain transport for XXXX.net:8038: null
      2019-02-26 14:04:43,816 ERROR org.apache.sentry.binding.metastore.SentrySyncHMSNotificationsPostEventListener: [pool-9-thread-130]: Encountered failure while notifying notification for CREATE_DATABASE (id: 19)
      org.apache.sentry.core.common.exception.SentryUserException: java.net.SocketException: Connection reset
      at org.apache.sentry.core.common.transport.RetryClientInvocationHandler.connect(RetryClientInvocationHandler.java:166) ~[sentry-core-common-2.1.0-cdh6.1.1.jar:?]
      at org.apache.sentry.core.common.transport.RetryClientInvocationHandler.invokeImpl(RetryClientInvocationHandler.java:90) ~[sentry-core-common-2.1.0-cdh6.1.1.jar:?]
      at org.apache.sentry.core.common.transport.SentryClientInvocationHandler.invoke(SentryClientInvocationHandler.java:41) ~[sentry-core-common-2.1.0-cdh6.1.1.jar:?]
      at com.sun.proxy.$Proxy34.notifyHmsEvent(Unknown Source) ~[?:?]
      ::::::

      Sentry stderr log shows:

      Exception in thread "main" java.util.concurrent.ExecutionException: java.lang.Exception: Error starting server
      at java.util.concurrent.FutureTask.report(FutureTask.java:122)
      at java.util.concurrent.FutureTask.get(FutureTask.java:192)
      at org.apache.sentry.service.thrift.SentryService$CommandImpl.run(SentryService.java:596)
      at org.apache.sentry.SentryMain.main(SentryMain.java:120)
      at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
      at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
      at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
      at java.lang.reflect.Method.invoke(Method.java:498)
      at org.apache.hadoop.util.RunJar.run(RunJar.java:313)
      at org.apache.hadoop.util.RunJar.main(RunJar.java:227)
      Caused by: java.lang.Exception: Error starting server
      at org.apache.sentry.service.thrift.SentryService.call(SentryService.java:207)
      at org.apache.sentry.service.thrift.SentryService.call(SentryService.java:81)
      at java.util.concurrent.FutureTask.run(FutureTask.java:266)
      at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
      at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
      at java.lang.Thread.run(Thread.java:748)
      Caused by: java.security.PrivilegedActionException: java.net.BindException: Address already in use
      at java.security.AccessController.doPrivileged(Native Method)
      at javax.security.auth.Subject.doAs(Subject.java:422)
      at org.apache.sentry.service.thrift.SentryService.call(SentryService.java:195)
      ... 5 more
      Caused by: java.net.BindException: Address already in use
      at sun.nio.ch.Net.bind0(Native Method)
      at sun.nio.ch.Net.bind(Net.java:433)
      at sun.nio.ch.Net.bind(Net.java:425)
      at sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:223)
      at sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:74)
      at org.eclipse.jetty.server.ServerConnector.open(ServerConnector.java:317)
      at org.eclipse.jetty.server.AbstractNetworkConnector.doStart(AbstractNetworkConnector.java:80)
      at org.eclipse.jetty.server.ServerConnector.doStart(ServerConnector.java:235)
      at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
      at org.eclipse.jetty.server.Server.doStart(Server.java:401)
      at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
      at org.apache.sentry.api.service.thrift.SentryWebServer.start(SentryWebServer.java:190)
      at org.apache.sentry.service.thrift.SentryService.startSentryWebServer(SentryService.java:428)
      at org.apache.sentry.service.thrift.SentryService.runServer(SentryService.java:273)
      at org.apache.sentry.service.thrift.SentryService.access$000(SentryService.java:81)
      at org.apache.sentry.service.thrift.SentryService$1.run(SentryService.java:198)
      at org.apache.sentry.service.thrift.SentryService$1.run(SentryService.java:195)
      ... 8 more

      port 29000 turns out to be taken by another application:

      $ netstat -tulpn | grep 29000
      tcp 0 0 0.0.0.0:29000 0.0.0.0:* LISTEN 9510/BLAH

      . But Sentry still starts as if normal although the clients (Hive, Impala, etc.) cannot run queries. That is extremely confusing. Sentry startup should be corrected to fail to start in those situations with a clear error pointing at the cause.

        Attachments

          Activity

            People

            • Assignee:
              Unassigned
              Reporter:
              bgitline Boris Gitline
            • Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

              Dates

              • Created:
                Updated: