[AMBARI-25469] Bad UTF encoding on Alert listener receiver

• Type: Bug
• Status: Resolved
• Priority: Critical
• Resolution: Fixed
• Affects Version/s: 2.7.5
• Fix Version/s: 2.8.0, 2.7.6
• Component/s: ambari-agent
• Labels: pull-request-available
• Assignee: Dmytro Grinenko
• Reporter: Dmytro Vitiuk
• Created: 30/Jan/20 11:54
• Updated: 17/Nov/22 09:21
• Resolved: 30/Jan/20 12:33

Description

The problem observed on gce cluster during testing of Ambari Kerberos slowness

2020-01-29 09:42:37,550 ERROR [alert-event-bus-2] AmbariJpaLocalTxnInterceptor:188 - [DETAILED ERROR] Internal exception (1) :
org.postgresql.util.PSQLException: ERROR: invalid byte sequence for encoding "UTF8": 0x00
        at org.postgresql.core.v3.QueryExecutorImpl.receiveErrorResponse(QueryExecutorImpl.java:2433)
        at org.postgresql.core.v3.QueryExecutorImpl.processResults(QueryExecutorImpl.java:2178)
        at org.postgresql.core.v3.QueryExecutorImpl.execute(QueryExecutorImpl.java:306)
        at org.postgresql.jdbc.PgStatement.executeInternal(PgStatement.java:441)
        at org.postgresql.jdbc.PgStatement.execute(PgStatement.java:365)
        at org.postgresql.jdbc.PgPreparedStatement.executeWithFlags(PgPreparedStatement.java:155)
        at org.postgresql.jdbc.PgPreparedStatement.executeUpdate(PgPreparedStatement.java:132)
        at org.eclipse.persistence.internal.databaseaccess.DatabaseAccessor.executeDirectNoSelect(DatabaseAccessor.java:892)
        at org.eclipse.persistence.internal.databaseaccess.DatabaseAccessor.executeNoSelect(DatabaseAccessor.java:964)
        at org.eclipse.persistence.internal.databaseaccess.DatabaseAccessor.basicExecuteCall(DatabaseAccessor.java:633)
        at org.eclipse.persistence.internal.databaseaccess.ParameterizedSQLBatchWritingMechanism.executeBatch(ParameterizedSQLBatchWritingMechanism.java:149)
        at org.eclipse.persistence.internal.databaseaccess.ParameterizedSQLBatchWritingMechanism.executeBatchedStatements(ParameterizedSQLBatchWritingMechanism.java:134)
        at org.eclipse.persistence.internal.databaseaccess.DatabaseAccessor.writesCompleted(DatabaseAccessor.java:1845)
        at org.eclipse.persistence.internal.sessions.AbstractSession.writesCompleted(AbstractSession.java:4300)
        at org.eclipse.persistence.internal.sessions.UnitOfWorkImpl.writesCompleted(UnitOfWorkImpl.java:5592)
        at org.eclipse.persistence.internal.sessions.UnitOfWorkImpl.acquireWriteLocks(UnitOfWorkImpl.java:1646)
        at org.eclipse.persistence.internal.sessions.UnitOfWorkImpl.commitTransactionAfterWriteChanges(UnitOfWorkImpl.java:1614)
        at org.eclipse.persistence.internal.sessions.RepeatableWriteUnitOfWork.commitRootUnitOfWork(RepeatableWriteUnitOfWork.java:285)
        at org.eclipse.persistence.internal.sessions.UnitOfWorkImpl.commitAndResume(UnitOfWorkImpl.java:1169)
        at org.eclipse.persistence.internal.jpa.transaction.EntityTransactionImpl.commit(EntityTransactionImpl.java:134)
        at org.apache.ambari.server.orm.AmbariJpaLocalTxnInterceptor.invoke(AmbariJpaLocalTxnInterceptor.java:153)
        at com.google.inject.internal.InterceptorStackCallback$InterceptedMethodInvocation.proceed(InterceptorStackCallback.java:77)
        at com.google.inject.internal.InterceptorStackCallback.intercept(InterceptorStackCallback.java:55)
        at org.apache.ambari.server.events.listeners.alerts.AlertReceivedListener$$EnhancerByGuice$$c6d5f173.saveEntities(<generated>)
        at org.apache.ambari.server.events.listeners.alerts.AlertReceivedListener.onAlertEvent(AlertReceivedListener.java:388)
        at org.apache.ambari.server.events.listeners.alerts.AlertReceivedListener$$EnhancerByGuice$$c6d5f173.CGLIB$onAlertEvent$0(<generated>)
        at org.apache.ambari.server.events.listeners.alerts.AlertReceivedListener$$EnhancerByGuice$$c6d5f173$$FastClassByGuice$$3f418344.invoke(<generated>)
        at com.google.inject.internal.cglib.proxy.$MethodProxy.invokeSuper(MethodProxy.java:228)
        at com.google.inject.internal.InterceptorStackCallback$InterceptedMethodInvocation.proceed(InterceptorStackCallback.java:76)
        at org.apache.ambari.server.orm.AmbariLocalSessionInterceptor.invoke(AmbariLocalSessionInterceptor.java:44)
        at com.google.inject.internal.InterceptorStackCallback$InterceptedMethodInvocation.proceed(InterceptorStackCallback.java:77)
        at com.google.inject.internal.InterceptorStackCallback.intercept(InterceptorStackCallback.java:55)
        at org.apache.ambari.server.events.listeners.alerts.AlertReceivedListener$$EnhancerByGuice$$c6d5f173.onAlertEvent(<generated>)
        at sun.reflect.GeneratedMethodAccessor718.invoke(Unknown Source)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at com.google.common.eventbus.Subscriber.invokeSubscriberMethod(Subscriber.java:87)
        at com.google.common.eventbus.Subscriber$1.run(Subscriber.java:72)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:745)
2020-01-29 09:42:37,556 ERROR [alert-event-bus-2] default:232 - Exception thrown by subscriber method onAlertEvent(org.apache.ambari.server.events.AlertReceivedEvent) on subscriber org.apache.ambari.server.events.listeners.alerts.AlertReceivedListener$$EnhancerByGuice$$c6d5f173@29ba63f0 when dispatching event: AlertReceivedEvent{cluserId=0, alerts=[{clusterId=2, state=CRITICAL, name=hive_server_process, service=HIVE, component=HIVE_SERVER, host=<hostName>, instance=null, text='Connection failed on host <hostName>:10000 (Traceback (most recent call last):
  File "/var/lib/ambari-agent/cache/stacks/HDP/3.0/services/HIVE/package/alerts/alert_hive_thrift_port.py", line 213, in execute
    ldap_password=ldap_password, pam_username=pam_username, pam_password=pam_password)
  File "/usr/lib/ambari-agent/lib/resource_management/libraries/functions/hive_check.py", line 95, in check_thrift_port_sasl
    timeout_kill_strategy=TerminateStrategy.KILL_PROCESS_TREE,
  File "/usr/lib/ambari-agent/lib/resource_management/core/base.py", line 166, in __init__
    self.env.run()
  File "/usr/lib/ambari-agent/lib/resource_management/core/environment.py", line 160, in run
    self.run_action(resource, action)
  File "/usr/lib/ambari-agent/lib/resource_management/core/environment.py", line 124, in run_action
    provider_action()
  File "/usr/lib/ambari-agent/lib/resource_management/core/providers/system.py", line 263, in action_run
    returns=self.resource.returns)
  File "/usr/lib/ambari-agent/lib/resource_management/core/shell.py", line 72, in inner
    result = function(command, **kwargs)
  File "/usr/lib/ambari-agent/lib/resource_management/core/shell.py", line 102, in checked_call
    tries=tries, try_sleep=try_sleep, timeout_kill_strategy=timeout_kill_strategy, returns=returns)
  File "/usr/lib/ambari-agent/lib/resource_management/core/shell.py", line 150, in _call_wrapper
    result = _call(command, **kwargs_copy)
  File "/usr/lib/ambari-agent/lib/resource_management/core/shell.py", line 314, in _call
    raise ExecutionFailed(err_msg, code, out, err)
ExecutionFailed: Execution of '! (beeline -u 'jdbc:hive2://<hostName>:10000/;transportMode=binary;principal=hive/[e-mail]' -n hive -e ';' 2>&1 | awk '{print}' | grep -vz -i -e 'Connected to:' -e 'Transaction isolation:' -e 'inactive HS2 instance; use service discovery')' returned 1. SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/usr/hdp/3.1.5.0-152/hive/lib/log4j-slf4j-impl-2.10.0.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/usr/hdp/3.1.5.0-152/hadoop/lib/slf4j-log4j12-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]
Connecting to jdbc:hive2://<hostName>:10000/;transportMode=binary;principal=hive/[e-mail]
20/01/29 09:42:34 [main]: WARN jdbc.HiveConnection: Failed to connect to <hostName>:10000
Could not open connection to the HS2 server. Please check the server URI and if the URI is correct, then ask the administrator to check the server status.
Error: Could not open client transport with JDBC Uri: jdbc:hive2://<hostName>:10000/;transportMode=binary;principal=hive/[e-mail]: java.net.ConnectException: Connection refused (Connection refused) (state=08S01,code=0)

)'}]}
javax.persistence.RollbackException: Exception [EclipseLink-4002] (Eclipse Persistence Services - 2.6.2.v20151217-774c696): org.eclipse.persistence.exceptions.DatabaseException
Internal Exception: org.postgresql.util.PSQLException: ERROR: invalid byte sequence for encoding "UTF8": 0x00
Error Code: 0

Issue Links

• links to: GitHub PR#3527
• links to: GitHub Pull Request #3175