Details
-
Bug
-
Status: Closed
-
Major
-
Resolution: Fixed
-
None
-
None
-
None
Description
Despite not changing our job's properties, we found it failing to connect to the DB:
2023-12-07 11:21:41 PST INFO [TaskStateModelFactory-task_thread-0] com.zaxxer.hikari.HikariDataSource - HikariPool-1-MysqlStateStore-redacted_db_name - Starting... 2023-12-07 11:21:42 PST ERROR [TaskStateModelFactory-task_thread-0] com.zaxxer.hikari.pool.HikariPool - HikariPool-1-MysqlStateStore-redacted_db_name - Exception during pool initialization. java.sql.SQLException: Access denied for user 'redacted_db_name'@'10.1.2.3' (using password: YES) at com.mysql.cj.jdbc.exceptions.SQLError.createSQLException(SQLError.java:129) at com.mysql.cj.jdbc.exceptions.SQLExceptionsMapping.translateException(SQLExceptionsMapping.java:122) ...
on previously successful runs, logging showed:
2023-12-07 15:24:55 PST INFO [TaskStateModelFactory-task_thread-0] com.zaxxer.hikari.HikariDataSource - HikariPool-1-MysqlStateStore-redacted_db_name - Starting... 2023-12-07 15:24:56 PST INFO [TaskStateModelFactory-task_thread-0] com.zaxxer.hikari.HikariDataSource - HikariPool-1-MysqlStateStore-redacted_db_name - Start completed. ...
throughout we supplied our DB password as:
state.store.db.password=ENC(rEdActeD=) ... encrypt.key.loc=/path/to/master/key ...
the issue wound up being that the master key path was removed from the FS. this would have been significantly easier to diagnose if the `PasswordManager` had logged this discovery upon detection, rather than silently skipping the configured path, upon finding the file not to exist.