[KAFKA-3759] Incorrect JDBC credentials cause Connect worker to permanently fail

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Duplicate
• Affects Version/s: 0.9.0.1
• Component/s: connect

Environment

I work on a cloudera quickstart VM. I installed kafka / confluent via yum :
 
[cloudera@quickstart confluent]$ yum list installed | grep confluent
confluent-camus.noarch 2.0.1-1 @confluent-2.0
confluent-common.noarch 2.0.1-1 @confluent-2.0
confluent-kafka-2.11.7.noarch 0.9.0.1-1 @confluent-2.0
confluent-kafka-connect-hdfs.noarch 2.0.1-1 @confluent-2.0
confluent-kafka-connect-jdbc.noarch 2.0.1-1 @confluent-2.0
confluent-kafka-rest.noarch 2.0.1-1 @confluent-2.0
confluent-rest-utils.noarch 2.0.1-1 @confluent-2.0
confluent-schema-registry.noarch 2.0.1-1 @confluent-2.0
 

• Assignee: Ewen Cheslack-Postava
• Reporter: Lars George
• Created: 26/May/16 19:02
• Updated: 07/Oct/16 16:43
• Resolved: 15/Jun/16 17:14

Description

As reported by our peers:

All the following steps are executed in one cloudera instance, each in a different terminal.

• I start the kafka server with the (unchanged) properties file in /etc/kafka/server.properties
• I start the schema-registry with the (unchanged) properties file in /etc/schema-registry/schema-registry.properties
• I start the connect worker with the properties file /etc/schema-registry/connect-avro-distributed.properties
  HOWEVER I have changed the following 2 properties for my test:
  config.storage.topic=connect-configs-broken
  offset.storage.topic=connect-offsets-broken

Now I execute a shell script that uses the REST API to post a connector configuration to the worker. Here the snippet from the script
that shows the connector config:

 
==========================================================
read -resp $'please enter the password for the postgresql user \n' postgres_password
 
echo posting connector config into the kafka cluster
data="{
                \"name\": \"projectx-postgres-test\",
                \"config\": {
                               \"connection.url\":\"jdbc:postgresql://$postgres_db_url/$postgres_db_name?user=$postgres_user&password=$postgres_password\",
                               \"connector.class\":\"io.confluent.connect.jdbc.JdbcSourceConnector\",
                               \"tasks.max\":\"1\",
                               \"mode\":\"incrementing\",
                               \"incrementing.column.name\":\"id\",
                               \"topic.prefix\":\"test-postgres-jdbc-\"
                }
}"
url=http://$kafka_connect_rest_url/connectors
ctype="Content-Type: application/json"
curl -X POST $url -H "$ctype" --data "$data"
 
==========================================================

• I exectute the script with correct connection settings, but incorrect postgresql username / password.
• I receive the following answer from the REST API:

{"name":"projectx-postgres-test","config":{"connection.url":"jdbc:postgresql://localhost/connect_test?user=cloudera&password=wrongPW","connector.class":"io.confluent.connect.jdbc.JdbcSourceConnector","tasks.max":"1","mode":"incrementing","incrementing.column.name":"id","topic.prefix":"test-postgres-jdbc-","name":"projectx-postgres-test"},"tasks":[]}

Now the connect worker stops and I see the following error and stack trace:

[2016-05-23 05:53:43,966] INFO 127.0.0.1 - - [23/May/2016:12:53:41 +0000] "POST /connectors HTTP/1.1" 201 343  2500 (org.apache.kafka.connect.runtime.rest.RestServer:60)
[2016-05-23 05:53:44,656] ERROR Couldn't open connection to jdbc:postgresql://localhost/connect_test?user=cloudera&password=wrongPW: org.postgresql.util.PSQLException: FATAL: password authentication failed for user "cloudera" (io.confluent.connect.jdbc.JdbcSourceConnector:76)
[2016-05-23 05:53:44,663] ERROR Uncaught exception in herder work thread, exiting:  (org.apache.kafka.connect.runtime.distributed.DistributedHerder:166)
org.apache.kafka.connect.errors.ConnectException: Connector threw an exception while starting
        at org.apache.kafka.connect.runtime.Worker.addConnector(Worker.java:188)
        at org.apache.kafka.connect.runtime.distributed.DistributedHerder.startConnector(DistributedHerder.java:670)
        at org.apache.kafka.connect.runtime.distributed.DistributedHerder.startWork(DistributedHerder.java:642)
        at org.apache.kafka.connect.runtime.distributed.DistributedHerder.handleRebalanceCompleted(DistributedHerder.java:600)
        at org.apache.kafka.connect.runtime.distributed.DistributedHerder.tick(DistributedHerder.java:184)
        at org.apache.kafka.connect.runtime.distributed.DistributedHerder.run(DistributedHerder.java:159)
        at java.lang.Thread.run(Thread.java:745)
Caused by: org.apache.kafka.connect.errors.ConnectException: org.postgresql.util.PSQLException: FATAL: password authentication failed for user "cloudera"
        at io.confluent.connect.jdbc.JdbcSourceConnector.start(JdbcSourceConnector.java:77)
        at org.apache.kafka.connect.runtime.Worker.addConnector(Worker.java:186)
        ... 6 more
Caused by: org.postgresql.util.PSQLException: FATAL: password authentication failed for user "cloudera"
        at org.postgresql.core.v3.ConnectionFactoryImpl.doAuthentication(ConnectionFactoryImpl.java:427)
        at org.postgresql.core.v3.ConnectionFactoryImpl.openConnectionImpl(ConnectionFactoryImpl.java:203)
        at org.postgresql.core.ConnectionFactory.openConnection(ConnectionFactory.java:65)
        at org.postgresql.jdbc2.AbstractJdbc2Connection.<init>(AbstractJdbc2Connection.java:149)
        at org.postgresql.jdbc3.AbstractJdbc3Connection.<init>(AbstractJdbc3Connection.java:35)
        at org.postgresql.jdbc3g.AbstractJdbc3gConnection.<init>(AbstractJdbc3gConnection.java:22)
        at org.postgresql.jdbc4.AbstractJdbc4Connection.<init>(AbstractJdbc4Connection.java:47)
        at org.postgresql.jdbc4.Jdbc4Connection.<init>(Jdbc4Connection.java:30)
        at org.postgresql.Driver.makeConnection(Driver.java:414)
        at org.postgresql.Driver.connect(Driver.java:282)
        at java.sql.DriverManager.getConnection(DriverManager.java:571)
        at java.sql.DriverManager.getConnection(DriverManager.java:233)
        at io.confluent.connect.jdbc.JdbcSourceConnector.start(JdbcSourceConnector.java:74)
        ... 7 more
[2016-05-23 05:53:44,706] INFO Kafka Connect stopping (org.apache.kafka.connect.runtime.Connect:68)

Issue Links

• duplicates: KAFKA-3054 Connect Herder fail forever if sent a wrong connector config or task config