[DIRSERVER-2318] StartTLS and LDAPS are not working

• Type: Bug
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 2.0.0-M24, 2.0.0.AM26
• Component/s: ldap, security

Environment

Ubuntu 20.04 clean installation used for both client and server. Used version 2.0.0~M24-3 from Ubuntu repository and version 2.0.0.AM26 deb package from official website. Using openjdk-14-jre and openjdk-11-jre from Ubuntu repository. Apache Studio 2.0.0-M15 from website.

• Assignee: Unassigned
• Reporter: Karl Frauendienst
• Created: 24/Jun/20 15:45
• Updated: 20/May/21 13:29

Description

Attempting to make a secure LDAP connection results in handshake failure with unknown error.  No error with unencrypted connections.  Tested on two separate systems.

First setup: Ubuntu Server 20.04 with apacheds 2.0.0~M24-3 installed from repository.  Tried both default-jre (openjdk-11-jre) and openjdk-14-jre.  Running Apache Studio 2.0.0-M15 from official website on a separate Ubuntu Desktop 20.04 system and tested with same two jre versions.  On this setup, I occasionally got an error stating the key was only 512 bits, so I used keytool according to the ApacheDS getting started guide to create and use a 2048 bit keypair.  Following that I only get the handshake failure.

Second setup: Ubuntu Desktop 20.04 running openjdk-14-jre with ApacheDS 2.0.0.AM26 deb pkg and Apache Studio 2.0.0-M15 from official website.  This produces the handshake error.  I believe the issue is server side because I can produce a similar handshake error using ldapsearch.  It works fine unencrypted, but fails using either StartTLS on port 10389 or LDAPS on 10636. I did not replace the keypair in this setup.  This setup occasionally will work with StartTLS and LDAPS but will seemingly work or not work intermittently with no configuration changes being made.

I have tested with Apache Studio SSL verification both enabled and disabled in both cases.

Errors produced include:

!MESSAGE Improper close state: Status = OK HandshakeStatus = NEED_WRAP

!MESSAGE The authentication failed

• ERR_04120_TLS_HANDSHAKE_ERROR The TLS handshake failed, reason: Unspecified

!MESSAGE org.apache.directory.api.ldap.model.exception.LdapTlsHandshakeException: ERR_04120_TLS_HANDSHAKE_ERROR The TLS handshake failed, reason: Unspecified

!MESSAGE ERR_01200_BAD_TRANSITION_FROM_STATE Bad transition from state START_STATE, tag 0x15

!MESSAGE org.apache.directory.api.ldap.codec.api.ResponseCarryingException: ERR_01200_BAD_TRANSITION_FROM_STATE Bad transition from state START_STATE, tag 0x15

!MESSAGE Error while opening connection

• PROTOCOL_ERROR: The server will disconnect!