[NIFI-1831] Allow encrypted passwords in configuration files

• Type: New Feature
• Status: Resolved
• Priority: Critical
• Resolution: Fixed
• Affects Version/s: 0.6.1
• Fix Version/s: 1.0.0
• Component/s: Configuration, Core Framework
• Labels: configuration encryption password security
• Assignee: Andy LoPresto
• Reporter: Andy LoPresto
• Created: 30/Apr/16 00:25
• Updated: 18/Jun/24 20:24
• Resolved: 24/Aug/16 04:08

Description

Storing passwords in plaintext in configuration files is not a security best practice. While file access can be restricted through OS permissions, these configuration files can be accidentally checked into source control, shared or deployed to multiple instances, etc.

NiFi should allow a deployer to provide an encrypted password in the configuration file to minimize exposure of the passwords. On application start-up, NiFi should decrypt the passwords in memory. NiFi should also include a utility to encrypt the raw passwords (and optionally populate the configuration files and provide additional metadata in the configuration files).

I am aware this simply shifts the responsibility/delegation of trust from the passwords in the properties file to a new location on the same system, but mitigating the visibility of the raw passwords in the properties file can be one step in a defense in depth approach and is often mandated by security policies within organizations using NiFi.

The key used for encryption should not be hard-coded into the application source code, nor should it be universally consistent. The key could be determined by reading static information from the deployed system and feeding it to a key derivation function based on a cryptographically-secure hash function, such as PBKDF2, bcrypt, or scrypt. However, this does introduce upgrade, system migration, and portability issues. These challenges will have to be kept in consideration when determining the key derivation process.

Manual key entry is a possibility, and then the master key would only be present in memory, but this prevents automatic reboot on loss of power or other recovery scenario.

This must be backward-compatible to allow systems with plaintext passwords to continue operating. Options for achieving this are to only attempt to decrypt passwords when a sibling property is present, or to match a specific format.

For these examples, I have used the following default values:

password: thisIsABadPassword
key: 0123456789ABCDEFFEDCBA98765432100123456789ABCDEFFEDCBA9876543210
iv:  0123456789ABCDEFFEDCBA9876543210
algorithm: AES/CBC 256-bit

*Note: These values should not be used in production systems – the key and IV are common test values, and an AEAD cipher is preferable to provide cipher text integrity assurances, however OpenSSL does not support the use of AEAD ciphers for command-line encryption at this time*

Example 1: here the sibling property indicates the password is encrypted and with which implementation; the absence of the property would default to a raw password

hw12203:/Users/alopresto/Workspace/scratch/encrypted-passwords (master) alopresto
🔓 0s @ 16:25:56 $ echo "thisIsABadPassword" > password.txt
hw12203:/Users/alopresto/Workspace/scratch/encrypted-passwords (master) alopresto
🔓 0s @ 16:26:47 $ ossl aes-256-cbc -e -nosalt -p -K 0123456789ABCDEFFEDCBA98765432100123456789ABCDEFFEDCBA9876543210 -iv 0123456789ABCDEFFEDCBA9876543210 -a -in password.txt -out password.enc
key=0123456789ABCDEFFEDCBA98765432100123456789ABCDEFFEDCBA9876543210
iv =0123456789ABCDEFFEDCBA9876543210
hw12203:/Users/alopresto/Workspace/scratch/encrypted-passwords (master) alopresto
🔓 0s @ 16:27:09 $ xxd password.enc
0000000: 5643 5856 6146 6250 4158 364f 5743 7646  VCXVaFbPAX6OWCvF
0000010: 6963 6b76 4a63 7744 3854 6b67 3731 4c76  ickvJcwD8Tkg71Lv
0000020: 4d38 6d32 7952 4776 5739 413d 0a         M8m2yRGvW9A=.
hw12203:/Users/alopresto/Workspace/scratch/encrypted-passwords (master) alopresto
🔓 0s @ 16:27:16 $ more password.enc
VCXVaFbPAX6OWCvFickvJcwD8Tkg71LvM8m2yRGvW9A=
hw12203:/Users/alopresto/Workspace/scratch/encrypted-passwords (master) alopresto
🔓 0s @ 16:27:55 $

In nifi.properties:

nifi.security.keystorePasswd=VCXVaFbPAX6OWCvFickvJcwD8Tkg71LvM8m2yRGvW9A=
nifi.security.keystorePasswd.encrypted=AES-CBC-256

Example 2: here the encrypted password has a header tag indicating both that it is encrypted and the algorithm used

    @Test
    public void testShouldDecryptPassword() throws Exception {
        // Arrange
        KeyedCipherProvider cipherProvider = new AESKeyedCipherProvider()

        final String PLAINTEXT = "thisIsABadPassword"
        logger.info("Expected: ${Hex.encodeHexString(PLAINTEXT.bytes)}")

        final byte[] IV = Hex.decodeHex("0123456789ABCDEFFEDCBA9876543210" as char[])
        final byte[] LOCAL_KEY = Hex.decodeHex("0123456789ABCDEFFEDCBA9876543210" * 2 as char[])
        // Generated via openssl enc -a
        final String CIPHER_TEXT = "VCXVaFbPAX6OWCvFickvJcwD8Tkg71LvM8m2yRGvW9A="
        byte[] cipherBytes = Base64.decoder.decode(CIPHER_TEXT)

        SecretKey localKey = new SecretKeySpec(LOCAL_KEY, "AES")

        EncryptionMethod encryptionMethod = EncryptionMethod.AES_CBC
        logger.info("Using algorithm: ${encryptionMethod.getAlgorithm()}")
        logger.info("Cipher text: \$nifipw\$${CIPHER_TEXT} ${cipherBytes.length + 8}")

        // Act
        Cipher cipher = cipherProvider.getCipher(encryptionMethod, localKey, IV, false)
        byte[] recoveredBytes = cipher.doFinal(cipherBytes)
        
        // OpenSSL adds a newline character during encryption
        String recovered = new String(recoveredBytes, "UTF-8").trim()
        logger.info("Recovered: ${recovered} ${Hex.encodeHexString(recoveredBytes)}")

        // Assert
        assert PLAINTEXT.equals(recovered)
    }

In nifi.properties:

nifi.security.keystorePasswd=$nifipw$VCXVaFbPAX6OWCvFickvJcwD8Tkg71LvM8m2yRGvW9A=

Ideally, NiFi would use a pluggable implementation architecture to allow users to integrate with a variety of secret management services. There are both commercial and open source solutions, including CyberArk Enterprise Password Vault [1], Hashicorp Vault [2], and Square Keywhiz [3]. In the future, this could also be extended to Hardware Security Modules (HSM) like SafeNet Luna [4] and Amazon CloudHSM [5].

[1] http://www.cyberark.com/products/privileged-account-security-solution/enterprise-password-vault/
[2] https://www.vaultproject.io/
[3] https://square.github.io/keywhiz/
[4] http://www.safenet-inc.com/data-encryption/hardware-security-modules-hsms/luna-hsms-key-management/luna-sa-network-hsm/
[5] https://aws.amazon.com/cloudhsm/

Issue Links

• blocks: NIFI-2451 Need to update Encryption Configuration content in Admin guide for 1.0 changes
• is related to: NIFI-2571 Deprecate NiFiProperties.getInstance() in 0.x
• relates to: NIFI-2640 Fix Windows command-line utility for encrypted configuration files
• relates to: NIFI-2656 Allow bootstrap process to prompt for password/key
• relates to: NIFI-2652 Handle multiple invocations of the encrypt-config tool
• relates to: NIFI-2653 Encrypted configs should handle variable registry
• relates to: NIFI-2654 Encrypted configs should handle login identity provider configs
• relates to: NIFI-3024 Encrypted configuration migrator should be able to update sensitive properties key and migrate flow.xml.gz
• relates to: NIFI-8586 Allow encrypted passwords in config yml
• relates to: NIFI-13414 Remove Property Protection Modules and Encrypt Config Tools
• relates to: NIFI-2655 Encrypted config tool should handle absence of "-n" flag
• links to: GitHub Pull Request #834

Sub-Tasks

• Update NiFiProperties usage to a context based accessor rather than static singleton: Resolved