The docs and default.xml state that the secret is randomly generated if the secret.file is not present, this is incorrect as the secret must be shared across all nodes in the cluster as it is used to verify the signature of the hadoop.auth cookie. If randomly generated it would be diff in all nodes.
AuthenticationFilterInitializer#initFilter fails if the configured hadoop.http.authentication.signature.secret.file does not exist, eg:
Creating /var/lib/hadoop-hdfs/hadoop-http-auth-signature-secret (populated with a string) fixes the issue. Per the auth docs "If a secret is not provided a random secret is generated at start up time.", which sounds like it means the file should be generated at startup with a random secrete, which doesn't seem to be the case. Also the instructions in the docs should be more clear in this regard.