[HDFS-15465] Support WebHDFS accesses to the data stored in secure Datanode through insecure Namenode

• Type: Wish
• Status: Resolved
• Priority: Minor
• Resolution: Fixed
• Fix Version/s: 3.4.0
• Component/s: federation, webhdfs
• Assignee: Toshihiko Uchida
• Reporter: Toshihiko Uchida
• Created: 12/Jul/20 06:20
• Updated: 27/Jul/20 17:12
• Resolved: 27/Jul/20 16:55

Description

We're federating a secure HDFS cluster with an insecure cluster.
Using HDFS RPC, we can access the data managed by insecure Namenode and stored in secure Datanode.
However, it does not work for WebHDFS due to HadoopIllegalArgumentException.

$ curl -i "http://<INSECURE_NAMENODE>:<PORT>/webhdfs/v1/<PATH>?op=OPEN"
HTTP/1.1 307 TEMPORARY_REDIRECT
(omitted)
Location: http://<SECURE_DATANODE>:<PORT>/webhdfs/v1/<PATH>?op=OPEN&namenoderpcaddress=<INSECURE_NAMENODE>&offset=0
$ curl -i "http://<SECURE_DATANODE>:<PORT>/webhdfs/v1/<PATH>?op=OPEN&namenoderpcaddress=<INSECURE_NAMENODE>&offset=0"
HTTP/1.1 400 Bad Request
(omitted)
{"RemoteException":{"exception":"HadoopIllegalArgumentException","javaClassName":"org.apache.hadoop.HadoopIllegalArgumentException","message":"Invalid argument, newValue is null"}}

This is because secure Datanode expects a delegation token, but insecure Namenode does not return it to a client.

• org.apache.hadoop.security.token.Token.decodeWritable

    private static void decodeWritable(Writable obj,
                                       String newValue) throws IOException {
      if (newValue == null) {
        throw new HadoopIllegalArgumentException(
                "Invalid argument, newValue is null");
      }
  

The issue proposes to support the access also for WebHDFS.
The attached PDF file webhdfs-federation.pdf depicts our current architecture and proposal.