[HDFS-14665] HttpFS: LISTSTATUS response is missing HDFS-specific fields

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 3.3.0, 3.2.1, 3.1.3
• Component/s: httpfs
• Target Version/s: 3.3.0
• Assignee: Siyao Meng
• Reporter: Siyao Meng
• Created: 24/Jul/19 05:16
• Updated: 02/Oct/19 17:15
• Resolved: 20/Aug/19 17:50

Description

WebHDFS:

GET /webhdfs/v1/tmp/?op=LISTSTATUS&user.name=hdfs HTTP/1.1

{
  "FileStatuses": {
    "FileStatus": [
...
      {
        "accessTime": 0,
        "blockSize": 0,
        "childrenNum": 0,
        "fileId": 16395,
        "group": "hadoop",
        "length": 0,
        "modificationTime": 1563893395614,
        "owner": "mapred",
        "pathSuffix": "logs",
        "permission": "1777",
        "replication": 0,
        "storagePolicy": 0,
        "type": "DIRECTORY"
      }
    ]
  }
}

HttpFS:

GET /webhdfs/v1/tmp/?op=LISTSTATUS&user.name=hdfs HTTP/1.1

{
  "FileStatuses": {
    "FileStatus": [
...
      {
        "pathSuffix": "logs",
        "type": "DIRECTORY",
        "length": 0,
        "owner": "mapred",
        "group": "hadoop",
        "permission": "1777",
        "accessTime": 0,
        "modificationTime": 1563893395614,
        "blockSize": 0,
        "replication": 0
      }
    ]
  }
}

You can see the same LISTSTATUS request to HttpFS is missing 3 fields:

"childrenNum" (should only be none 0 for directories)
"fileId"
"storagePolicy"

The same applies to LISTSTATUS_BATCH, which might be using the same underlying calls to compose the response.

Root cause:

toJsonInner didn't serialize the HDFS-specific keys from FileStatus.

Also may file another Jira to align the order of the keys in the responses.

Issue Links

• is a parent of: HDFS-14718 HttpFS: Sort response by key names as WebHDFS does
• links to: GitHub Pull Request #1267
• links to: GitHub Pull Request #1289
• links to: GitHub Pull Request #1291
• links to: GitHub Pull Request #1294