[RANGER-4883] audit to HDFS fails with error: org.apache.hadoop.fs.UnsupportedFileSystemException: No FileSystem for scheme "hdfs"

• Type: Bug
• Status: In Progress
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 3.0.0, 2.5.0
• Component/s: audit, plugins
• Assignee: Madhan Neethiraj
• Reporter: Madhan Neethiraj
• Created: 01/Aug/24 05:17
• Updated: 01/Aug/24 05:18

Description

Writing audit log to HDFS fails with the following error in Kafka and Knox plugins:

Kafka plugin:

[2024-08-01 01:48:50,242] ERROR Exception encountered while writing audits to HDFS! (org.apache.ranger.audit.utils.RangerJSONAuditWriter)
org.apache.hadoop.fs.UnsupportedFileSystemException: No FileSystem for scheme "hdfs"
        at org.apache.hadoop.fs.FileSystem.getFileSystemClass(FileSystem.java:3443)
        at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:3466)
        at org.apache.hadoop.fs.FileSystem.access$300(FileSystem.java:174)
        at org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:3574)
        at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:3521)
        at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:540)
        at org.apache.ranger.audit.utils.AbstractRangerAuditWriter.createFileSystemFolders(AbstractRangerAuditWriter.java:97)

 

Knox plugin:

01:52:21.193 [org.apache.ranger.audit.queue.AuditBatchQueue1] ERROR org.apache.ranger.audit.utils.RangerJSONAuditWriter - Exception encountered while writing audits to HDFS!
org.apache.hadoop.fs.UnsupportedFileSystemException: No FileSystem for scheme "hdfs"
        at org.apache.hadoop.fs.FileSystem.getFileSystemClass(FileSystem.java:3443)
        at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:3466)
        at org.apache.hadoop.fs.FileSystem.access$300(FileSystem.java:174)
        at org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:3574)
        at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:3521)
        at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:540)
        at org.apache.ranger.audit.utils.AbstractRangerAuditWriter.createFileSystemFolders(AbstractRangerAuditWriter.java:97)

 

This is due to the plugins packaging missing library hadoop-hdfs-client, which includes the necessary entry to handle "hdfs" file scheme in META-INF/services/
org.apache.hadoop.hdfs.fs.FileSystem.