[HADOOP-13288] Guard null stats key in FileSystemStorageStatistics

• Type: Sub-task
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 2.8.0, 3.0.0-alpha1
• Fix Version/s: 2.8.0, 3.0.0-alpha1
• Component/s: fs
• Target Version/s: 2.8.0
• Assignee: Mingliang Liu
• Reporter: Mingliang Liu
• Created: 17/Jun/16 22:10
• Updated: 30/Aug/16 01:13
• Resolved: 20/Jun/16 21:31

Description

Currently in FileSystemStorageStatistics we simply returns data from FileSystem#Statistics. However there is no null key check, which leads to NPE problems to downstream applications. For example, we got a NPE when passing a null key to FileSystemStorageStatistics#getLong(), exception stack as following:

NullPointerException
at org.apache.hadoop.fs.FileSystemStorageStatistics.fetch(FileSystemStorageStatistics.java:80)
at org.apache.hadoop.fs.FileSystemStorageStatistics.getLong(FileSystemStorageStatistics.java:108)
at org.apache.tez.runtime.metrics.FileSystemStatisticsUpdater2.updateCounters(FileSystemStatisticsUpdater2.java:60)
at org.apache.tez.runtime.metrics.TaskCounterUpdater.updateCounters(TaskCounterUpdater.java:118)
at org.apache.tez.runtime.RuntimeTask.setFrameworkCounters(RuntimeTask.java:172)
at org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:100)
at org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:37)
at org.apache.tez.common.CallableWithNdc.call(CallableWithNdc.java:36)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)

This jira is to add null stat key check to FileSystemStorageStatistics.

Thanks Hitesh Shah for trying in Tez and reporting this.