Details
-
Sub-task
-
Status: Resolved
-
Major
-
Resolution: Fixed
-
2.8.0, 3.0.0-alpha1
-
None
Description
Currently in FileSystemStorageStatistics we simply returns data from FileSystem#Statistics. However there is no null key check, which leads to NPE problems to downstream applications. For example, we got a NPE when passing a null key to FileSystemStorageStatistics#getLong(), exception stack as following:
NullPointerException
at org.apache.hadoop.fs.FileSystemStorageStatistics.fetch(FileSystemStorageStatistics.java:80)
at org.apache.hadoop.fs.FileSystemStorageStatistics.getLong(FileSystemStorageStatistics.java:108)
at org.apache.tez.runtime.metrics.FileSystemStatisticsUpdater2.updateCounters(FileSystemStatisticsUpdater2.java:60)
at org.apache.tez.runtime.metrics.TaskCounterUpdater.updateCounters(TaskCounterUpdater.java:118)
at org.apache.tez.runtime.RuntimeTask.setFrameworkCounters(RuntimeTask.java:172)
at org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:100)
at org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:37)
at org.apache.tez.common.CallableWithNdc.call(CallableWithNdc.java:36)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
This jira is to add null stat key check to FileSystemStorageStatistics.
Thanks hitesh for trying in Tez and reporting this.