-
Type:
Sub-task
-
Status: Resolved
-
Priority:
Major
-
Resolution: Fixed
-
Affects Version/s: 2.8.0, 3.0.0-alpha1
-
Fix Version/s: 2.8.0, 3.0.0-alpha1
-
Component/s: fs
-
Labels:None
-
Target Version/s:
Currently in FileSystemStorageStatistics we simply returns data from FileSystem#Statistics. However there is no null key check, which leads to NPE problems to downstream applications. For example, we got a NPE when passing a null key to FileSystemStorageStatistics#getLong(), exception stack as following:
NullPointerException
at org.apache.hadoop.fs.FileSystemStorageStatistics.fetch(FileSystemStorageStatistics.java:80)
at org.apache.hadoop.fs.FileSystemStorageStatistics.getLong(FileSystemStorageStatistics.java:108)
at org.apache.tez.runtime.metrics.FileSystemStatisticsUpdater2.updateCounters(FileSystemStatisticsUpdater2.java:60)
at org.apache.tez.runtime.metrics.TaskCounterUpdater.updateCounters(TaskCounterUpdater.java:118)
at org.apache.tez.runtime.RuntimeTask.setFrameworkCounters(RuntimeTask.java:172)
at org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:100)
at org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:37)
at org.apache.tez.common.CallableWithNdc.call(CallableWithNdc.java:36)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
This jira is to add null stat key check to FileSystemStorageStatistics.
Thanks Hitesh Shah for trying in Tez and reporting this.