XMLWordPrintableJSON

Details

    • Sub-task
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • 2.8.0, 3.0.0-alpha1
    • 2.8.0, 3.0.0-alpha1
    • fs
    • None

    Description

      Currently in FileSystemStorageStatistics we simply returns data from FileSystem#Statistics. However there is no null key check, which leads to NPE problems to downstream applications. For example, we got a NPE when passing a null key to FileSystemStorageStatistics#getLong(), exception stack as following:

      NullPointerException
      at org.apache.hadoop.fs.FileSystemStorageStatistics.fetch(FileSystemStorageStatistics.java:80)
      at org.apache.hadoop.fs.FileSystemStorageStatistics.getLong(FileSystemStorageStatistics.java:108)
      at org.apache.tez.runtime.metrics.FileSystemStatisticsUpdater2.updateCounters(FileSystemStatisticsUpdater2.java:60)
      at org.apache.tez.runtime.metrics.TaskCounterUpdater.updateCounters(TaskCounterUpdater.java:118)
      at org.apache.tez.runtime.RuntimeTask.setFrameworkCounters(RuntimeTask.java:172)
      at org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:100)
      at org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:37)
      at org.apache.tez.common.CallableWithNdc.call(CallableWithNdc.java:36)
      at java.util.concurrent.FutureTask.run(FutureTask.java:262)
      at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
      at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
      at java.lang.Thread.run(Thread.java:745)

      This jira is to add null stat key check to FileSystemStorageStatistics.

      Thanks hitesh for trying in Tez and reporting this.

      Attachments

        1. HADOOP-13288.000.patch
          1 kB
          Mingliang Liu
        2. HADOOP-13288.001.patch
          2 kB
          Mingliang Liu

        Activity

          People

            liuml07 Mingliang Liu
            liuml07 Mingliang Liu
            Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: