Uploaded image for project: 'Apache Storm'
  1. Apache Storm
  2. STORM-2745

Hdfs Open Files problem

    XMLWordPrintableJSON

Details

    Description

      Issue:

      Problem exists when there are multiple HDFS writers in writersMap. Each writer keeps an open hdfs handle to the file. Incase of Inactive writer(i.e. one which is not consuming any data from long period), the files are not closed and always remain in open state.

      Ideally, these files should get closed and Hdfs writers removed from the WritersMap.

      Solution:
      Implement a ClosingFilesPolicy that is based on Tick tuple intervals. At each tick tuple all Writers are checked and closed if they exist for a long time.

      Attachments

        Issue Links

          Activity

            People

              Unassigned Unassigned
              shoeb.aamer Shoeb
              Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Time Tracking

                  Estimated:
                  Original Estimate - 48h
                  48h
                  Remaining:
                  Time Spent - 50m Remaining Estimate - 47h 10m
                  47h 10m
                  Logged:
                  Time Spent - 50m Remaining Estimate - 47h 10m
                  50m