Uploaded image for project: 'Apache Storm'
  1. Apache Storm
  2. STORM-2745

Hdfs Open Files problem

Log workAgile BoardRank to TopRank to BottomAttach filesAttach ScreenshotBulk Copy AttachmentsBulk Move AttachmentsAdd voteVotersWatch issueWatchersCreate sub-taskConvert to sub-taskLinkCloneLabelsUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

Details

    Description

      Issue:

      Problem exists when there are multiple HDFS writers in writersMap. Each writer keeps an open hdfs handle to the file. Incase of Inactive writer(i.e. one which is not consuming any data from long period), the files are not closed and always remain in open state.

      Ideally, these files should get closed and Hdfs writers removed from the WritersMap.

      Solution:
      Implement a ClosingFilesPolicy that is based on Tick tuple intervals. At each tick tuple all Writers are checked and closed if they exist for a long time.

      Attachments

        Activity

          This comment will be Viewable by All Users Viewable by All Users
          Cancel

          People

            Unassigned Unassigned Assign to me
            shoeb.aamer Shoeb

            Dates

              Created:
              Updated:

              Time Tracking

              Estimated:
              Original Estimate - 48h
              48h
              Remaining:
              Time Spent - 50m Remaining Estimate - 47h 10m
              47h 10m
              Logged:
              Time Spent - 50m Remaining Estimate - 47h 10m
              50m

              Slack

                Issue deployment