Details

      Description

      Issue:

      Problem exists when there are multiple HDFS writers in writersMap. Each writer keeps an open hdfs handle to the file. Incase of Inactive writer(i.e. one which is not consuming any data from long period), the files are not closed and always remain in open state.

      Ideally, these files should get closed and Hdfs writers removed from the WritersMap.

      Solution:
      Implement a ClosingFilesPolicy that is based on Tick tuple intervals. At each tick tuple all Writers are checked and closed if they exist for a long time.

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                Unassigned
                Reporter:
                shoeb.aamer Shoeb
              • Votes:
                0 Vote for this issue
                Watchers:
                2 Start watching this issue

                Dates

                • Created:
                  Updated:

                  Time Tracking

                  Estimated:
                  Original Estimate - 48h
                  48h
                  Remaining:
                  Time Spent - 50m Remaining Estimate - 47h 10m
                  47h 10m
                  Logged:
                  Time Spent - 50m Remaining Estimate - 47h 10m
                  50m