Uploaded image for project: 'Flume'
  1. Flume
  2. FLUME-2306

onIdleCallback is not canceled when stop hdfs sink

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Duplicate
    • Affects Version/s: 1.4.0
    • Fix Version/s: None
    • Component/s: Sinks+Sources
    • Labels:
      None

      Description

      the hdfs sink cached 5000 open files by default and it cost quite a lot of memory in total when using lzo CompressedStream. We should open the idleTimeout feature to resolve it. But there seems to be a bug with this feature. When stopping flume, HDFSWriter does not cancel the idle scheduler, which might cause flume not to stop. So I extend the current close() method in HDFSWriter as follows, and use it in HDFSEventSink when stop the sink component :

      /**

      • when stop flume, all schedulers should be canceled
      • @param cancelIdleCallback
      • @throws IOException
      • @throws InterruptedException

      */

      public void close(boolean cancelIdleCallback) throws IOException, InterruptedException{

      close();

      if(cancelIdleCallback){

      if (idleFuture != null && !idleFuture.isDone())

      { idleFuture.cancel(false); // do not cancel myself if running! idleFuture = null; }

      }

      }

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                Unassigned
                Reporter:
                chenshangan521@163.com chenshangan
              • Votes:
                0 Vote for this issue
                Watchers:
                2 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: