Details

    • Type: Bug
    • Status: Resolved
    • Priority: Blocker
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: None
    • Component/s: Data Collection
    • Labels:
      None

      Description

      Current FileAdaptor has several issue but the most critical one is the fac that the remote application could be blocked waiting on the socket if the InMemoryQueue is full.

      1. CHUKWA-70.patch
        9 kB
        Jerome Boulon

        Issue Links

          Activity

          Hide
          asrabkin Ari Rabkin added a comment -

          How critical a problem is that? In production, does the queue actually come close to filling? I'd have thought that the data rate per agent was only a few k or tens of k per second, and that unless all the collectors are down, this won't be a problem.

          Show
          asrabkin Ari Rabkin added a comment - How critical a problem is that? In production, does the queue actually come close to filling? I'd have thought that the data rate per agent was only a few k or tens of k per second, and that unless all the collectors are down, this won't be a problem.
          Hide
          jboulon Jerome Boulon added a comment -

          Even if collectors are not down, if you add a new file (more than 15MB) the queue will become full for a short period of time. During that period if someone try to register a file to this adaptor, the agent/remote application will block.

          Show
          jboulon Jerome Boulon added a comment - Even if collectors are not down, if you add a new file (more than 15MB) the queue will become full for a short period of time. During that period if someone try to register a file to this adaptor, the agent/remote application will block.
          Hide
          eyang Eric Yang added a comment -

          There is a quick patch that we tested internally, where it uses a background thread to send the chunk. I think that should be sufficient to prevent the blocking, isn't it? The background thread should check for queue availability, and retry when necessary. I believe this is more like a patch than rewrite.

          Show
          eyang Eric Yang added a comment - There is a quick patch that we tested internally, where it uses a background thread to send the chunk. I think that should be sufficient to prevent the blocking, isn't it? The background thread should check for queue availability, and retry when necessary. I believe this is more like a patch than rewrite.
          Hide
          jboulon Jerome Boulon added a comment -

          I'm the one that wrote this quick emergency fix and it looks like an emergency fix not something that we want to keep and maintain ...

          Show
          jboulon Jerome Boulon added a comment - I'm the one that wrote this quick emergency fix and it looks like an emergency fix not something that we want to keep and maintain ...
          Hide
          eyang Eric Yang added a comment -

          What do you have in mind as a proper fix?

          Show
          eyang Eric Yang added a comment - What do you have in mind as a proper fix?
          Hide
          jboulon Jerome Boulon added a comment -

          Implementation is similar to the FileTailingAdaptor

          • 1 static tailer for all files
          • Add command, just register the file
          • shutdown is not doing anything since the goal is to let the agent send it over
          • stop remove from the tailer
          • lastModified time should at least one minute ago (best effort to get a complete file)
          • timeOut after 5 minutes (Similar to TerminatorThread timeout)
          Show
          jboulon Jerome Boulon added a comment - Implementation is similar to the FileTailingAdaptor 1 static tailer for all files Add command, just register the file shutdown is not doing anything since the goal is to let the agent send it over stop remove from the tailer lastModified time should at least one minute ago (best effort to get a complete file) timeOut after 5 minutes (Similar to TerminatorThread timeout)
          Hide
          asrabkin Ari Rabkin added a comment -

          I just committed this. Thanks for taking this on, Jerome!

          By the way, at some point I think we should rename FileAdaptor. since the name isn't particularly descriptive. It's an incompatible change, so we might not want to do it right now...

          Show
          asrabkin Ari Rabkin added a comment - I just committed this. Thanks for taking this on, Jerome! By the way, at some point I think we should rename FileAdaptor. since the name isn't particularly descriptive. It's an incompatible change, so we might not want to do it right now...

            People

            • Assignee:
              jboulon Jerome Boulon
              Reporter:
              jboulon Jerome Boulon
            • Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development