Giraph
  1. Giraph
  2. GIRAPH-319

Receiving two responses for a request causes an exception

    Details

    • Type: Bug Bug
    • Status: Resolved
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: None
    • Component/s: None
    • Labels:
      None

      Description

      Related to GIRAPH-306, and resending requests for which we haven't received response yet. If because of that we send request two times, and get two responses to it, the second one will cause an exception (ResponseClientHandler:111).

      1. GIRAPH-319.patch
        2 kB
        Alessandro Presta
      2. GIRAPH-319.patch
        1 kB
        Alessandro Presta

        Issue Links

          Activity

          Hide
          Alessandro Presta added a comment -

          Simply replaced the exception with a log. This way we don't notice if we ever get a response for a request we didn't send, but we also don't crash when we get a (legit) duplicate response.

          Show
          Alessandro Presta added a comment - Simply replaced the exception with a log. This way we don't notice if we ever get a response for a request we didn't send, but we also don't crash when we get a (legit) duplicate response.
          Hide
          Alessandro Presta added a comment -

          As per Avery's suggestion, also setting MAX_REQUEST_MILLISECONDS_DEFAULT to 10 minutes instead of 1.
          When the dataset is huge, we get a lot of resent requests which are just being processed very slowly.

          Show
          Alessandro Presta added a comment - As per Avery's suggestion, also setting MAX_REQUEST_MILLISECONDS_DEFAULT to 10 minutes instead of 1. When the dataset is huge, we get a lot of resent requests which are just being processed very slowly.
          Hide
          Avery Ching added a comment -

          +1, will commit in a little unless someone objects.

          Show
          Avery Ching added a comment - +1, will commit in a little unless someone objects.
          Hide
          Avery Ching added a comment -

          Thanks Maja for reporting and for the fix Alessandro.

          Show
          Avery Ching added a comment - Thanks Maja for reporting and for the fix Alessandro.
          Hide
          Eugene Koontz added a comment -

          Should this perhaps be a WARN, not an INFO? Or is it happening so often and typically in your experiences that it's not worth escalating to that level?

          Show
          Eugene Koontz added a comment - Should this perhaps be a WARN, not an INFO? Or is it happening so often and typically in your experiences that it's not worth escalating to that level?
          Hide
          Alessandro Presta added a comment -

          This happens almost any time a request has been resent, as we receive a response for both tries.

          Show
          Alessandro Presta added a comment - This happens almost any time a request has been resent, as we receive a response for both tries.
          Hide
          Eugene Koontz added a comment -

          Sorry, I meant, do you find that clients are often needing to resend requests? Perhaps it would be helpful to include an advisory to increase giraph.maxRequestMilliseconds if multiple responses are frequent. Something like:

          LOG.warn("messageReceived: Already received response for request id = " +
          requestId + ". If this is occurring frequently, consider increasing your giraph.maxRequestMilliseconds parameter.");

          Show
          Eugene Koontz added a comment - Sorry, I meant, do you find that clients are often needing to resend requests? Perhaps it would be helpful to include an advisory to increase giraph.maxRequestMilliseconds if multiple responses are frequent. Something like: LOG.warn("messageReceived: Already received response for request id = " + requestId + ". If this is occurring frequently, consider increasing your giraph.maxRequestMilliseconds parameter.");
          Hide
          Alessandro Presta added a comment -

          Oh ok, this sounds reasonable. Feel free to make the change (I'm on vacation without my work laptop).
          In my experience, a long-running job would systematically trigger many resent requests, which in turn caused the same number of responses to be sent back (leading to discovering this bug).
          I have yet to come across the case where a request is received multiple times but only one response comes back, hence in practice I always got both the "Already completed request" info and this... so the two seem kind of redundant, but I have no strong opinion.

          Show
          Alessandro Presta added a comment - Oh ok, this sounds reasonable. Feel free to make the change (I'm on vacation without my work laptop). In my experience, a long-running job would systematically trigger many resent requests, which in turn caused the same number of responses to be sent back (leading to discovering this bug). I have yet to come across the case where a request is received multiple times but only one response comes back, hence in practice I always got both the "Already completed request" info and this... so the two seem kind of redundant, but I have no strong opinion.

            People

            • Assignee:
              Alessandro Presta
              Reporter:
              Maja Kabiljo
            • Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development