Avro
  1. Avro
  2. AVRO-1069

HttpTransceiver never closes its OutputStream, hinders java reuse of HTTP connections

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 1.7.1
    • Component/s: java
    • Labels:
      None

      Description

      The class org.apache.avro.ipc.HttpTransceiver opens an OutputStream and never explicitly closes it. That seems like very bad behavior.

      I think you should also be closing the InputStream.

      In particular, Java has built-in the ability to keep HttpURLConnections open, and re-use them. You might think that not closing these streams would help Java in this effort, but actually, the streams are not the raw connections, but wrappers. The javadoc says: "Calling the close() methods on the InputStream or OutputStream of an HttpURLConnection after a request may free network resources associated with this instance but has no effect on any shared persistent connection."

      In other words, when you fail to close these streams, Java doesn't know you are done with the request, so it cannot re-use the connection. You only end up able to re-use the connection when the HttpURLConnection gets garbage-collected.

      1. AVRO-1069.patch
        1 kB
        Doug Cutting

        Activity

        Thomas Andrews created issue -
        Thomas Andrews made changes -
        Field Original Value New Value
        Description I'm not sure if this is the cause of my underlying problem, but the class org.apache.avro.ipc.HttpTransceiver opens an OutputStream and never explicitly closes it. That seems like very bad behavior, especially depending on whether the HttpURLConnection class holds onto the OutputStream or not - if it does, then potentially the Transceiver will never close the OutputStream.

        The specific oddity I am seeing is related to Flume. I have an Flume agent and a client using a FlumeEventAvroServer with the Avro HttpTransceiver class.

        Occasionally, when the Avro agent gets refreshed or otherwise reset, it fails with a BindException error.

        Even rarer, but still often enough to cause concern, the client software holds onto something which blocks the Flume agent from rebinding to the port even when the Flume agent is restarted. Essentially, something is now "occupying" the port. It is my guess that it is the transceiver which has failed to close the OutputStream, but I'm not 100% sure.

        I think you should also be closing the InputStream, as well.
        I'm not sure if this is the cause of my underlying problem, but the class org.apache.avro.ipc.HttpTransceiver opens an OutputStream and never explicitly closes it. That seems like very bad behavior, especially depending on whether the HttpURLConnection class holds onto the OutputStream or not - if it does, then potentially the Transceiver will never close the OutputStream.

        The specific oddity I am seeing is related to Flume. I have a Flume agent listening on port 60000 and a client using a FlumeEventAvroServer with the Avro HttpTransceiver class.

        Occasionally, when the Avro agent gets refreshed or otherwise reset, it fails with a BindException error.

        Even rarer, but still often enough to cause concern, the client software holds onto something which blocks the Flume agent from rebinding to the port even when the Flume agent is restarted. Essentially, something is now "occupying" the port. It is my guess that it is the transceiver which has failed to close the OutputStream, but I'm not 100% sure.

        I think you should also be closing the InputStream, as well.
        Thomas Andrews made changes -
        Description I'm not sure if this is the cause of my underlying problem, but the class org.apache.avro.ipc.HttpTransceiver opens an OutputStream and never explicitly closes it. That seems like very bad behavior, especially depending on whether the HttpURLConnection class holds onto the OutputStream or not - if it does, then potentially the Transceiver will never close the OutputStream.

        The specific oddity I am seeing is related to Flume. I have a Flume agent listening on port 60000 and a client using a FlumeEventAvroServer with the Avro HttpTransceiver class.

        Occasionally, when the Avro agent gets refreshed or otherwise reset, it fails with a BindException error.

        Even rarer, but still often enough to cause concern, the client software holds onto something which blocks the Flume agent from rebinding to the port even when the Flume agent is restarted. Essentially, something is now "occupying" the port. It is my guess that it is the transceiver which has failed to close the OutputStream, but I'm not 100% sure.

        I think you should also be closing the InputStream, as well.
        I'm not sure if this is the cause of my underlying problem, but the class org.apache.avro.ipc.HttpTransceiver opens an OutputStream and never explicitly closes it. That seems like very bad behavior, especially depending on whether the HttpURLConnection class holds onto the OutputStream or not - if it does, then potentially the Transceiver will never close the OutputStream.

        I think you should also be closing the InputStream, as well.

        In particular, Java has built-in the ability to keep HttpURLConnections open, and re-use them. You might think that not closing these streams would help Java in this effort, but actually, the streams are not the raw connections, but wrappers. The javadoc says: "Calling the close() methods on the InputStream or OutputStream of an HttpURLConnection after a request may free network resources associated with this instance but has no effect on any shared persistent connection."

        In other words, when you fail to close these streams, Java doesn't know you are done with the request, so it cannot re-use the connection. You only end up able to re-use the connection when the HttpURLConnection gets garbage-collected.
        Thomas Andrews made changes -
        Description I'm not sure if this is the cause of my underlying problem, but the class org.apache.avro.ipc.HttpTransceiver opens an OutputStream and never explicitly closes it. That seems like very bad behavior, especially depending on whether the HttpURLConnection class holds onto the OutputStream or not - if it does, then potentially the Transceiver will never close the OutputStream.

        I think you should also be closing the InputStream, as well.

        In particular, Java has built-in the ability to keep HttpURLConnections open, and re-use them. You might think that not closing these streams would help Java in this effort, but actually, the streams are not the raw connections, but wrappers. The javadoc says: "Calling the close() methods on the InputStream or OutputStream of an HttpURLConnection after a request may free network resources associated with this instance but has no effect on any shared persistent connection."

        In other words, when you fail to close these streams, Java doesn't know you are done with the request, so it cannot re-use the connection. You only end up able to re-use the connection when the HttpURLConnection gets garbage-collected.
        The class org.apache.avro.ipc.HttpTransceiver opens an OutputStream and never explicitly closes it. That seems like very bad behavior.

        I think you should also be closing the InputStream.

        In particular, Java has built-in the ability to keep HttpURLConnections open, and re-use them. You might think that not closing these streams would help Java in this effort, but actually, the streams are not the raw connections, but wrappers. The javadoc says: "Calling the close() methods on the InputStream or OutputStream of an HttpURLConnection after a request may free network resources associated with this instance but has no effect on any shared persistent connection."

        In other words, when you fail to close these streams, Java doesn't know you are done with the request, so it cannot re-use the connection. You only end up able to re-use the connection when the HttpURLConnection gets garbage-collected.
        Thomas Andrews made changes -
        Summary HttpTransceiver never closes its OutputStream HttpTransceiver never closes its OutputStream, hinders java reuse of HTTP connections
        Doug Cutting made changes -
        Attachment AVRO-1069.patch [ 12531718 ]
        Doug Cutting made changes -
        Status Open [ 1 ] Patch Available [ 10002 ]
        Assignee Doug Cutting [ cutting ]
        Fix Version/s 1.7.1 [ 12321552 ]
        Doug Cutting made changes -
        Status Patch Available [ 10002 ] Resolved [ 5 ]
        Resolution Fixed [ 1 ]
        Doug Cutting made changes -
        Status Resolved [ 5 ] Closed [ 6 ]

          People

          • Assignee:
            Doug Cutting
            Reporter:
            Thomas Andrews
          • Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development