Traffic Server
  1. Traffic Server
  2. TS-1247

Data from a response transform in a plugin causes excess memory usage

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Minor Minor
    • Resolution: Duplicate
    • Affects Version/s: 3.0.4
    • Fix Version/s: 3.3.2
    • Component/s: HTTP
    • Labels:
      None

      Description

      If you have a plugin implementing a response transform and the data sending to the UA is slow TS will use huge amounts of memory for larger files. (e.g. 16G Vsize for a 50M file.)
      This seems to be several problems combined, first, the MIOBuffer used for the chunking tunnel uses 2M buffers, and when handling data from a transform it ends up putting just the chunk markers in a buffer, then gets a buffer from the transform, then a new 2M buffer...Once I fix that, 50M files are ok, but 1G files still exhaust memory as they end up all in memory because there is no flow-control in some of the VConnections and such between a transform and the chunking tunnel. I did a lame patch that I need to revise that fixed the problem by making TransformTerminus not send data if the destination that it is writing to has too much data in it, plus I made my plugin check how much data the destination for the transform has and don't write to it then either. But it seems like I need to find the other side of both of those, and suppress WRITE_READY (and maybe IMMEDIATE) event sending when they are full, but I haven't been able to figure out what the other side is for those yet.

        Issue Links

          Activity

          William Bardwell created issue -
          Leif Hedstrom made changes -
          Field Original Value New Value
          Fix Version/s 3.3.0 [ 12316495 ]
          Leif Hedstrom made changes -
          Fix Version/s 3.3.1 [ 12321686 ]
          Fix Version/s 3.3.0 [ 12316495 ]
          Leif Hedstrom made changes -
          Fix Version/s 3.3.2 [ 12321745 ]
          Fix Version/s 3.3.1 [ 12321686 ]
          Leif Hedstrom made changes -
          Fix Version/s 3.3.3 [ 12322479 ]
          Fix Version/s 3.3.2 [ 12321745 ]
          Alan M. Carroll made changes -
          Assignee Alan M. Carroll [ amc ]
          Alan M. Carroll made changes -
          Link This issue Is contained by TS-1496 [ TS-1496 ]
          Alan M. Carroll made changes -
          Link This issue Is contained by TS-1457 [ TS-1457 ]
          Alan M. Carroll made changes -
          Status Open [ 1 ] Resolved [ 5 ]
          Fix Version/s 3.3.2 [ 12321745 ]
          Fix Version/s 3.3.3 [ 12322479 ]
          Resolution Duplicate [ 3 ]
          Leif Hedstrom made changes -
          Status Resolved [ 5 ] Closed [ 6 ]

            People

            • Assignee:
              Alan M. Carroll
              Reporter:
              William Bardwell
            • Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development