[THRIFT-4105] used too much memory randomly

• Type: Bug
• Status: Closed
• Priority: Critical
• Resolution: Not A Bug
• Component/s: Go - Library
• Assignee: Unassigned
• Reporter: chengning
• Created: 23/Feb/17 09:26
• Updated: 31/May/17 13:01
• Resolved: 31/May/17 13:01

Description

In my golang project used thrift, high memory usage once or twice a week.
Use go tool pprof can find the following information.
We did not use thrift to pass larger strings, but they allocate a string with the size of 352.20MB
I think there may be a problem

(pprof) top
704.39MB of 709.32MB total (99.31%)
Dropped 215 nodes (cum <= 3.55MB)
flat flat% sum% cum cum%
704.39MB 99.31% 99.31% 704.39MB 99.31% git.apache.org/thrift.git/lib/go/thrift.(*TBinaryProtocol).readStringBody
0 0% 99.31% 704.39MB 99.31% git.apache.org/thrift.git/lib/go/thrift.(*TBinaryProtocol).ReadMessageBegin
0 0% 99.31% 705.03MB 99.39% git.apache.org/thrift.git/lib/go/thrift.(*TSimpleServer).AcceptLoop.func1
0 0% 99.31% 705.03MB 99.39% git.apache.org/thrift.git/lib/go/thrift.(*TSimpleServer).processRequests
0 0% 99.31% 705.03MB 99.39% cn.com/rpc_server/thrift.(*VideoCallThriftProcessor).Process
0 0% 99.31% 708.82MB 99.93% runtime.goexit

(pprof) list readStringBody
Total: 709.32MB
ROUTINE ======================== git.apache.org/thrift.git/lib/go/thrift.(*TBinaryProtocol).readStringBody in /src/git.apache.org/thrift.git/lib/go/thrift/binary_protocol.go
704.39MB 704.39MB (flat, cum) 99.31% of Total
. . 482: }
. . 483: var buf []byte
. . 484: if int(size) <= len(p.buffer)

{ . . 485: buf = p.buffer[0:size] . . 486: }

else

{ 352.20MB 352.20MB 487: buf = make([]byte, size) . . 488: }

. . 489: _, e := io.ReadFull(p.trans, buf)
352.20MB 352.20MB 490: return string(buf), NewTProtocolException(e)
. . 491:}

(pprof) list ReadMessageBegin
Total: 709.32MB
ROUTINE ======================== git.apache.org/thrift.git/lib/go/thrift.(*TBinaryProtocol).ReadMessageBegin in /src/git.apache.org/thrift.git/lib/go/thrift/binary_protocol.go
0 704.39MB (flat, cum) 99.31% of Total
. . 256: return name, typeId, seqId, nil
. . 257: }
. . 258: if p.strictRead

{ . . 259: return name, typeId, seqId, NewTProtocolExceptionWithType(BAD_VERSION, fmt.Errorf("Missing version in ReadMessageBegin")) . . 260: }

. 704.39MB 261: name, e2 := p.readStringBody(size)
. . 262: if e2 != nil

{ . . 263: return name, typeId, seqId, e2 . . 264: }

. . 265: b, e3 := p.ReadByte()
. . 266: if e3 != nil {