[THRIFT-5733] Building code with circular `include`s can result in tons of memory usage and eventual segfault

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 0.18.1
• Fix Version/s: 0.20.0
• Component/s: Compiler (General)

Environment

I'm on Linux, but this also happens to my coworkers on macOS.

• Assignee: Jens Geyer
• Reporter: Jeremy Fleischman
• Created: 02/Sep/23 19:05
• Updated: 22/Mar/24 23:06
• Resolved: 03/Sep/23 08:55

Description

If I try to build the following thrift code, it pretty quickly segfaults:

setup:

$ cat foo.thrift
include "bar.thrift"
$ cat bar.thrift
include "foo.thrift"

build:

$ thrift --allow-64bit-consts --gen py:slots foo.thrift
[2] 210654 segmentation fault (core dumped) thrift --allow-64bit-consts --gen py:slots foo.thrift

Not very user friendly error message I've ever received , but pretty must just a cosmetic issue (maybe there's a buffer overflow somewhere and some potential security exploit to worry about if you're compiling untrusted thrift code, but I personally never do that, so it doesn't stress me out).

However, if you add a 3rd file to the mix, things can get really weird. If I try to build the following code, it'll suck up all 32 GiB of RAM on my machine and render my computer completely unusable. If you reduce the number of entries in LargeEnum, you can get the thrift compiler to use a ton of RAM before it finally segfaults as in the first example. I've attached a screenshot so you can see how RAM and CPU gets used on my machine while attempting to build the above code.

problematic code:

$ cat foo.thrift
include "bar.thrift"
$ cat bar.thrift
include "large-enum.thrift"
include "foo.thrift"
$ cat large-enum.thrift
enum LargeEnum {
    FOO0 = 0,
    FOO1 = 1,
    ... [FOO2 through FOO1998] ...
    FOO1999 = 1999,
}

I've also put together a simple repro on https://github.com/jfly/2023-09-01-thrift-circular-import which can autogenerate the 3 files described above. (Just be careful when running it that you kill it before it soaks up all of your ram!)

Yesterday, this explosive use of RAM brought our company's build server (with 128 GiB of RAM!) to its knees. We spent a lot of time flailing around before we finally tracked it down to one problematic PR that introduced a circular include.

Issue Links

• links to: pull request 2851