Major Description
The rapidjson code that parses numbers mishandles kMinInt32 and kMinInt64. Moreover, it only appears to do so when compiled with -O3 (the default for RELEASE builds) by certain compilers. I could not reproduce the mishandling when compiling with no optimizations (DEBUG) or with -O1 (FASTDEBUG).
I can't speak to kMinInt64, but when parsing kMinInt32, rapidjson identifies the number as a possible unsigned, unsigned long, signed, or signed long, instead of just a possible signed or signed long. Currently, this manifests as a SIGSEGV in jsonreader-test.cc:
*** Aborted at 1551300683 (unix time) try "date -d @1551300683" if you are using GNU date *** PC: @ 0x55d7ab64ae19 kudu::JsonReader::ExtractUint32() *** SIGSEGV (@0x0) received by PID 10903 (TID 0x7f578ebdf840) from PID 0; stack trace: *** @ 0x7f578da84890 (unknown) @ 0x55d7ab64ae19 kudu::JsonReader::ExtractUint32() @ 0x55d7ab5f95cf kudu::JsonReaderTest_SignedAndUnsignedInts_Test::TestBody() @ 0x55d7ab83d0cd testing::internal::HandleExceptionsInMethodIfSupported<>() @ 0x55d7ab8331f2 testing::Test::Run() @ 0x55d7ab8332d4 testing::TestInfo::Run() @ 0x55d7ab833417 testing::TestCase::Run() @ 0x55d7ab8338e8 testing::internal::UnitTestImpl::RunAllTests() @ 0x55d7ab83d5ad testing::internal::HandleExceptionsInMethodIfSupported<>() @ 0x55d7ab833a41 testing::UnitTest::Run() @ 0x55d7ab5e5776 main @ 0x7f578d6a2b97 __libc_start_main @ 0x55d7ab5eba7a _start
The offending line is:
ASSERT_TRUE(r.ExtractUint32(r.root(), signed_small32, nullptr).IsInvalidArgument());
Internally, our JsonReader class successfully parses the number as an unsigned, which then leads to a crash because we passed in 'nullptr' as the result (because we expected an earlier parsing error).
We've reproduced this with the gcc found on Ubuntu 18, but not the one in Ubuntu 16. It also repros if you use clang-6.0 from thirdparty with RELEASE builds.
We already knew that this was an issue as early as commit 02e82ca14, which disabled this test in UBSAN builds, because the mishandling triggers UBSAN:
thirdparty/installed/common/include/rapidjson/reader.h:644:18: runtime error: negation of -2147483648 cannot be represented in type 'int'; cast to an unsigned type to negate this value to itself
#0 0x7f75d4db9d58 in void rapidjson::GenericReader<rapidjson::UTF8<char>, rapidjson::MemoryPoolAllocator<rapidjson::CrtAllocator> >::ParseNumber<0u, rapidjson::GenericStringStream<rapidjson::UTF8<char> >, rapidjson::GenericDocument<rapidjson::UTF8<char>, rapidjson::MemoryPoolAllocator<rapidjson::CrtAllocator> > >(rapidjson::GenericStringStream<rapidjson::UTF8<char> >&, rapidjson::GenericDocument<rapidjson::UTF8<char>, rapidjson::MemoryPoolAllocator<rapidjson::CrtAllocator> >&) ../thirdparty/installed/common/include/rapidjson/reader.h:644:18
#1 0x7f75d4db6b84 in void rapidjson::GenericReader<rapidjson::UTF8<char>, rapidjson::MemoryPoolAllocator<rapidjson::CrtAllocator> >::ParseObject<0u, rapidjson::GenericStringStream<rapidjson::UTF8<char> >, rapidjson::GenericDocument<rapidjson::UTF8<char>, rapidjson::MemoryPoolAllocator<rapidjson::CrtAllocator> > >(rapidjson::GenericStringStream<rapidjson::UTF8<char> >&, rapidjson::GenericDocument<rapidjson::UTF8<char>, rapidjson::MemoryPoolAllocator<rapidjson::CrtAllocator> >&) ../thirdparty/installed/common/include/rapidjson/reader.h:290:4
#2 0x7f75d4db646f in bool rapidjson::GenericReader<rapidjson::UTF8<char>, rapidjson::MemoryPoolAllocator<rapidjson::CrtAllocator> >::Parse<0u, rapidjson::GenericStringStream<rapidjson::UTF8<char> >, rapidjson::GenericDocument<rapidjson::UTF8<char>, rapidjson::MemoryPoolAllocator<rapidjson::CrtAllocator> > >(rapidjson::GenericStringStream<rapidjson::UTF8<char> >&, rapidjson::GenericDocument<rapidjson::UTF8<char>, rapidjson::MemoryPoolAllocator<rapidjson::CrtAllocator> >&) ../thirdparty/installed/common/include/rapidjson/reader.h:243:15
#3 0x7f75d4db6105 in rapidjson::GenericDocument<rapidjson::UTF8<char>, rapidjson::MemoryPoolAllocator<rapidjson::CrtAllocator> >& rapidjson::GenericDocument<rapidjson::UTF8<char>, rapidjson::MemoryPoolAllocator<rapidjson::CrtAllocator> >::ParseStream<0u, rapidjson::GenericStringStream<rapidjson::UTF8<char> > >(rapidjson::GenericStringStream<rapidjson::UTF8<char> >&) ../thirdparty/installed/common/include/rapidjson/document.h:712:23
#4 0x7f75d4db543a in rapidjson::GenericDocument<rapidjson::UTF8<char>, rapidjson::MemoryPoolAllocator<rapidjson::CrtAllocator> >& rapidjson::GenericDocument<rapidjson::UTF8<char>, rapidjson::MemoryPoolAllocator<rapidjson::CrtAllocator> >::Parse<0u>(char const*) ../thirdparty/installed/common/include/rapidjson/document.h:745:10
#5 0x7f75d4db1db9 in kudu::JsonReader::Init() ../src/kudu/util/jsonreader.cc:65:13
#6 0x53aa5a in kudu::JsonReaderTest_SignedAndUnsignedInts_Test::TestBody() ../src/kudu/util/jsonreader-test.cc:159:3
#7 0x7f75d2cc3f9c in void testing::internal::HandleSehExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::*)(), char const*) /home/adar/Source/kudu/thirdparty/src/googletest-release-1.8.0/googletest/src/gtest.cc:2402
#8 0x7f75d2cc3f9c in void testing::internal::HandleExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::*)(), char const*) /home/adar/Source/kudu/thirdparty/src/googletest-release-1.8.0/googletest/src/gtest.cc:2438
#9 0x7f75d2cb9621 in testing::Test::Run() /home/adar/Source/kudu/thirdparty/src/googletest-release-1.8.0/googletest/src/gtest.cc:2474
#10 0x7f75d2cb9703 in testing::TestInfo::Run() /home/adar/Source/kudu/thirdparty/src/googletest-release-1.8.0/googletest/src/gtest.cc:2656
#11 0x7f75d2cb9846 in testing::TestCase::Run() /home/adar/Source/kudu/thirdparty/src/googletest-release-1.8.0/googletest/src/gtest.cc:2774
#12 0x7f75d2cb9d17 in testing::internal::UnitTestImpl::RunAllTests() /home/adar/Source/kudu/thirdparty/src/googletest-release-1.8.0/googletest/src/gtest.cc:4649
#13 0x7f75d2cc447c in bool testing::internal::HandleSehExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool>(testing::internal::UnitTestImpl*, bool (testing::internal::UnitTestImpl::*)(), char const*) /home/adar/Source/kudu/thirdparty/src/googletest-release-1.8.0/googletest/src/gtest.cc:2402
#14 0x7f75d2cc447c in bool testing::internal::HandleExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool>(testing::internal::UnitTestImpl*, bool (testing::internal::UnitTestImpl::*)(), char const*) /home/adar/Source/kudu/thirdparty/src/googletest-release-1.8.0/googletest/src/gtest.cc:2438
#15 0x7f75d2cb9e70 in testing::UnitTest::Run() /home/adar/Source/kudu/thirdparty/src/googletest-release-1.8.0/googletest/src/gtest.cc:4257
#16 0x7f75d57ed2c6 in main ../src/kudu/util/test_main.cc:106:13
#17 0x7f75d1943b96 in __libc_start_main csu/libc-start.c:310
#18 0x4325e9 in _start (/home/adar/Source/kudu/build/asan/bin/jsonreader-test+0x4325e9)
SUMMARY: UndefinedBehaviorSanitizer: undefined-behavior thirdparty/installed/common/include/rapidjson/reader.h:644:18 in