Blocker Description
I have a fairly big table (~100GBs) that, when bootstrapped on current master, fails after a while with:
0118 12:37:05.870913 60088320 rowset_info.cc:73] Check failed: imax.compare(max) <= 0 (1 vs. 0)
*** Check failure stack trace: ***
@ 0x105f43233 google::LogMessage::SendToLog()
@ 0x105f43841 google::LogMessage::Flush()
@ 0x105f472df google::LogMessageFatal::~LogMessageFatal()
@ 0x105f440b9 google::LogMessageFatal::~LogMessageFatal()
@ 0x1033b69d2 kudu::tablet::(anonymous namespace)::DCheckInside()
@ 0x1033b659b kudu::tablet::(anonymous namespace)::StringFractionInRange()
@ 0x1033b48be kudu::tablet::(anonymous namespace)::WidthByDataSize()
@ 0x1033b37ba kudu::tablet::RowSetInfo::CollectOrdered()
@ 0x1033435b1 kudu::tablet::BudgetedCompactionPolicy::SetupKnapsackInput()
@ 0x103343a76 kudu::tablet::BudgetedCompactionPolicy::PickRowSets()
@ 0x103262349 kudu::tablet::Tablet::UpdateCompactionStats()
@ 0x103261dd6 kudu::tablet::CompactRowSetsOp::UpdateStats()
@ 0x10336db4a kudu::MaintenanceManager::FindBestOp()
@ 0x10336bd31 kudu::MaintenanceManager::RunSchedulerThread()
@ 0x103375f02 boost::_mfi::mf0<>::operator()()
@ 0x103375e50 boost::_bi::list1<>::operator()<>()
@ 0x103375dcc boost::_bi::bind_t<>::operator()()
@ 0x103375b8d boost::detail::function::void_function_obj_invoker0<>::invoke()
@ 0x105a4e60b boost::function0<>::operator()()
@ 0x105ad51c4 kudu::Thread::SuperviseThread()
@ 0x7fff93aebc13 _pthread_body
@ 0x7fff93aebb90 _pthread_start
@ 0x7fff93ae9375 thread_start
This happens always, after a while, and I knew it didn't happen a few weeks ago so I ran git bisect and the culprit is the patch that enables c++ 11. I don't know whether this is a mac only problem or a general problem.
Git bisect results:
9806863e78107505a622b44112a897189d9b3c24 is the first bad commit
commit 9806863e78107505a622b44112a897189d9b3c24
Author: Dan Burkert <dan@cloudera.com>
Date: Mon Nov 30 12:15:36 2015 -0800
Enable C++11
This is a rollup of changes necessary to move Kudu to C++11.
Component changes have been individually reviewed, as follows:
-----------------------------------------------------------------------
Enable Red Hat devtoolset in RHEL 6 Environments
We are moving to C++11 in the near future, and will require a C++11 compatible
compiler on all suported platforms. Since the RHEL 6 system compiler (GCC 4.4)
does not support C++11, we need an alternate. The Red Hat devtoolset provides a
toolchain with GCC 4.9. Libraries (such as libkudu_client.so) which are compiled
with the Red Hat devtoolset toolchain automatically statically link portions of
the standard library which are new for C++11. This will allow us to take
advantage of C++11 features in the client, but still remain compatible with
stock RHEL 6 systems.
This commit modifies the Jenkins build and test scripts to take advantage of the
devtoolset toolchain, hides the std library symbols that the devtoolset
links into library objects, and adds the devtoolset location to the default gcc
toolchain search paths for the thirdparty LLVM compiler. Additionally, since a
system compiler which supports C++11 can now be assumed, the precompiled clang
compiler is no longer a dependency on Linux and has been removed.
Change Id: I5505911044aa9fda8fb5010c6fe446c8f5c5616e
Build thirdparty clang with system compiler
We build LLVM 3.4.2 in thirdparty, but it requires a c++11 compiler to build.
Now that we require the system compiler to be c++11 compatible, we no longer
need the bootstrapping compiler.
Change Id: I4800d9eec0a566bcf71c59abdeae6a6d8b6cb40a
http://gerrit.cloudera.org:8080/1476
-----------------------------------------------------------------------
Enable C++11
Change Id: Ia9463b46428450e1a29c1684d47b46f96bf965cb
http://gerrit.cloudera.org:8080/#/c/1466
-----------------------------------------------------------------------
C++11: replace list_of with initializer lists
Change Id: I0d258e35a38be3a47854e51bd110dc07f05bdd3f
http://gerrit.cloudera.org:8080/#/c/1467
-----------------------------------------------------------------------
Replace boost::tuple with std::tuple
Change Id: Ia258a1833a4ae3509a3781717386324fdd63ef88
http://gerrit.cloudera.org:8080/#/c/1468
-----------------------------------------------------------------------
C++11: replace util::gtl::is_sorted with std::is_sorted
Change Id: Ifc51c4519c5c055aa62887fc6dcc1836572102f5
http://gerrit.cloudera.org:8080/#/c/1470
-----------------------------------------------------------------------
C++11: remove tr1
Change Id: I6c5559f9d14860fd568f86809377d94e5e1d3e6e
http://gerrit.cloudera.org:8080/#/c/1472
-----------------------------------------------------------------------
Add kudu::client::sp::shared_ptr as public shared pointer type
Change Id: I02d1c3691294f82498b9dfd25253cfca0b196dda
http://gerrit.cloudera.org:8080/#/c/1473
-----------------------------------------------------------------------
Replace gnu_cxx::hash with std::hash
gnu_cxx::hash is obsolete since c++11 introduced std::hash.
Change Id: Ic7e6af40cf8e6ca5069ad0185d4c136c4793fc31
http://gerrit.cloudera.org:8080/#/c/1477
-----------------------------------------------------------------------
Add OS X search paths when compiling with thirdparty clang
Change Id: I56051b321c9c9d3a1a521b4188e47961f64acd50
http://gerrit.cloudera.org:8080/#/c/1478
-----------------------------------------------------------------------
Replace boost::unique_lock with std::unique_lock
These particular uses of boost locks were failing to compile on Centos6 because
the system boost version is buggy when compiling with move semantics.
Change Id: I424077b29ac2ae10761df4afb12f554213c049be
http://gerrit.cloudera.org:8080/#/c/1479
-----------------------------------------------------------------------
Replace boost::shared_lock with kudu::shared_lock in WriteTransaction
The Boost shared_lock implementation distributed with Centos6 does not compile
with move semantics (specifically the operator= method). std::shared_lock is not
available till c++14.
Change Id: Icd42e3cb88f8c5e5d992d3f0d0324b6c9f81d915
http://gerrit.cloudera.org:8080/#/c/1480
-----------------------------------------------------------------------
Update mem tracker allocator for libc++ compatibility
The unordered_map type in libc++ is finicky about custom allocators.
Change Id: I8ca226abb2c7e3ca95c68533979e97dea4552b63
http://gerrit.cloudera.org:8080/#/c/1481
-----------------------------------------------------------------------
Update thirdparty llvm from 3.4 to 3.7
This updates our thirdparty LLVM version from 3.4 to 3.7. Major changes:
* ASAN and TSAN builds now use the thirdparty clang instead of the bootstrapping
clang toolchain. The clang toolchain has been removed from the build process.
* Python 2.7 is now built in thirdparty on systems which have a different system
Python version. Python 2.7 is a build dependency of LLVM 3.6+.
* Updated LLVM API calls in the codegen module.
Change Id: I9023dd138373bb580892877e437b9199b690bde3
http://gerrit.cloudera.org:8080/#/c/1572
-----------------------------------------------------------------------
Build instrumented thirdparty with libstdc++ in TSAN mode
The new c++11 toolchain is more sensitive to TSAN issues in the standard
library. This commit adds a TSAN instrumented libstdcxx 4.9.3 to the thirdparty
build when building with TSAN.
Change Id: I2157469910c38b6e1c750e431c564e7d2f7b7e58
http://gerrit.cloudera.org:8080/#/c/1657
-----------------------------------------------------------------------
Update YCM config for c++11
Change Id: I64b24b148d19fe3b0fa0acd655f081143118b191
http://gerrit.cloudera.org:8080/#/c/1658
-----------------------------------------------------------------------
Remove import of macos mach-o/loader.h from gutil/port.h
The llvm/Support/MachO.h header redefines constants as enum variants using the
same identifier as macro constants declared in the mach-o/loader.h system
header. This causes issues when macho-o/loader.h is included before the LLVM
header.
This commit removes the reliance on mach-o/loader.h in gutil/port.h by removing
some unused macros for working with attribute sections in binaries. The solution
is not perfect, since the system header may be necessary in the future, but hopefully
by then a fix will have landed in upstream LLVM
(https://llvm.org/bugs/show_bug.cgi?id=25871).
Change Id: Ib41ee2b53f39589d48622a12378a51e1a49c74bb
http://gerrit.cloudera.org:8080/#/c/1592
-----------------------------------------------------------------------
C++11: Simplify thirdparty TSAN build
This commit addresses Adar's feedback on the C++11 patch. TSAN instrumented
dependencies are now always built on Linux and installed into an alternate
prefix. This commit will be squashed into the C++11 mega patch prior to
submission.
The thirdparty directory now includes three prefixes: thirdparty/installed for
tools and C-only libraries, thirdparty/installed-deps for C++ dependencies, and
thirdparty/installed-deps-tsan for C++ dependencies for TSAN builds. Keeping the
prefixes seperate allows us to not require rebuilding thirdparty when switching
to or from a TSAN instrumented build.
Change Id: I5e9824db19383556da88053779bdde1e66cdfa44
-----------------------------------------------------------------------
Change-Id: Ic52cbf98f583a3a953b1910e0017a8af448660d7
Reviewed-on: http://gerrit.cloudera.org:8080/1687
Reviewed-by: Dan Burkert <dan@cloudera.com>
Tested-by: Dan Burkert <dan@cloudera.com>
:100644 100644 38d2071b50d7e235eda11053181852e6cfbd487a e4537343c1fcd8c88477b70ee42591917eeeae57 M .ycm_extra_conf.py
:100644 100644 90ea9c95d8f630bf5ea1ea166c43d4deb682fd4c f803da71df0ae18e8a380f01bdcc446b8c9c91c6 M CMakeLists.txt
:100644 100644 7f03ac70584d4f5a522ebc30c401b45dbda539e7 c24a731d68e4121aa88ae968dd1c2429da3af30f M README.adoc
:040000 040000 e049c85ff18124127683cad228edcb546ae9398f 4c66c1fb76901d336ca76ed6438560b236a6900a M build-support
:040000 040000 643ad42af5e4b90cc28b3f1eee00b7b9b0496fb8 43c19d66e503ee3a3f3a8c0dcd77f75ac03bcd3f M cmake_modules
:040000 040000 a53aedb3c452ab78c9eb68874e06b03d5c6a4e56 e425ad4afb97b5ee88af5097b6284c6b22b0520a M java
:040000 040000 b51eaf41acad0abe9db8999db63aecbff63e7139 220757244d44805199d615102bc8e3b69a732712 M src
:040000 040000 2c124089a9e2e1141195cbb45429f4a101e74684 0bde0f74bb9eed9e9b61fb2d945fc784eeb3c29c M thirdparty