Details
-
Improvement
-
Status: Resolved
-
Blocker
-
Resolution: Fixed
-
None
-
ghx-label-8
Description
We still have a minimum CPU requirement of SSSE3 support https://impala.apache.org/docs/build/html/topics/impala_prereqs.html. I.e. we don't assume SSE4.2 or AVX or AVX2.
There is a lot of legacy code to support CPUs without SSE4.2 and various other extensions. As a start, here are all the locations in the code where we branch based on CPU feature:
:~/impala/impala$ git grep CpuInfo::IsSupport
be/src/benchmarks/int-hash-benchmark.cc: if (CpuInfo::IsSupported(CpuInfo::SSE4_2)) suite32.BENCH(uint32_t, CRC);
be/src/benchmarks/int-hash-benchmark.cc: if (CpuInfo::IsSupported(CpuInfo::SSE4_2)) {
be/src/benchmarks/int-hash-benchmark.cc: if (CpuInfo::IsSupported(CpuInfo::SSE4_1)) {
be/src/benchmarks/int-hash-benchmark.cc: if (CpuInfo::IsSupported(CpuInfo::AVX2)) {
be/src/benchmarks/string-compare-benchmark.cc: if (CpuInfo::IsSupported(CpuInfo::SSE4_2)) {
be/src/exec/delimited-text-parser.cc: if (CpuInfo::IsSupported(CpuInfo::SSE4_2)) {
be/src/exec/delimited-text-parser.cc: if (CpuInfo::IsSupported(CpuInfo::SSE4_2)) {
be/src/exec/delimited-text-parser.inline.h: DCHECK(CpuInfo::IsSupported(CpuInfo::SSE4_2));
be/src/exec/delimited-text-parser.inline.h: if (LIKELY(CpuInfo::IsSupported(CpuInfo::SSE4_2))) {
be/src/runtime/io/disk-io-mgr.cc: if (!CpuInfo::IsSupported(CpuInfo::SSE4_2)) {
be/src/util/bit-util-test.cc: if (CpuInfo::IsSupported(CpuInfo::SSSE3)) {
be/src/util/bit-util-test.cc: if (CpuInfo::IsSupported(CpuInfo::AVX2)) {
be/src/util/bit-util-test.cc: if (CpuInfo::IsSupported(cpu_info_flag)) {
be/src/util/bit-util-test.cc:// CpuInfo::IsSupported() checks. This doesn't test the bug precisely but is a canary for
be/src/util/bit-util.cc: if (CpuInfo::IsSupported(CpuInfo::AVX2)) {
be/src/util/bit-util.cc: } else if (LIKELY(CpuInfo::IsSupported(CpuInfo::SSSE3))) {
be/src/util/bit-util.cc: if (LIKELY(CpuInfo::IsSupported(CpuInfo::SSSE3))) {
be/src/util/bit-util.h: if (LIKELY(CpuInfo::IsSupported(CpuInfo::POPCNT))) {
be/src/util/bloom-filter.cc: if (CpuInfo::IsSupported(CpuInfo::AVX)) {
be/src/util/bloom-filter.h: if (CpuInfo::IsSupported(CpuInfo::AVX2)) {
be/src/util/bloom-filter.h: if (CpuInfo::IsSupported(CpuInfo::AVX2)) {
be/src/util/cpu-info.cc: if (!CpuInfo::IsSupported(CpuInfo::SSSE3)) {
be/src/util/cpu-info.h: /// // line, CpuInfo::IsSupported(CpuInfo::AVX2) will return false.
be/src/util/cpu-info.h: : feature_(feature), reenable_(CpuInfo::IsSupported(feature)) {
be/src/util/hash-util.h: DCHECK(CpuInfo::IsSupported(CpuInfo::SSE4_2));
be/src/util/hash-util.h: DCHECK(CpuInfo::IsSupported(CpuInfo::SSE4_2));
be/src/util/hash-util.h: DCHECK(CpuInfo::IsSupported(CpuInfo::SSE4_2));
be/src/util/hash-util.h: DCHECK(CpuInfo::IsSupported(CpuInfo::SSE4_2));
be/src/util/hash-util.h: DCHECK(CpuInfo::IsSupported(CpuInfo::SSE4_2));
be/src/util/hash-util.h: DCHECK(CpuInfo::IsSupported(CpuInfo::SSE4_2));
be/src/util/hash-util.h: DCHECK(CpuInfo::IsSupported(CpuInfo::SSE4_2));
be/src/util/hash-util.h: if (LIKELY(CpuInfo::IsSupported(CpuInfo::SSE4_2))) {
be/src/util/openssl-util.cc: return (CpuInfo::IsSupported(CpuInfo::PCLMULQDQ)
We also ship two versions of the codegen module, one of which (nosse42) is essentially never used.
I think it would be uncontroversial to bump the minimum requirement to SSE4.2, which would allow us to delete some old fallbacks. I think the last time Intel or AMD shipped a processor without this was 2010 or 2011. Jumping to AVX is probably almost as uncontroversial, since it looks like that has been universal for nearly as long: https://en.wikipedia.org/wiki/Advanced_Vector_Extensions#CPUs_with_AVX.
Some older lower-performance cloud instance types don't support AVX, it looks like, but I think this is an edge case.
It would be very nice to require AVX2 because that could remove a bunch of conditional code (I think the contributors adding ARM support might want to keep the scalar fallback code though, potentially). It looks like most Intel and AMD processors have supported it since 2013 and 2015 respectively, except some low-end Intel processors: https://en.wikipedia.org/wiki/Advanced_Vector_Extensions#CPUs_with_AVX2.
In the past we've been a bit conservative about this - people do sometimes use very old hardware as a test cluster because it was the hardware that they could get their hands on (e.g. I remember someone complaining several years back that they couldn't get Impala running on some old AMD CPUs that were missing SSSE3 support).
Attachments
Issue Links
- is duplicated by
-
IMPALA-6166 Investigate requiring support for newer instruction sets
-
- Resolved
-