[HIVE-23133] Numeric operations can have different result across hardware archs

• Type: Sub-task
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 4.0.0-alpha-1
• Assignee: Zhenyu Zheng
• Reporter: Zhenyu Zheng
• Created: 03/Apr/20 06:49
• Updated: 17/Mar/23 09:37
• Resolved: 14/May/20 08:56

Description

Use java.lang.StrictMath rather than java.lang.Math for compatibility in the following UDFs:

UDFCos.java

UDFExp.java

UDFLn.java

UDFLog.java

UDFLog2.java

UDFLog10.java

 

Currently, we have set up an ARM CI to test out how Hive works on ARM platform:

https://builds.apache.org/view/H-L/view/Hive/job/Hive-linux-ARM-trunk/

Among the failures, we have observed that some numeric operations can have different result across hardware archs, such as:

https://builds.apache.org/view/H-L/view/Hive/job/Hive-linux-ARM-trunk/25/testReport/org.apache.hadoop.hive.cli/TestCliDriver/testCliDriver_vector_decimal_udf2_/

https://builds.apache.org/view/H-L/view/Hive/job/Hive-linux-ARM-trunk/25/testReport/org.apache.hadoop.hive.cli/TestSparkCliDriver/testCliDriver_subquery_select_/

https://builds.apache.org/view/H-L/view/Hive/job/Hive-linux-ARM-trunk/25/testReport/org.apache.hadoop.hive.cli/TestSparkCliDriver/testCliDriver_vectorized_math_funcs_/

we can see that the calculation results of log, exp, cos, toRadians etc is slitly different than the .out file results that we are

comparing(they are tested and wrote on X86 machines), this is because of we use Math Library for these kind of calculations.

and according to the illustration:

Unlike some of the numeric methods of class StrictMath, all implementations of the equivalent functions of class Math are not
defined to return the bit-for-bit same results. This relaxation permits better-performing implementations where strict reproducibility
is not required.

By default many of the Math methods simply call the equivalent method in StrictMath for their implementation.
Code generators are encouraged to use platform-specific native libraries or microprocessor instructions, where available,
to provide higher-performance implementations of Math methods.

so the result will have difference across hardware archs.

On the other hand, JAVA provided another library StrictMath that will not have this kind of problem as according to its' reference:

To help ensure portability of Java programs, the definitions of some of the numeric functions in this package require that they produce
the same results as certain published algorithms.

So in order to fix the above mentioned problem, we have to consider switch to use StrictMath instead of Math.
 

 

 

Issue Links

• relates to: HIVE-27145 Use StrictMath for remaining Math functions as followup of HIVE-23133