[SPARK-41246] When RddId exceeds Integer.Max_VALUE, RddId becomes negative

• Type: Bug
• Status: In Progress
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 3.3.1
• Component/s: Spark Core
• Assignee: Unassigned
• Reporter: Qiang Yang
• Created: 24/Nov/22 04:23
• Updated: 11/Dec/22 13:31

Description

The incrementAndGet method of AtomicInteger keeps increasing. When AtomicInteger increases to 2147483647 (Integer. MAX_VALUE) and then adds one, the value of AtomicInteger will become negative - 2147483648 (Integer. MIN_VALUE).

This problem occurs in long running tasks, such as stream tasks.

When BlockManager generates BlockId, BlockId only supports positive rddid, so BlockId generation fails

object BlockId {
val RDD = "rdd_([0-9])_([0-9])".r

Issue Links

• links to: [Github] Pull Request #38781 (yorksity)