[SPARK-34844] JDBCRelation columnPartition function includes the first stride in the lower partition

• Type: Improvement
• Status: Resolved
• Priority: Minor
• Resolution: Not A Problem
• Affects Version/s: 3.0.0
• Component/s: SQL
• Assignee: Unassigned
• Reporter: Jason Yarbrough
• Created: 23/Mar/21 20:52
• Updated: 25/Mar/21 00:19
• Resolved: 25/Mar/21 00:18

Description

Currently, columnPartition in JDBCRelation contains logic that adds the first stride into the lower partition. Because of this, the lower bound isn't used as the ceiling for the lower partition.

For example, say we have data 0-10, 10 partitions, and the lowerBound is set to 1. The lower/first partition should contain anything < 1. However, in the current implementation, it would include anything < 2.

A possible easy fix would be changing the following code on line 132:

currentValue += stride

To:

if (i != 0) currentValue += stride

Or include currentValue += stride within the if statement on line 131... although this creates a pretty bad looking side-effect.

Issue Links

• is related to: SPARK-34843 JDBCRelation columnPartition function improperly determines stride size. Upper bound is skewed due to stride alignment.