Details
-
Improvement
-
Status: Closed
-
Major
-
Resolution: Fixed
-
None
Description
The online job runs for 13 days and finds that there are subtasks but no data processing, as shown in the figure below, this job uses the update time as the partition, uses the bucket index, the number of buckets is 128, and the write parallelism is 128. The key is uniform because the file size of each bucket is not much different from the storage point of view. After positioning, there is a skew in the shuffle algorithm.
Potential disadvantages of algorithmic tilt:
1. The memory usage is uneven, some nodes may have high pressure on the JVM, and TM nodes are prone to timeout
2. It may cause the checkpoint to time out, because the data will be flushed to hdfs during the snapshot state. If the skew is serious, it will cause some nodes to take too long and cause timeout.
current algorithm：
Algorithm flaws:
1. curBucket ∈ [0, numBuckets -1]
2. For the number of globalHash values in the same partition <= numBuckets number, globalHash is divergent, and mod(globalHash, numPartitions) is easy to conflict
3. When numBuckets is relatively large, shuffleIndex is prone to conflicts, resulting in skew
Algorithm optimization:
kb = key % b; kb ∈ [0, b-1] pw = pt % w;
pw ∈ [0, w-1] shuffleIndex = (pw + kb) % w
shuffleIndex ∈ [0, w-1]
In fact, it is to calculate a pw according to the partition first. Pw can be understood as a slot Wn allocated to the partition. Different partitions have a slot.
Then move b slots back on the basis of this slot as the writing of data for this partition
Attachments
Attachments
Issue Links
- links to