[PHOENIX-3778] Local Index - Writes get exponentially slower as batch size increases

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Duplicate
• Affects Version/s: 4.10.0
• Assignee: Unassigned
• Reporter: Mujtaba Chohan
• Created: 10/Apr/17 22:04
• Updated: 24/Apr/17 18:01
• Resolved: 24/Apr/17 18:01

Description

CREATE TABLE IF NOT EXISTS T (
    PK_PART_A CHAR(15) NOT NULL,
    PK_PART_B CHAR(3) NOT NULL,
    PK_PART_C CHAR(15) NOT NULL,
    PK_PART_DATE DATE NOT NULL,
    PK_PART_LAST CHAR(15) NOT NULL,
    VAL_ID CHAR(15), 
    CONSTRAINT PK PRIMARY KEY 
    (
        PK_PART_A, 
        PK_PART_B,
        PK_PART_C,
        PK_PART_DATE,
        PK_PART_LAST
    )
) IMMUTABLE_ROWS=true

CREATE LOCAL INDEX IF NOT EXISTS T_IDX ON T (PK_PART_DATE)

With HBase 0.98.23 and head of 4.x. writes to local index gets exponentially slower as batch size increases compared to global index when data is upserted to base table with index.

Following is the average time per batch comparing local vs global index. Tested on cluster with 12GB heap per region server and also on single node machine with 4GB heap. Memory usage was within limits with minimal GC activity on client and server. samarthjain and I looked for anything obvious for CPU in jvisualvm but nothing popped up.

Batch Size	Local Index time in ms per batch	Global Index time in ms per batch
10	15	11
100	175	15
500	3500	50
1000	150000	95

rajeshbabu