[KYLIN-5274] Improve performance of getSubstitutor

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 5.0-alpha
• Assignee: Zhong Yanghong
• Reporter: Xiaoxiang Yu
• Created: 08/Oct/22 09:18
• Updated: 29/Mar/23 06:46
• Resolved: 29/Mar/23 06:46

Description

Background

The following code are called for each call of KylinConfig#getOptional , which needs to be optimized. In some case, it will improve query performance.

 

tected StrSubstitutor getSubstitutor() {
    // env > properties
    final Map<String, Object> all = Maps.newHashMap(); // create a new map every time
    all.putAll((Map) properties);
    all.putAll(STATIC_SYSTEM_ENV);

    return new StrSubstitutor(all);
} 

 

 

How to fix

1. Not to create a new map each time.

2. Not to use Properties because it extends Hashtable.

Micro Benchmark

Use JMH to show performance(avg time):

 

import org.apache.kylin.common.util.NLocalFileMetadataTestCase;
import org.openjdk.jmh.annotations.Benchmark;
import org.openjdk.jmh.annotations.BenchmarkMode;
import org.openjdk.jmh.annotations.Fork;
import org.openjdk.jmh.annotations.Measurement;
import org.openjdk.jmh.annotations.Mode;
import org.openjdk.jmh.annotations.OutputTimeUnit;
import org.openjdk.jmh.annotations.Scope;
import org.openjdk.jmh.annotations.Setup;
import org.openjdk.jmh.annotations.State;
import org.openjdk.jmh.annotations.Threads;
import org.openjdk.jmh.annotations.Warmup;

import java.util.concurrent.TimeUnit;

@BenchmarkMode(Mode.AverageTime)
@OutputTimeUnit(TimeUnit.MILLISECONDS)
@Warmup(iterations = 1)
@Measurement(iterations = 10, time = 10, timeUnit = TimeUnit.MILLISECONDS)
@Threads(1)
@Fork(value = 1, jvmArgs = {"-Xms2G", "-Xmx2G"})
@State(Scope.Benchmark)
public class KylinConfigBenchmark {
    @Setup
    public void setUp() throws Exception {
        NLocalFileMetadataTestCase case1 = new NLocalFileMetadataTestCase();
        case1.createTestMetadata();
    }

    @Benchmark
    public void getProperty() {
        KylinConfig config = KylinConfig.getInstanceFromEnv();
        for(int i = 0; i<= 1000_000; i++ ){
            config.getJdbcDriverClass();
        }
    }

    public static void main(String[] args) throws Exception {
        org.openjdk.jmh.Main.main(args);
    }
} 

 

 

Before Applied

 

After Applied

 

Conclusion

From JMH, KylinConfig#getOptional will improve about 100 times.

From loading test, query performance will improve about 2-5%(https://github.com/apache/kylin/pull/1991#issuecomment-1262139953), eBay says it will improve 30% in their testing.