Details
-
Improvement
-
Status: Resolved
-
Major
-
Resolution: Fixed
-
None
-
None
-
None
Description
Background
The following code are called for each call of KylinConfig#getOptional , which needs to be optimized. In some case, it will improve query performance.
tected StrSubstitutor getSubstitutor() { // env > properties final Map<String, Object> all = Maps.newHashMap(); // create a new map every time all.putAll((Map) properties); all.putAll(STATIC_SYSTEM_ENV); return new StrSubstitutor(all); }
How to fix
1. Not to create a new map each time.
2. Not to use Properties because it extends Hashtable.
Micro Benchmark
Use JMH to show performance(avg time):
import org.apache.kylin.common.util.NLocalFileMetadataTestCase; import org.openjdk.jmh.annotations.Benchmark; import org.openjdk.jmh.annotations.BenchmarkMode; import org.openjdk.jmh.annotations.Fork; import org.openjdk.jmh.annotations.Measurement; import org.openjdk.jmh.annotations.Mode; import org.openjdk.jmh.annotations.OutputTimeUnit; import org.openjdk.jmh.annotations.Scope; import org.openjdk.jmh.annotations.Setup; import org.openjdk.jmh.annotations.State; import org.openjdk.jmh.annotations.Threads; import org.openjdk.jmh.annotations.Warmup; import java.util.concurrent.TimeUnit; @BenchmarkMode(Mode.AverageTime) @OutputTimeUnit(TimeUnit.MILLISECONDS) @Warmup(iterations = 1) @Measurement(iterations = 10, time = 10, timeUnit = TimeUnit.MILLISECONDS) @Threads(1) @Fork(value = 1, jvmArgs = {"-Xms2G", "-Xmx2G"}) @State(Scope.Benchmark) public class KylinConfigBenchmark { @Setup public void setUp() throws Exception { NLocalFileMetadataTestCase case1 = new NLocalFileMetadataTestCase(); case1.createTestMetadata(); } @Benchmark public void getProperty() { KylinConfig config = KylinConfig.getInstanceFromEnv(); for(int i = 0; i<= 1000_000; i++ ){ config.getJdbcDriverClass(); } } public static void main(String[] args) throws Exception { org.openjdk.jmh.Main.main(args); } }
Before Applied
After Applied
Conclusion
From JMH, KylinConfig#getOptional will improve about 100 times.
From loading test, query performance will improve about 2-5%(https://github.com/apache/kylin/pull/1991#issuecomment-1262139953), eBay says it will improve 30% in their testing.