Major Description
We observed a performance problem in production. As could be seen by thread dumps, many threads were waiting for the same lock, an instance of com.oracle.security.ucrypto.UcryptoProvider.
The base class java.security.Provider.java extends java.util.Properties and many methos are synchronized(). So calling lots of operations on the provider results in a bottleneck.
The calls stacks waiting for the lock were the following (line numbers for Wicket 6.0.22, but from looking at the code I expect the same problems should happen even for master).
First the frames close to the bottom of the stack, they are always the same:
at org.apache.wicket.core.request.mapper.CryptoMapper.encryptEntireUrl(CryptoMapper.java:313)
at org.apache.wicket.core.request.mapper.CryptoMapper.encryptUrl(CryptoMapper.java:295)
at com.mycorp.application.myapp.security.AppCryptoMapper.mapHandler(AppCryptoMapper.java:62)
at org.apache.wicket.request.cycle.RequestCycle.mapUrlFor(RequestCycle.java:429)
at org.apache.wicket.request.cycle.RequestCycle.urlFor(RequestCycle.java:529)
at org.apache.wicket.Component.urlFor(Component.java:3374)
at org.apache.wicket.markup.html.link.Link.getURL(Link.java:327)
Now the four most frequent detail stacks which sit above the common stack:
at java.util.Hashtable.get(Hashtable.java:433)
- waiting on (a com.oracle.security.ucrypto.UcryptoProvider@0xHEXADDR)
at java.util.Properties.getProperty(Properties.java:951)
at java.security.Provider.getProperty(Provider.java:390)
at java.security.Security.getProviderProperty(Security.java:262)
at java.security.Security.isCriterionSatisfied(Security.java:914)
at java.security.Security.getProvidersNotUsingCache(Security.java:889)
at java.security.Security.getAllQualifyingCandidates(Security.java:877)
at java.security.Security.getProviders(Security.java:625)
at java.security.Security.getProviders(Security.java:552)
at org.apache.wicket.util.crypt.SunJceCrypt.<init>(SunJceCrypt.java:81)
at org.apache.wicket.core.util.crypt.KeyInSessionSunJceCryptFactory.createCrypt(KeyInSessionSunJceCryptFactory.java:92)
at org.apache.wicket.core.util.crypt.KeyInSessionSunJceCryptFactory.newCrypt(KeyInSessionSunJceCryptFactory.java:82)
at java.security.Provider.getService(Provider.java:680)
- waiting on (a com.oracle.security.ucrypto.UcryptoProvider@0xHEXADDR)
at sun.security.jca.ProviderList.getService(ProviderList.java:331)
at sun.security.jca.GetInstance.getInstance(GetInstance.java:157)
at java.security.Security.getImpl(Security.java:695)
at java.security.MessageDigest.getInstance(MessageDigest.java:167)
at com.sun.crypto.provider.PBECipherCore.<init>(PBECipherCore.java:75)
at com.sun.crypto.provider.PBEWithMD5AndDESCipher.<init>(PBEWithMD5AndDESCipher.java:61)
at sun.reflect.GeneratedConstructorAccessor108.newInstance(Unknown Source)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:526)
at java.security.Provider$Service.newInstance(Provider.java:1240)
at javax.crypto.Cipher.chooseProvider(Cipher.java:850)
at javax.crypto.Cipher.init(Cipher.java:1374)
at javax.crypto.Cipher.init(Cipher.java:1308)
at org.apache.wicket.util.crypt.SunJceCrypt.createCipher(SunJceCrypt.java:133)
at org.apache.wicket.util.crypt.SunJceCrypt.crypt(SunJceCrypt.java:114)
at org.apache.wicket.util.crypt.AbstractCrypt.encryptStringToByteArray(AbstractCrypt.java:172)
at org.apache.wicket.util.crypt.AbstractCrypt.encryptUrlSafe(AbstractCrypt.java:87)
at org.apache.wicket.core.request.mapper.CryptoMapper.encryptEntireUrl(CryptoMapper.java:313)
at java.security.Provider.getService(Provider.java:680)
- waiting on (a com.oracle.security.ucrypto.UcryptoProvider@0xHEXADDR)
at sun.security.jca.ProviderList$ServiceList.tryGet(ProviderList.java:443)
at sun.security.jca.ProviderList$ServiceList.access$200(ProviderList.java:375)
at sun.security.jca.ProviderList$ServiceList$1.hasNext(ProviderList.java:485)
at javax.crypto.Cipher.getInstance(Cipher.java:502)
at org.apache.wicket.util.crypt.SunJceCrypt.createCipher(SunJceCrypt.java:132)
at org.apache.wicket.util.crypt.SunJceCrypt.crypt(SunJceCrypt.java:114)
at org.apache.wicket.util.crypt.AbstractCrypt.encryptStringToByteArray(AbstractCrypt.java:172)
at org.apache.wicket.util.crypt.AbstractCrypt.encryptUrlSafe(AbstractCrypt.java:87)
at org.apache.wicket.core.request.mapper.CryptoMapper.encryptEntireUrl(CryptoMapper.java:313)
at java.security.Provider.getService(Provider.java:680)
- waiting on (a com.oracle.security.ucrypto.UcryptoProvider@0xHEXADDR)
at sun.security.jca.ProviderList$ServiceList.tryGet(ProviderList.java:436)
at sun.security.jca.ProviderList$ServiceList.access$200(ProviderList.java:375)
at sun.security.jca.ProviderList$ServiceList$1.hasNext(ProviderList.java:485)
at javax.crypto.SecretKeyFactory.nextSpi(SecretKeyFactory.java:292)
at javax.crypto.SecretKeyFactory.<init>(SecretKeyFactory.java:120)
at javax.crypto.SecretKeyFactory.getInstance(SecretKeyFactory.java:159)
at org.apache.wicket.util.crypt.SunJceCrypt.generateSecretKey(SunJceCrypt.java:152)
at org.apache.wicket.util.crypt.SunJceCrypt.crypt(SunJceCrypt.java:112)
at org.apache.wicket.util.crypt.AbstractCrypt.encryptStringToByteArray(AbstractCrypt.java:172)
at org.apache.wicket.util.crypt.AbstractCrypt.encryptUrlSafe(AbstractCrypt.java:87)
at org.apache.wicket.core.request.mapper.CryptoMapper.encryptEntireUrl(CryptoMapper.java:313)
There's a couple of possible improvements/solutions:
In SunJceCrypt.java when creating a new instance there's a check
Security.getProviders("Cipher." + cryptMethod).length > 0
The result of the check could be cached in a static ConcurrentSkipListSet<String>. Note that the method Security.getProviders() is in one of the synchronized stacks above.
The SecretKey derived from the string types session key is created in SunJceCrypt using method generateSecretKey() every time a crypt() call is executed. Since the SecretKey only depends on the cryptMethod and the string key, and furthermore it is serializable and should be thread-safe (read-only) it could be generated once and cached in the session. Creation of the SecretKey is another of the synchronized stacks above.
Furthemore one could cache the cipher per session, more precisely one instance created for encrypt mode and one for decrypt mode. Unfortunately the MetaDataKey based way of putting stuff into the session only allows serializable objects and Cipher is not serializable. The use of such cached ciphers per session would need to be synchronized on the cipher object, but that looks much better than stressing the global provider lock.
Regards,
Rainer