Uploaded image for project: 'WSS4J'
  1. WSS4J
  2. WSS-238

Switch to wsse:KeyIdentifier instead of wsse:Reference for SAML references within SOAP:body EncryptedData elements.

    Details

    • Type: Improvement
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 1.5.9
    • Fix Version/s: 1.5.10, 1.6
    • Component/s: WSS4J Core
    • Labels:
      None

      Description

      Per CXF bug CXF-2894: http://tinyurl.com/23jx6cx

      Within the soap:body/EncryptedData/SecurityTokenReference element, Glassfish Metro is requiring wsse:KeyIdentifiers instead of wsse:Reference elements when referring to SAML Assertions. Metro appears correct because the SAML Token Profile does not define usage of wsse:Reference for SAML Assertions, only KeyIdentifier or EmbeddedReference. (Section 3.3 of SAML Token Profile of 1 Dec. 2004 pdf lines 250-272.)

      The attached patch will switch SecurityTokenReference from wsse:Reference to wsse:KeyIdentifier when handling SAML Assertions. I've confirmed Metro web service providers will now work with this patch. However, backwards compatibility issues with systems expecting the current wsse:Reference may need to be taken into account.

      WSS4J has another problem with not being able to decrypt SOAP responses that use wsse:KeyIdentifier instead of wsse:Reference for SAML Assertions. Namely, org.apache.ws.security.processor.ReferenceListProcessor's getKeyFromSecurityTokenReference() method will need changing to be able to work with SAML Assertions coming from a wsse:KeyIdentifier element instead of wsse:Reference. I was not immediately successful in getting this second part to work because I could not see how a SAMLTokenProcessor can be initialized from a KeyIdentifier instead of the Reference element within this method.

      1. EncryptedDataPatch.txt
        2 kB
        Glen Mazza
      2. patch238.txt
        4 kB
        Glen Mazza
      3. TestWSSecuritySAMLKeyIdentifier.java
        9 kB
        Glen Mazza
      4. WSS238_CXFClient_ALWAYS.txt
        46 kB
        Glen Mazza
      5. WSS238_MetroClient_ALWAYS.txt
        46 kB
        Glen Mazza
      6. WSS238Results.txt
        50 kB
        Glen Mazza
      7. wss-238-revised.patch
        19 kB
        Colm O hEigeartaigh

        Activity

        Hide
        gmazza Glen Mazza added a comment -

        Patch file.

        Show
        gmazza Glen Mazza added a comment - Patch file.
        Hide
        coheigea Colm O hEigeartaigh added a comment -

        I'm marking this as fix for 1.6 for the backwards compatibility reasons outlined. The patch looks fine, but would you be able to write a test to verify the changes?

        Thanks,

        Colm.

        Show
        coheigea Colm O hEigeartaigh added a comment - I'm marking this as fix for 1.6 for the backwards compatibility reasons outlined. The patch looks fine, but would you be able to write a test to verify the changes? Thanks, Colm.
        Hide
        gmazza Glen Mazza added a comment -

        I can only supply code patches for the 1.5.x series because CXF presently won't compile with WSS4J 1.6. But it may not be a big deal for you to make the change to the 1.6 series once CXF is coded for it.

        When you say a "test" do you mean a JUnit test that will be incorporated into the WSS4J source code or a ZIP file consisting of a Metro STS, web service etc., that one can run to see the results? If the former, can you point to me a test already in WSS4J that I can leverage/learn from while making this JUnit test? It would be helpful for me to see something similar that I can copy from.

        Glen

        Show
        gmazza Glen Mazza added a comment - I can only supply code patches for the 1.5.x series because CXF presently won't compile with WSS4J 1.6. But it may not be a big deal for you to make the change to the 1.6 series once CXF is coded for it. When you say a "test" do you mean a JUnit test that will be incorporated into the WSS4J source code or a ZIP file consisting of a Metro STS, web service etc., that one can run to see the results? If the former, can you point to me a test already in WSS4J that I can leverage/learn from while making this JUnit test? It would be helpful for me to see something similar that I can copy from. Glen
        Hide
        coheigea Colm O hEigeartaigh added a comment -

        > I can only supply code patches for the 1.5.x series because CXF presently won't compile with WSS4J 1.6. But it may not be a big deal for you to make > the change to the 1.6 series once CXF is coded for it.

        Yup, just write the patch for 1.5.x and I can port it.

        > When you say a "test" do you mean a JUnit test that will be incorporated into the WSS4J source code or a ZIP file consisting of a Metro STS, web
        > service etc., that one can run to see the results? If the former, can you point to me a test already in WSS4J that I can leverage/learn from while making
        > this JUnit test? It would be helpful for me to see something similar that I can copy from.

        I meant the former. This just involves writing a test to make sure that the generated XML has the correct field, i.e. KeyIdentifier instead of Reference, and that WSS4J can process the document on the inbound side. Something like this...

        http://svn.apache.org/viewvc/webservices/wss4j/branches/1_5_x-fixes/test/wssec/TestWSSecurityNewST3.java?revision=998471&view=markup

        Thanks,

        Colm.

        Show
        coheigea Colm O hEigeartaigh added a comment - > I can only supply code patches for the 1.5.x series because CXF presently won't compile with WSS4J 1.6. But it may not be a big deal for you to make > the change to the 1.6 series once CXF is coded for it. Yup, just write the patch for 1.5.x and I can port it. > When you say a "test" do you mean a JUnit test that will be incorporated into the WSS4J source code or a ZIP file consisting of a Metro STS, web > service etc., that one can run to see the results? If the former, can you point to me a test already in WSS4J that I can leverage/learn from while making > this JUnit test? It would be helpful for me to see something similar that I can copy from. I meant the former. This just involves writing a test to make sure that the generated XML has the correct field, i.e. KeyIdentifier instead of Reference, and that WSS4J can process the document on the inbound side. Something like this... http://svn.apache.org/viewvc/webservices/wss4j/branches/1_5_x-fixes/test/wssec/TestWSSecurityNewST3.java?revision=998471&view=markup Thanks, Colm.
        Hide
        gmazza Glen Mazza added a comment -

        I'll see what I can do, I just typed up a blog entry on this issue: http://www.jroller.com/gmazza/entry/cxf_stsclient_with_metro_sts

        Show
        gmazza Glen Mazza added a comment - I'll see what I can do, I just typed up a blog entry on this issue: http://www.jroller.com/gmazza/entry/cxf_stsclient_with_metro_sts
        Hide
        gmazza Glen Mazza added a comment -

        New patch; ignore previous one.

        Show
        gmazza Glen Mazza added a comment - New patch; ignore previous one.
        Hide
        gmazza Glen Mazza added a comment -

        Requested test case.

        Show
        gmazza Glen Mazza added a comment - Requested test case.
        Hide
        coheigea Colm O hEigeartaigh added a comment -

        Hi Glen,

        Apologies for not looking at this issue sooner. Please see attached for a revised patch (and test-case) for this issue.

        On the outbound side, the patch gives the ability to create EncryptedData and EncryptedKey tokens where the key used to encrypt the token is referenced using a KeyIdentifier with a custom token value and id. I changed things around a bit from the patch you submitted. WSSecEncryptedKey has two new methods to set both the value and id. These methods already exist in WSSecEncrypt, so it was just a matter of supporting the SAML specific case. The current CXF code should work fine with this patch without any code changes.

        The reason I did not move the setCustomReferenceValue method from WSSecEncrypt to WSSecEncryptedKey as per your patch, was a) for backwards compatibility, and b) To accomodate the use-case, where the EncryptedData element points to the EncryptedKey, and the EncryptedKey points to the SAML assertion. I changed the test-case to illustrate this scenario.

        I also added support to the EncryptedKeyProcessor to support processing an EncryptedKey that points to a SAML Assertion containing an X509Certificate, and the ReferenceListProcessor, to support processing an EncryptedData token that points to a SAML Assertion containing an EncryptedKey.

        Could you apply this patch to 1_5_x-fixes and test with CXF to see whether it's producing an EncryptedData of the right format, which can be processed by Metro? The good news is that this patch is 100% backwards compatible, and so I will fix it for 1.5.10 if you ok the patch.

        Colm.

        Show
        coheigea Colm O hEigeartaigh added a comment - Hi Glen, Apologies for not looking at this issue sooner. Please see attached for a revised patch (and test-case) for this issue. On the outbound side, the patch gives the ability to create EncryptedData and EncryptedKey tokens where the key used to encrypt the token is referenced using a KeyIdentifier with a custom token value and id. I changed things around a bit from the patch you submitted. WSSecEncryptedKey has two new methods to set both the value and id. These methods already exist in WSSecEncrypt, so it was just a matter of supporting the SAML specific case. The current CXF code should work fine with this patch without any code changes. The reason I did not move the setCustomReferenceValue method from WSSecEncrypt to WSSecEncryptedKey as per your patch, was a) for backwards compatibility, and b) To accomodate the use-case, where the EncryptedData element points to the EncryptedKey, and the EncryptedKey points to the SAML assertion. I changed the test-case to illustrate this scenario. I also added support to the EncryptedKeyProcessor to support processing an EncryptedKey that points to a SAML Assertion containing an X509Certificate, and the ReferenceListProcessor, to support processing an EncryptedData token that points to a SAML Assertion containing an EncryptedKey. Could you apply this patch to 1_5_x-fixes and test with CXF to see whether it's producing an EncryptedData of the right format, which can be processed by Metro? The good news is that this patch is 100% backwards compatible, and so I will fix it for 1.5.10 if you ok the patch. Colm.
        Hide
        gmazza Glen Mazza added a comment -

        Results of new patch on Metro STS and Metro web service calls from a CXF client.

        Show
        gmazza Glen Mazza added a comment - Results of new patch on Metro STS and Metro web service calls from a CXF client.
        Hide
        gmazza Glen Mazza added a comment -

        Looks good Colm, thanks! I applied the patch to the WSS4J 1.5.x branch and linked it into CXF 2.3.1-SNAPSHOT, which I then tied to the CXF client. It seems to work fine (the attachment lists the Client > STS, STS> Client, Client -> WSP and WSP -> Client results via Wireshark). I would go ahead and apply this patch.

        I'm not sure that moving setCustomReferenceValue from WSSecEncrypt to WSSecEncryptedKey would raise a backward compatibility issue because the latter is the superclass of the former (going in the other direction, I can more easily see the backward compatibility concern). So long as the second reason you gave holds though, your modification sounds like a good idea.

        After this patch is applied, we will have one last problem, the one I gave in the description field for this bug (at the top of this page): WSS4J can't handle the final SOAP response from the web service provider back to the SOAP client (the SOAP response is at the bottom of the latest attachment.) The blog entry I listed above has the downloadable source for the full CXF client, Metro client, Metro STS and Metro web service (all Maven-based). I can walk you through running them all using IRC if it would help you replicate this bug--it's a reasonably fast process if you know what you're doing.

        This is the stack trace from the SOAP client at the end as WSS4J tries to process the SOAP response:

        [INFO] Nov 18, 2010 1:00:36 AM org.apache.cxf.service.factory.ReflectionServiceFactoryBean buildServiceFromWSDL
        [INFO] INFO: Creating Service

        {http://www.example.org/contract/DoubleIt}

        DoubleItService from WSDL: file:/media/NewDriveExt3/workspace/DoubleItMetroWSTrust/client-cxf/src/main/resources/DoubleItService.wsdl
        [INFO] Nov 18, 2010 1:00:37 AM org.apache.cxf.ws.policy.AssertionBuilderRegistryImpl build
        [INFO] WARNING: No assertion builder for type

        {http://docs.oasis-open.org/ws-sx/ws-securitypolicy/200702}

        RequireInternalReference registered.
        [INFO] Nov 18, 2010 1:00:38 AM org.apache.cxf.ws.security.wss4j.WSS4JInInterceptor handleMessage
        [INFO] WARNING:
        [INFO] org.apache.ws.security.WSSecurityException: Referenced security token could not be retrieved (Reference "#uuid-526391a2-929e-49db-b890-54812517062b")
        [INFO] at org.apache.ws.security.message.token.SecurityTokenReference.getKeyIdentifierTokenElement(SecurityTokenReference.java:200)
        [INFO] at org.apache.ws.security.processor.ReferenceListProcessor.getKeyFromSecurityTokenReference(ReferenceListProcessor.java:356)
        [INFO] at org.apache.ws.security.processor.ReferenceListProcessor.decryptDataRefEmbedded(ReferenceListProcessor.java:162)
        [INFO] at org.apache.ws.security.processor.ReferenceListProcessor.handleReferenceList(ReferenceListProcessor.java:113)
        [INFO] at org.apache.ws.security.processor.ReferenceListProcessor.handleToken(ReferenceListProcessor.java:76)
        [INFO] at org.apache.ws.security.WSSecurityEngine.processSecurityHeader(WSSecurityEngine.java:328)
        [INFO] at org.apache.ws.security.WSSecurityEngine.processSecurityHeader(WSSecurityEngine.java:245)
        [INFO] at org.apache.cxf.ws.security.wss4j.WSS4JInInterceptor.handleMessage(WSS4JInInterceptor.java:215)
        [INFO] at org.apache.cxf.ws.security.wss4j.WSS4JInInterceptor.handleMessage(WSS4JInInterceptor.java:81)
        [INFO] at org.apache.cxf.phase.PhaseInterceptorChain.doIntercept(PhaseInterceptorChain.java:247)
        [INFO] at org.apache.cxf.endpoint.ClientImpl.onMessage(ClientImpl.java:733)
        [INFO] at org.apache.cxf.transport.http.HTTPConduit$WrappedOutputStream.handleResponseInternal(HTTPConduit.java:2304)
        [INFO] at org.apache.cxf.transport.http.HTTPConduit$WrappedOutputStream.handleResponse(HTTPConduit.java:2166)
        [INFO] at org.apache.cxf.transport.http.HTTPConduit$WrappedOutputStream.close(HTTPConduit.java:2019)
        [INFO] at org.apache.cxf.transport.AbstractConduit.close(AbstractConduit.java:56)
        [INFO] at org.apache.cxf.transport.http.HTTPConduit.close(HTTPConduit.java:712)
        [INFO] at org.apache.cxf.interceptor.MessageSenderInterceptor$MessageSenderEndingInterceptor.handleMessage(MessageSenderInterceptor.java:62)
        [INFO] at org.apache.cxf.phase.PhaseInterceptorChain.doIntercept(PhaseInterceptorChain.java:247)
        [INFO] at org.apache.cxf.endpoint.ClientImpl.invoke(ClientImpl.java:516)
        [INFO] at org.apache.cxf.endpoint.ClientImpl.invoke(ClientImpl.java:313)
        [INFO] at org.apache.cxf.endpoint.ClientImpl.invoke(ClientImpl.java:265)
        [INFO] at org.apache.cxf.frontend.ClientProxy.invokeSync(ClientProxy.java:73)
        [INFO] at org.apache.cxf.jaxws.JaxWsClientProxy.invoke(JaxWsClientProxy.java:124)
        [INFO] at $Proxy24.doubleIt(Unknown Source)
        [INFO] at client.WSClient.doubleIt(WSClient.java:17)
        [INFO] at client.WSClient.main(WSClient.java:11)
        [INFO] Nov 18, 2010 1:00:38 AM org.apache.cxf.phase.PhaseInterceptorChain doDefaultLogging
        [INFO] WARNING: Interceptor for

        {http://www.example.org/contract/DoubleIt}

        DoubleItService#

        {http://www.example.org/contract/DoubleIt}

        DoubleIt has thrown exception, unwinding now
        .... stack trace repeated twice ....
        [INFO] Caused by: org.apache.ws.security.WSSecurityException: Referenced security token could not be retrieved (Reference "#uuid-526391a2-929e-49db-b890-54812517062b")
        [INFO] at org.apache.ws.security.message.token.SecurityTokenReference.getKeyIdentifierTokenElement(SecurityTokenReference.java:200)
        [INFO] at org.apache.ws.security.processor.ReferenceListProcessor.getKeyFromSecurityTokenReference(ReferenceListProcessor.java:356)
        [INFO] at org.apache.ws.security.processor.ReferenceListProcessor.decryptDataRefEmbedded(ReferenceListProcessor.java:162)
        [INFO] at org.apache.ws.security.processor.ReferenceListProcessor.handleReferenceList(ReferenceListProcessor.java:113)
        [INFO] at org.apache.ws.security.processor.ReferenceListProcessor.handleToken(ReferenceListProcessor.java:76)
        [INFO] at org.apache.ws.security.WSSecurityEngine.processSecurityHeader(WSSecurityEngine.java:328)
        [INFO] at org.apache.ws.security.WSSecurityEngine.processSecurityHeader(WSSecurityEngine.java:245)
        [INFO] at org.apache.cxf.ws.security.wss4j.WSS4JInInterceptor.handleMessage(WSS4JInInterceptor.java:215)
        [INFO] at org.apache.cxf.ws.security.wss4j.WSS4JInInterceptor.handleMessage(WSS4JInInterceptor.java:81)
        [INFO] at org.apache.cxf.phase.PhaseInterceptorChain.doIntercept(PhaseInterceptorChain.java:247)
        [INFO] at org.apache.cxf.endpoint.ClientImpl.onMessage(ClientImpl.java:733)
        [INFO] at org.apache.cxf.transport.http.HTTPConduit$WrappedOutputStream.handleResponseInternal(HTTPConduit.java:2304)
        [INFO] at org.apache.cxf.transport.http.HTTPConduit$WrappedOutputStream.handleResponse(HTTPConduit.java:2166)
        [INFO] at org.apache.cxf.transport.http.HTTPConduit$WrappedOutputStream.close(HTTPConduit.java:2019)
        [INFO] at org.apache.cxf.transport.AbstractConduit.close(AbstractConduit.java:56)
        [INFO] at org.apache.cxf.transport.http.HTTPConduit.close(HTTPConduit.java:712)
        [INFO] at org.apache.cxf.interceptor.MessageSenderInterceptor$MessageSenderEndingInterceptor.handleMessage(MessageSenderInterceptor.java:62)
        [INFO] at org.apache.cxf.phase.PhaseInterceptorChain.doIntercept(PhaseInterceptorChain.java:247)
        [INFO] at org.apache.cxf.endpoint.ClientImpl.invoke(ClientImpl.java:516)
        [INFO] at org.apache.cxf.endpoint.ClientImpl.invoke(ClientImpl.java:313)
        [INFO] at org.apache.cxf.endpoint.ClientImpl.invoke(ClientImpl.java:265)
        [INFO] at org.apache.cxf.frontend.ClientProxy.invokeSync(ClientProxy.java:73)
        [INFO] at org.apache.cxf.jaxws.JaxWsClientProxy.invoke(JaxWsClientProxy.java:124)
        [INFO] ... 3 more
        [INFO] ------------------------------------------

        This may not be an easy bug to fix--it's beyond my scope of knowledge right now so I don't think I'll be able to supply a patch suggestion to you like I did earlier.

        Show
        gmazza Glen Mazza added a comment - Looks good Colm, thanks! I applied the patch to the WSS4J 1.5.x branch and linked it into CXF 2.3.1-SNAPSHOT, which I then tied to the CXF client. It seems to work fine (the attachment lists the Client > STS, STS > Client, Client -> WSP and WSP -> Client results via Wireshark). I would go ahead and apply this patch. I'm not sure that moving setCustomReferenceValue from WSSecEncrypt to WSSecEncryptedKey would raise a backward compatibility issue because the latter is the superclass of the former (going in the other direction, I can more easily see the backward compatibility concern). So long as the second reason you gave holds though, your modification sounds like a good idea. After this patch is applied, we will have one last problem, the one I gave in the description field for this bug (at the top of this page): WSS4J can't handle the final SOAP response from the web service provider back to the SOAP client (the SOAP response is at the bottom of the latest attachment.) The blog entry I listed above has the downloadable source for the full CXF client, Metro client, Metro STS and Metro web service (all Maven-based). I can walk you through running them all using IRC if it would help you replicate this bug--it's a reasonably fast process if you know what you're doing. This is the stack trace from the SOAP client at the end as WSS4J tries to process the SOAP response: [INFO] Nov 18, 2010 1:00:36 AM org.apache.cxf.service.factory.ReflectionServiceFactoryBean buildServiceFromWSDL [INFO] INFO: Creating Service {http://www.example.org/contract/DoubleIt} DoubleItService from WSDL: file:/media/NewDriveExt3/workspace/DoubleItMetroWSTrust/client-cxf/src/main/resources/DoubleItService.wsdl [INFO] Nov 18, 2010 1:00:37 AM org.apache.cxf.ws.policy.AssertionBuilderRegistryImpl build [INFO] WARNING: No assertion builder for type {http://docs.oasis-open.org/ws-sx/ws-securitypolicy/200702} RequireInternalReference registered. [INFO] Nov 18, 2010 1:00:38 AM org.apache.cxf.ws.security.wss4j.WSS4JInInterceptor handleMessage [INFO] WARNING: [INFO] org.apache.ws.security.WSSecurityException: Referenced security token could not be retrieved (Reference "#uuid-526391a2-929e-49db-b890-54812517062b") [INFO] at org.apache.ws.security.message.token.SecurityTokenReference.getKeyIdentifierTokenElement(SecurityTokenReference.java:200) [INFO] at org.apache.ws.security.processor.ReferenceListProcessor.getKeyFromSecurityTokenReference(ReferenceListProcessor.java:356) [INFO] at org.apache.ws.security.processor.ReferenceListProcessor.decryptDataRefEmbedded(ReferenceListProcessor.java:162) [INFO] at org.apache.ws.security.processor.ReferenceListProcessor.handleReferenceList(ReferenceListProcessor.java:113) [INFO] at org.apache.ws.security.processor.ReferenceListProcessor.handleToken(ReferenceListProcessor.java:76) [INFO] at org.apache.ws.security.WSSecurityEngine.processSecurityHeader(WSSecurityEngine.java:328) [INFO] at org.apache.ws.security.WSSecurityEngine.processSecurityHeader(WSSecurityEngine.java:245) [INFO] at org.apache.cxf.ws.security.wss4j.WSS4JInInterceptor.handleMessage(WSS4JInInterceptor.java:215) [INFO] at org.apache.cxf.ws.security.wss4j.WSS4JInInterceptor.handleMessage(WSS4JInInterceptor.java:81) [INFO] at org.apache.cxf.phase.PhaseInterceptorChain.doIntercept(PhaseInterceptorChain.java:247) [INFO] at org.apache.cxf.endpoint.ClientImpl.onMessage(ClientImpl.java:733) [INFO] at org.apache.cxf.transport.http.HTTPConduit$WrappedOutputStream.handleResponseInternal(HTTPConduit.java:2304) [INFO] at org.apache.cxf.transport.http.HTTPConduit$WrappedOutputStream.handleResponse(HTTPConduit.java:2166) [INFO] at org.apache.cxf.transport.http.HTTPConduit$WrappedOutputStream.close(HTTPConduit.java:2019) [INFO] at org.apache.cxf.transport.AbstractConduit.close(AbstractConduit.java:56) [INFO] at org.apache.cxf.transport.http.HTTPConduit.close(HTTPConduit.java:712) [INFO] at org.apache.cxf.interceptor.MessageSenderInterceptor$MessageSenderEndingInterceptor.handleMessage(MessageSenderInterceptor.java:62) [INFO] at org.apache.cxf.phase.PhaseInterceptorChain.doIntercept(PhaseInterceptorChain.java:247) [INFO] at org.apache.cxf.endpoint.ClientImpl.invoke(ClientImpl.java:516) [INFO] at org.apache.cxf.endpoint.ClientImpl.invoke(ClientImpl.java:313) [INFO] at org.apache.cxf.endpoint.ClientImpl.invoke(ClientImpl.java:265) [INFO] at org.apache.cxf.frontend.ClientProxy.invokeSync(ClientProxy.java:73) [INFO] at org.apache.cxf.jaxws.JaxWsClientProxy.invoke(JaxWsClientProxy.java:124) [INFO] at $Proxy24.doubleIt(Unknown Source) [INFO] at client.WSClient.doubleIt(WSClient.java:17) [INFO] at client.WSClient.main(WSClient.java:11) [INFO] Nov 18, 2010 1:00:38 AM org.apache.cxf.phase.PhaseInterceptorChain doDefaultLogging [INFO] WARNING: Interceptor for {http://www.example.org/contract/DoubleIt} DoubleItService# {http://www.example.org/contract/DoubleIt} DoubleIt has thrown exception, unwinding now .... stack trace repeated twice .... [INFO] Caused by: org.apache.ws.security.WSSecurityException: Referenced security token could not be retrieved (Reference "#uuid-526391a2-929e-49db-b890-54812517062b") [INFO] at org.apache.ws.security.message.token.SecurityTokenReference.getKeyIdentifierTokenElement(SecurityTokenReference.java:200) [INFO] at org.apache.ws.security.processor.ReferenceListProcessor.getKeyFromSecurityTokenReference(ReferenceListProcessor.java:356) [INFO] at org.apache.ws.security.processor.ReferenceListProcessor.decryptDataRefEmbedded(ReferenceListProcessor.java:162) [INFO] at org.apache.ws.security.processor.ReferenceListProcessor.handleReferenceList(ReferenceListProcessor.java:113) [INFO] at org.apache.ws.security.processor.ReferenceListProcessor.handleToken(ReferenceListProcessor.java:76) [INFO] at org.apache.ws.security.WSSecurityEngine.processSecurityHeader(WSSecurityEngine.java:328) [INFO] at org.apache.ws.security.WSSecurityEngine.processSecurityHeader(WSSecurityEngine.java:245) [INFO] at org.apache.cxf.ws.security.wss4j.WSS4JInInterceptor.handleMessage(WSS4JInInterceptor.java:215) [INFO] at org.apache.cxf.ws.security.wss4j.WSS4JInInterceptor.handleMessage(WSS4JInInterceptor.java:81) [INFO] at org.apache.cxf.phase.PhaseInterceptorChain.doIntercept(PhaseInterceptorChain.java:247) [INFO] at org.apache.cxf.endpoint.ClientImpl.onMessage(ClientImpl.java:733) [INFO] at org.apache.cxf.transport.http.HTTPConduit$WrappedOutputStream.handleResponseInternal(HTTPConduit.java:2304) [INFO] at org.apache.cxf.transport.http.HTTPConduit$WrappedOutputStream.handleResponse(HTTPConduit.java:2166) [INFO] at org.apache.cxf.transport.http.HTTPConduit$WrappedOutputStream.close(HTTPConduit.java:2019) [INFO] at org.apache.cxf.transport.AbstractConduit.close(AbstractConduit.java:56) [INFO] at org.apache.cxf.transport.http.HTTPConduit.close(HTTPConduit.java:712) [INFO] at org.apache.cxf.interceptor.MessageSenderInterceptor$MessageSenderEndingInterceptor.handleMessage(MessageSenderInterceptor.java:62) [INFO] at org.apache.cxf.phase.PhaseInterceptorChain.doIntercept(PhaseInterceptorChain.java:247) [INFO] at org.apache.cxf.endpoint.ClientImpl.invoke(ClientImpl.java:516) [INFO] at org.apache.cxf.endpoint.ClientImpl.invoke(ClientImpl.java:313) [INFO] at org.apache.cxf.endpoint.ClientImpl.invoke(ClientImpl.java:265) [INFO] at org.apache.cxf.frontend.ClientProxy.invokeSync(ClientProxy.java:73) [INFO] at org.apache.cxf.jaxws.JaxWsClientProxy.invoke(JaxWsClientProxy.java:124) [INFO] ... 3 more [INFO] ------------------------------------------ This may not be an easy bug to fix--it's beyond my scope of knowledge right now so I don't think I'll be able to supply a patch suggestion to you like I did earlier.
        Hide
        coheigea Colm O hEigeartaigh added a comment - - edited

        Hi Glen,

        Thanks for confirming that the patch fixes the (first) problem.

        The next problem is caused by the fact that the service endpoint is sending back a reference in the EncryptedData to the SAML assertion like this:

        <wsse:KeyIdentifier ValueType="http://docs.oasis-open.org/wss/oasis-wss-saml-token-profile-1.0#SAMLAssertionID">#uuid-526391a2-929e-49db-b890-54812517062b</wsse:KeyIdentifier>

        There are a couple of things to consider here. First off, I think it is a bug that Metro is constructing a KeyIdentifier that uses a relative reference to the SAML Assertion ID, as the SAML Assertion in question is not in the SOAP request. Secondly, even if it were an absolute reference, it is not spec-compliant as the following piece from the SAML 1.1 spec says:

        "When a key identifier is used to reference a V1.1 SAML assertion that is not contained in the same
        message as the key identifier, a <saml:AuthorityBinding> element MUST be contained in the
        <wsse:SecurityTokenReference> element containing the key identifier. "

        The problem WSS4J has, is that it can't find the SAML Assertion that is referenced, as it is not in the message. There are two things to think about here:

        1) Could you try changing the WSDL of the Metro service endpoint, so that the IssuedToken is included "Always" instead of "AlwaysToRecipient"? Is it a requirement that the Assertion should not be sent to the initiator?

        2) If it is a requirement, then I could look at finding a SAML Assertion via a CUSTOM_TOKEN callback. Something similar is done for the SecurityContextToken case.

        Colm.

        Show
        coheigea Colm O hEigeartaigh added a comment - - edited Hi Glen, Thanks for confirming that the patch fixes the (first) problem. The next problem is caused by the fact that the service endpoint is sending back a reference in the EncryptedData to the SAML assertion like this: <wsse:KeyIdentifier ValueType="http://docs.oasis-open.org/wss/oasis-wss-saml-token-profile-1.0#SAMLAssertionID">#uuid-526391a2-929e-49db-b890-54812517062b</wsse:KeyIdentifier> There are a couple of things to consider here. First off, I think it is a bug that Metro is constructing a KeyIdentifier that uses a relative reference to the SAML Assertion ID, as the SAML Assertion in question is not in the SOAP request. Secondly, even if it were an absolute reference, it is not spec-compliant as the following piece from the SAML 1.1 spec says: "When a key identifier is used to reference a V1.1 SAML assertion that is not contained in the same message as the key identifier, a <saml:AuthorityBinding> element MUST be contained in the <wsse:SecurityTokenReference> element containing the key identifier. " The problem WSS4J has, is that it can't find the SAML Assertion that is referenced, as it is not in the message. There are two things to think about here: 1) Could you try changing the WSDL of the Metro service endpoint, so that the IssuedToken is included "Always" instead of "AlwaysToRecipient"? Is it a requirement that the Assertion should not be sent to the initiator? 2) If it is a requirement, then I could look at finding a SAML Assertion via a CUSTOM_TOKEN callback. Something similar is done for the SecurityContextToken case. Colm.
        Hide
        gmazza Glen Mazza added a comment -

        OK, I wouldn't want WSS4J to accommodate any Metro bugs (just as we just fixed WSS4J when it was wrong so Metro wouldn't have to accommodate it.)

        From what you're saying, there's apparently two bugs in Metro's WSIT – I can place these in their JIRA (http://java.net/jira/browse/WSIT) (or would you like to handle this?):
        1.) Incorrect usage of a relative ("#") reference for the KeyIdentifier when the SAML Assertion is not in the SOAP Envelope.

        2.) But something may be wrong with your second assertion about saml:AuthorityBinding needing to be included. Reading from the Assertions and Protocol for the OASIS
        Security Assertion Markup Language (SAML) V1.1 5 OASIS Standard, 2 September 2003:

        2.4.3.2 Element <AuthorityBinding>
        741 The <AuthorityBinding> element MAY be used to indicate to a SAML relying party processing an
        742 AuthenticationStatement that a SAML authority may be available to provide additional information about
        743 the subject of the statement. A single SAML authority may advertise its presence over multiple protocol
        744 bindings, at multiple locations, and as more than one kind of authority by sending multiple elements as
        745 needed.
        746 NOTE: This element is deprecated; use of this element SHOULD be avoided because it is planned to be
        747 removed in the next major version of SAML.

        Lines 746-747 say this element shouldn't be used, so it would strange for it to be elsewhere required. I got this document from http://saml.xml.org/saml-specifications under SAML 1.1. Where do you get your quote about saml:Authoritybinding being required? I couldn't find it in that document.

        Question: What would be the point of the saml:AuthorityBinding anyway – how would that help.

        Let me test the last two points you write and I'll get back to you on that shortly.

        Show
        gmazza Glen Mazza added a comment - OK, I wouldn't want WSS4J to accommodate any Metro bugs (just as we just fixed WSS4J when it was wrong so Metro wouldn't have to accommodate it.) From what you're saying, there's apparently two bugs in Metro's WSIT – I can place these in their JIRA ( http://java.net/jira/browse/WSIT ) (or would you like to handle this?): 1.) Incorrect usage of a relative ("#") reference for the KeyIdentifier when the SAML Assertion is not in the SOAP Envelope. 2.) But something may be wrong with your second assertion about saml:AuthorityBinding needing to be included. Reading from the Assertions and Protocol for the OASIS Security Assertion Markup Language (SAML) V1.1 5 OASIS Standard, 2 September 2003: 2.4.3.2 Element <AuthorityBinding> 741 The <AuthorityBinding> element MAY be used to indicate to a SAML relying party processing an 742 AuthenticationStatement that a SAML authority may be available to provide additional information about 743 the subject of the statement. A single SAML authority may advertise its presence over multiple protocol 744 bindings, at multiple locations, and as more than one kind of authority by sending multiple elements as 745 needed. 746 NOTE: This element is deprecated; use of this element SHOULD be avoided because it is planned to be 747 removed in the next major version of SAML. Lines 746-747 say this element shouldn't be used, so it would strange for it to be elsewhere required. I got this document from http://saml.xml.org/saml-specifications under SAML 1.1. Where do you get your quote about saml:Authoritybinding being required? I couldn't find it in that document. Question: What would be the point of the saml:AuthorityBinding anyway – how would that help. Let me test the last two points you write and I'll get back to you on that shortly.
        Hide
        gmazza Glen Mazza added a comment -

        OK, changing the Metro WSP to Always results in neither the Metro nor the CXF client working. I'm attaching the SOAP calls and responses for Metro and CXF.
        Just to make sure I did nothing screwy in the interim, switching back to AlwaysToRecipient results in the Metro client working again and the CXF failing as it did yesterday.

        The two stacks' error messages are as follows:

        CXF error message:

        [INFO] Nov 18, 2010 12:10:11 PM org.apache.cxf.service.factory.ReflectionServiceFactoryBean buildServiceFromWSDL
        [INFO] INFO: Creating Service

        {http://www.example.org/contract/DoubleIt}

        DoubleItService from WSDL: file:/media/NewDriveExt3/workspace/DoubleItMetroWSTrust/client-cxf/src/main/resources/DoubleItService.wsdl
        [INFO] Nov 18, 2010 12:10:11 PM org.apache.cxf.ws.policy.AssertionBuilderRegistryImpl build
        [INFO] WARNING: No assertion builder for type

        {http://docs.oasis-open.org/ws-sx/ws-securitypolicy/200702}

        RequireInternalReference registered.
        [INFO] Nov 18, 2010 12:10:13 PM org.apache.cxf.ws.security.wss4j.WSS4JInInterceptor handleMessage
        [INFO] WARNING:
        [INFO] org.apache.ws.security.WSSecurityException: The signature or decryption was invalid (<Reference> token could not be retrieved)
        [INFO] at org.apache.ws.security.processor.ReferenceListProcessor.getKeyFromSecurityTokenReference(ReferenceListProcessor.java:393)
        [INFO] at org.apache.ws.security.processor.ReferenceListProcessor.decryptDataRefEmbedded(ReferenceListProcessor.java:162)
        [INFO] at org.apache.ws.security.processor.ReferenceListProcessor.handleReferenceList(ReferenceListProcessor.java:113)
        [INFO] at org.apache.ws.security.processor.ReferenceListProcessor.handleToken(ReferenceListProcessor.java:76)
        [INFO] at org.apache.ws.security.WSSecurityEngine.processSecurityHeader(WSSecurityEngine.java:328)
        [INFO] at org.apache.ws.security.WSSecurityEngine.processSecurityHeader(WSSecurityEngine.java:245)
        [INFO] at org.apache.cxf.ws.security.wss4j.WSS4JInInterceptor.handleMessage(WSS4JInInterceptor.java:215)
        [INFO] at org.apache.cxf.ws.security.wss4j.WSS4JInInterceptor.handleMessage(WSS4JInInterceptor.java:81)
        [INFO] at org.apache.cxf.phase.PhaseInterceptorChain.doIntercept(PhaseInterceptorChain.java:247)
        [INFO] at org.apache.cxf.endpoint.ClientImpl.onMessage(ClientImpl.java:733)
        [INFO] at org.apache.cxf.transport.http.HTTPConduit$WrappedOutputStream.handleResponseInternal(HTTPConduit.java:2304)
        [INFO] at org.apache.cxf.transport.http.HTTPConduit$WrappedOutputStream.handleResponse(HTTPConduit.java:2166)
        [INFO] at org.apache.cxf.transport.http.HTTPConduit$WrappedOutputStream.close(HTTPConduit.java:2019)
        [INFO] at org.apache.cxf.transport.AbstractConduit.close(AbstractConduit.java:56)
        [INFO] at org.apache.cxf.transport.http.HTTPConduit.close(HTTPConduit.java:712)
        [INFO] at org.apache.cxf.interceptor.MessageSenderInterceptor$MessageSenderEndingInterceptor.handleMessage(MessageSenderInterceptor.java:62)
        [INFO] at org.apache.cxf.phase.PhaseInterceptorChain.doIntercept(PhaseInterceptorChain.java:247)
        [INFO] at org.apache.cxf.endpoint.ClientImpl.invoke(ClientImpl.java:516)
        [INFO] at org.apache.cxf.endpoint.ClientImpl.invoke(ClientImpl.java:313)
        [INFO] at org.apache.cxf.endpoint.ClientImpl.invoke(ClientImpl.java:265)
        [INFO] at org.apache.cxf.frontend.ClientProxy.invokeSync(ClientProxy.java:73)
        [INFO] at org.apache.cxf.jaxws.JaxWsClientProxy.invoke(JaxWsClientProxy.java:124)
        [INFO] at $Proxy24.doubleIt(Unknown Source)
        [INFO] at client.WSClient.doubleIt(WSClient.java:17)
        [INFO] at client.WSClient.main(WSClient.java:11)
        [INFO] Nov 18, 2010 12:10:13 PM org.apache.cxf.phase.PhaseInterceptorChain doDefaultLogging
        [INFO] WARNING: Interceptor for

        {http://www.example.org/contract/DoubleIt}

        DoubleItService#

        {http://www.example.org/contract/DoubleIt}

        DoubleIt has thrown exception, unwinding now

        Metro error message:
        [INFO] INFO: WSP5018: Loaded WSIT configuration from file: file:/media/NewDriveExt3/workspace/DoubleItMetroWSTrust/client-metro/target/classes/wsit-client.xml.
        [INFO] Nov 18, 2010 12:15:50 PM com.sun.xml.ws.security.opt.impl.incoming.EncryptedData getCipherInputStream
        [INFO] SEVERE: WSS1926: Key not set for EncryptedData
        [INFO] Exception in thread "main" javax.xml.ws.WebServiceException: com.sun.xml.wss.impl.WssSoapFaultException: WSS1926: Key not set for EncryptedData
        [INFO] at com.sun.xml.wss.jaxws.impl.SecurityClientTube.processResponse(SecurityClientTube.java:348)
        ...
        [INFO] at com.sun.xml.ws.client.Stub.process(Stub.java:319)
        [INFO] at com.sun.xml.ws.client.sei.SEIStub.doProcess(SEIStub.java:157)
        [INFO] at com.sun.xml.ws.client.sei.SyncMethodHandler.invoke(SyncMethodHandler.java:109)
        [INFO] at com.sun.xml.ws.client.sei.SyncMethodHandler.invoke(SyncMethodHandler.java:89)
        [INFO] at com.sun.xml.ws.client.sei.SEIStub.invoke(SEIStub.java:140)
        [INFO] at $Proxy43.doubleIt(Unknown Source)
        [INFO] at client.WSClient.doubleIt(WSClient.java:18)
        [INFO] at client.WSClient.main(WSClient.java:11)
        [INFO] Caused by: com.sun.xml.wss.impl.WssSoapFaultException: WSS1926: Key not set for EncryptedData
        [INFO] at com.sun.xml.ws.security.opt.impl.util.SOAPUtil.newSOAPFaultException(SOAPUtil.java:133)
        [INFO] at com.sun.xml.ws.security.opt.impl.incoming.EncryptedData.getCipherInputStream(EncryptedData.java:216)
        [INFO] at com.sun.xml.ws.security.opt.impl.incoming.EncryptedData.getDecryptedData(EncryptedData.java:236)
        [INFO] ... 12 more

        I'll mention to the Metro JIRA the items we discussed earlier.

        Show
        gmazza Glen Mazza added a comment - OK, changing the Metro WSP to Always results in neither the Metro nor the CXF client working. I'm attaching the SOAP calls and responses for Metro and CXF. Just to make sure I did nothing screwy in the interim, switching back to AlwaysToRecipient results in the Metro client working again and the CXF failing as it did yesterday. The two stacks' error messages are as follows: CXF error message: [INFO] Nov 18, 2010 12:10:11 PM org.apache.cxf.service.factory.ReflectionServiceFactoryBean buildServiceFromWSDL [INFO] INFO: Creating Service {http://www.example.org/contract/DoubleIt} DoubleItService from WSDL: file:/media/NewDriveExt3/workspace/DoubleItMetroWSTrust/client-cxf/src/main/resources/DoubleItService.wsdl [INFO] Nov 18, 2010 12:10:11 PM org.apache.cxf.ws.policy.AssertionBuilderRegistryImpl build [INFO] WARNING: No assertion builder for type {http://docs.oasis-open.org/ws-sx/ws-securitypolicy/200702} RequireInternalReference registered. [INFO] Nov 18, 2010 12:10:13 PM org.apache.cxf.ws.security.wss4j.WSS4JInInterceptor handleMessage [INFO] WARNING: [INFO] org.apache.ws.security.WSSecurityException: The signature or decryption was invalid (<Reference> token could not be retrieved) [INFO] at org.apache.ws.security.processor.ReferenceListProcessor.getKeyFromSecurityTokenReference(ReferenceListProcessor.java:393) [INFO] at org.apache.ws.security.processor.ReferenceListProcessor.decryptDataRefEmbedded(ReferenceListProcessor.java:162) [INFO] at org.apache.ws.security.processor.ReferenceListProcessor.handleReferenceList(ReferenceListProcessor.java:113) [INFO] at org.apache.ws.security.processor.ReferenceListProcessor.handleToken(ReferenceListProcessor.java:76) [INFO] at org.apache.ws.security.WSSecurityEngine.processSecurityHeader(WSSecurityEngine.java:328) [INFO] at org.apache.ws.security.WSSecurityEngine.processSecurityHeader(WSSecurityEngine.java:245) [INFO] at org.apache.cxf.ws.security.wss4j.WSS4JInInterceptor.handleMessage(WSS4JInInterceptor.java:215) [INFO] at org.apache.cxf.ws.security.wss4j.WSS4JInInterceptor.handleMessage(WSS4JInInterceptor.java:81) [INFO] at org.apache.cxf.phase.PhaseInterceptorChain.doIntercept(PhaseInterceptorChain.java:247) [INFO] at org.apache.cxf.endpoint.ClientImpl.onMessage(ClientImpl.java:733) [INFO] at org.apache.cxf.transport.http.HTTPConduit$WrappedOutputStream.handleResponseInternal(HTTPConduit.java:2304) [INFO] at org.apache.cxf.transport.http.HTTPConduit$WrappedOutputStream.handleResponse(HTTPConduit.java:2166) [INFO] at org.apache.cxf.transport.http.HTTPConduit$WrappedOutputStream.close(HTTPConduit.java:2019) [INFO] at org.apache.cxf.transport.AbstractConduit.close(AbstractConduit.java:56) [INFO] at org.apache.cxf.transport.http.HTTPConduit.close(HTTPConduit.java:712) [INFO] at org.apache.cxf.interceptor.MessageSenderInterceptor$MessageSenderEndingInterceptor.handleMessage(MessageSenderInterceptor.java:62) [INFO] at org.apache.cxf.phase.PhaseInterceptorChain.doIntercept(PhaseInterceptorChain.java:247) [INFO] at org.apache.cxf.endpoint.ClientImpl.invoke(ClientImpl.java:516) [INFO] at org.apache.cxf.endpoint.ClientImpl.invoke(ClientImpl.java:313) [INFO] at org.apache.cxf.endpoint.ClientImpl.invoke(ClientImpl.java:265) [INFO] at org.apache.cxf.frontend.ClientProxy.invokeSync(ClientProxy.java:73) [INFO] at org.apache.cxf.jaxws.JaxWsClientProxy.invoke(JaxWsClientProxy.java:124) [INFO] at $Proxy24.doubleIt(Unknown Source) [INFO] at client.WSClient.doubleIt(WSClient.java:17) [INFO] at client.WSClient.main(WSClient.java:11) [INFO] Nov 18, 2010 12:10:13 PM org.apache.cxf.phase.PhaseInterceptorChain doDefaultLogging [INFO] WARNING: Interceptor for {http://www.example.org/contract/DoubleIt} DoubleItService# {http://www.example.org/contract/DoubleIt} DoubleIt has thrown exception, unwinding now Metro error message: [INFO] INFO: WSP5018: Loaded WSIT configuration from file: file:/media/NewDriveExt3/workspace/DoubleItMetroWSTrust/client-metro/target/classes/wsit-client.xml . [INFO] Nov 18, 2010 12:15:50 PM com.sun.xml.ws.security.opt.impl.incoming.EncryptedData getCipherInputStream [INFO] SEVERE: WSS1926: Key not set for EncryptedData [INFO] Exception in thread "main" javax.xml.ws.WebServiceException: com.sun.xml.wss.impl.WssSoapFaultException: WSS1926: Key not set for EncryptedData [INFO] at com.sun.xml.wss.jaxws.impl.SecurityClientTube.processResponse(SecurityClientTube.java:348) ... [INFO] at com.sun.xml.ws.client.Stub.process(Stub.java:319) [INFO] at com.sun.xml.ws.client.sei.SEIStub.doProcess(SEIStub.java:157) [INFO] at com.sun.xml.ws.client.sei.SyncMethodHandler.invoke(SyncMethodHandler.java:109) [INFO] at com.sun.xml.ws.client.sei.SyncMethodHandler.invoke(SyncMethodHandler.java:89) [INFO] at com.sun.xml.ws.client.sei.SEIStub.invoke(SEIStub.java:140) [INFO] at $Proxy43.doubleIt(Unknown Source) [INFO] at client.WSClient.doubleIt(WSClient.java:18) [INFO] at client.WSClient.main(WSClient.java:11) [INFO] Caused by: com.sun.xml.wss.impl.WssSoapFaultException: WSS1926: Key not set for EncryptedData [INFO] at com.sun.xml.ws.security.opt.impl.util.SOAPUtil.newSOAPFaultException(SOAPUtil.java:133) [INFO] at com.sun.xml.ws.security.opt.impl.incoming.EncryptedData.getCipherInputStream(EncryptedData.java:216) [INFO] at com.sun.xml.ws.security.opt.impl.incoming.EncryptedData.getDecryptedData(EncryptedData.java:236) [INFO] ... 12 more I'll mention to the Metro JIRA the items we discussed earlier.
        Hide
        gmazza Glen Mazza added a comment -

        Potential bugs given to Metro team: http://java.net/jira/browse/WSIT-1490

        I think the WSS4J patch should still be applied even with this issue pending, as it's unrelated to it and needs to be done anyway.

        Show
        gmazza Glen Mazza added a comment - Potential bugs given to Metro team: http://java.net/jira/browse/WSIT-1490 I think the WSS4J patch should still be applied even with this issue pending, as it's unrelated to it and needs to be done anyway.
        Hide
        coheigea Colm O hEigeartaigh added a comment -

        Thanks for logging that issue with WSIT Glen. It looks like the error you report above is another bug in Metro. The response to the client from the Metro web service contains a reference to the SAML Assertion like this:

        <wsse:SecurityTokenReference xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:nil="true" />

        This is a clear bug, as the policy specifies to "Always" include the Token.

        Would it be possible to try to duplicate the Metro endpoint using CXF? I'm curious to know if a) the CXF endpoint correctly includes the SAML Assertion back to the client if "Always" is specified in the policy, and b) if the Metro client can correctly parse the response the CXF endpoint sends back for the "AlwaysToRecipient" case?

        I will apply the WSS4J patch as it fixes the main problem here anyway.

        Colm.

        Show
        coheigea Colm O hEigeartaigh added a comment - Thanks for logging that issue with WSIT Glen. It looks like the error you report above is another bug in Metro. The response to the client from the Metro web service contains a reference to the SAML Assertion like this: <wsse:SecurityTokenReference xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:nil="true" /> This is a clear bug, as the policy specifies to "Always" include the Token. Would it be possible to try to duplicate the Metro endpoint using CXF? I'm curious to know if a) the CXF endpoint correctly includes the SAML Assertion back to the client if "Always" is specified in the policy, and b) if the Metro client can correctly parse the response the CXF endpoint sends back for the "AlwaysToRecipient" case? I will apply the WSS4J patch as it fixes the main problem here anyway. Colm.
        Hide
        gmazza Glen Mazza added a comment -

        Better formatted version of Wednesday's results.

        Show
        gmazza Glen Mazza added a comment - Better formatted version of Wednesday's results.

          People

          • Assignee:
            coheigea Colm O hEigeartaigh
            Reporter:
            gmazza Glen Mazza
          • Votes:
            0 Vote for this issue
            Watchers:
            0 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development