HttpComponents HttpClient
  1. HttpComponents HttpClient
  2. HTTPCLIENT-1315

NTLM or digest authentication using a local user on a domain host doesn't work

    Details

      Description

      The default AuthScheme cannot authenticate local users if the host is included in a domain. Authetication with domain users or local users if the host is in a workgroup works fine.

      If using ntlm or digest authentication:

      • Authentication with a domain user works fine
      • Authentication with a local user if the host is in a workgroup works fine
      • Authentication with a local user (e.g. Administrator) if the host is in a domain returns 401 - Unauthorized. (Note: this works with JCIFS implementation)

      To reproduce:

      //using local user returns "401 - Unauthorized" if the host is part of a domain
      NTCredentials creds = new NTCredentials("Administrator", "password", "myworkstation", "HOSTNAME");
      //domain user works fine:
      //NTCredentials creds = new NTCredentials("USERNAME", "password", "myworkstation", "DOMAIN");

      DefaultHttpClient httpclient = new DefaultHttpClient();
      httpclient.getCredentialsProvider().setCredentials(AuthScope.ANY, creds);
      HttpHost target = new HttpHost("xx.xx.xx.xx", 81, "http");
      HttpContext localContext = new BasicHttpContext();
      HttpGet httpget = new HttpGet("/Orchestrator2012/Orchestrator.svc/Jobs");
      List<String> authpref = new ArrayList<String>();
      authpref.add(AuthPolicy.NTLM);
      httpclient.getParams().setParameter(AuthPNames.TARGET_AUTH_PREF, authpref);
      HttpResponse response1 = httpclient.execute(target, httpget, localContext);
      HttpEntity entity1 = response1.getEntity();

      The code works if I use jcifs-1.3.17 to create an NTLMEngine like in the example: http://hc.apache.org/httpcomponents-client-ga/ntlm.html

      1. zero_lm.patch
        0.8 kB
        Karl Wright
      2. disable_flags.patch
        0.8 kB
        Karl Wright
      3. browser login with local user.txt
        4 kB
        Mihai David
      4. httpclient login with local user.txt
        18 kB
        Mihai David
      5. digestlog.txt
        14 kB
        Mihai David

        Activity

        Mihai David created issue -
        Hide
        Oleg Kalnichevski added a comment -

        Are you sure DIGEST is affected as well? If you are, please attach a complete wire / context of the session that exhibits the problem. As far as NTLM is concerned only Karl is qualified to deal with the issue. I hope he will get around to having a look at it.

        Oleg

        Show
        Oleg Kalnichevski added a comment - Are you sure DIGEST is affected as well? If you are, please attach a complete wire / context of the session that exhibits the problem. As far as NTLM is concerned only Karl is qualified to deal with the issue. I hope he will get around to having a look at it. Oleg
        Hide
        Karl Wright added a comment -

        NTLM authentication for when a machine name is provided instead of a domain works quite a bit differently in Windows. It is entirely possible that this case is not adequately handled at this time. However, before we conclude that, let's try to figure out the digest issue - and please also provide some details as to the FORM of the domain field value you are providing, e.g. is it "machine.domain.com" or just "machine", etc. Also, the form of the username matters as will - simple user name, or "user@domain", or "user@machine", etc.?

        Show
        Karl Wright added a comment - NTLM authentication for when a machine name is provided instead of a domain works quite a bit differently in Windows. It is entirely possible that this case is not adequately handled at this time. However, before we conclude that, let's try to figure out the digest issue - and please also provide some details as to the FORM of the domain field value you are providing, e.g. is it "machine.domain.com" or just "machine", etc. Also, the form of the username matters as will - simple user name, or "user@domain", or "user@machine", etc.?
        Hide
        Mihai David added a comment - - edited

        I attached the log for digest authentication using local user. The code used to generate it is below:
        I use only alphanumeric simple names for hostname or domain. No extra ".ext". Username is alphanumeric also - without "@domain".

        UsernamePasswordCredentials creds = new UsernamePasswordCredentials("hostname
        Administrator", "password");
        DefaultHttpClient httpclient = new DefaultHttpClient();
        httpclient.getCredentialsProvider().setCredentials(AuthScope.ANY, creds);
        HttpHost target = new HttpHost("xx.xx.xx.xx", 81, "http");
        HttpContext localContext = new BasicHttpContext();
        HttpGet httpget = new HttpGet("/Orchestrator2012/Orchestrator.svc/Jobs");
        List<String> authpref = new ArrayList<String>();
        authpref.add(AuthPolicy.DIGEST);
        httpclient.getParams().setParameter(AuthPNames.TARGET_AUTH_PREF, authpref);
        HttpResponse response1 = httpclient.execute(target, httpget, localContext);
        HttpEntity entity1 = response1.getEntity();

        Show
        Mihai David added a comment - - edited I attached the log for digest authentication using local user. The code used to generate it is below: I use only alphanumeric simple names for hostname or domain. No extra ".ext". Username is alphanumeric also - without "@domain". UsernamePasswordCredentials creds = new UsernamePasswordCredentials("hostname Administrator", "password"); DefaultHttpClient httpclient = new DefaultHttpClient(); httpclient.getCredentialsProvider().setCredentials(AuthScope.ANY, creds); HttpHost target = new HttpHost("xx.xx.xx.xx", 81, "http"); HttpContext localContext = new BasicHttpContext(); HttpGet httpget = new HttpGet("/Orchestrator2012/Orchestrator.svc/Jobs"); List<String> authpref = new ArrayList<String>(); authpref.add(AuthPolicy.DIGEST); httpclient.getParams().setParameter(AuthPNames.TARGET_AUTH_PREF, authpref); HttpResponse response1 = httpclient.execute(target, httpget, localContext); HttpEntity entity1 = response1.getEntity();
        Mihai David made changes -
        Field Original Value New Value
        Attachment digestlog.txt [ 12567595 ]
        Hide
        Oleg Kalnichevski added a comment -

        Why double slash in 'myhostname
        Administrator'? It has been ages since I touched anything Microsoft but if my memory serves me well it would be 'myhostname\Administrator'.

        As far as I am concerned DIGEST auth exchanges look correct and authentication fails due to invalid credentials.

        Oleg

        Show
        Oleg Kalnichevski added a comment - Why double slash in 'myhostname Administrator'? It has been ages since I touched anything Microsoft but if my memory serves me well it would be 'myhostname\Administrator'. As far as I am concerned DIGEST auth exchanges look correct and authentication fails due to invalid credentials. Oleg
        Hide
        Karl Wright added a comment -

        If the machine is joined to the domain, I think you might need to use the complete machine name rather than just the BIOS name. For example, if machine "A" is in domain "B.com", instead of just "A" you may need to specify "A.B.com".

        Show
        Karl Wright added a comment - If the machine is joined to the domain, I think you might need to use the complete machine name rather than just the BIOS name. For example, if machine "A" is in domain "B.com", instead of just "A" you may need to specify "A.B.com".
        Mihai David made changes -
        Comment [ Backslash has special meaning in a string literal so it needs to be escaped. I also tried "A.B.com" and it doesn't work. The login credentials work in the browser (i.e. Chrome) ]
        Hide
        Karl Wright added a comment -

        I'll try to reproduce what you are seeing here, and see if there are any flag issues I can identify. FWIW, the earliest I can do this would be Sunday evening (2/3). If you were so inclined, you could speed things along by getting Wireshark or tcpdump packet captures of 2 NTLM sessions: the first session being a browser interaction that logs in successfully, where the browser is NOT running on a machine that is on the domain, and the second session being when you try to connect using HttpClient with the same credentials. If you decide to go ahead and get these, please attach both of them to this ticket in a well-labeled manner, so I can inspect them at my leisure.

        Show
        Karl Wright added a comment - I'll try to reproduce what you are seeing here, and see if there are any flag issues I can identify. FWIW, the earliest I can do this would be Sunday evening (2/3). If you were so inclined, you could speed things along by getting Wireshark or tcpdump packet captures of 2 NTLM sessions: the first session being a browser interaction that logs in successfully, where the browser is NOT running on a machine that is on the domain, and the second session being when you try to connect using HttpClient with the same credentials. If you decide to go ahead and get these, please attach both of them to this ticket in a well-labeled manner, so I can inspect them at my leisure.
        Hide
        Mihai David added a comment -

        I was wrong about logging the local user in with digest from the local browser. It only works with ntlm. I'll attach the brwser captured session with wireshark and HttpClient session.

        Show
        Mihai David added a comment - I was wrong about logging the local user in with digest from the local browser. It only works with ntlm. I'll attach the brwser captured session with wireshark and HttpClient session.
        Mihai David made changes -
        Attachment browser login with local user.txt [ 12567628 ]
        Attachment httpclient login with local user.txt [ 12567629 ]
        Mihai David made changes -
        Attachment browser login with local user.txt [ 12567628 ]
        Mihai David made changes -
        Attachment browser login with local user.txt [ 12567630 ]
        Karl Wright made changes -
        Assignee Karl Wright [ kwright@metacarta.com ]
        Mihai David made changes -
        Attachment browser login with local user.txt [ 12567630 ]
        Mihai David made changes -
        Attachment browser login with local user.txt [ 12567631 ]
        Mihai David made changes -
        Attachment httpclient login with local user.txt [ 12567629 ]
        Mihai David made changes -
        Attachment httpclient login with local user.txt [ 12567632 ]
        Hide
        Karl Wright added a comment -

        Thanks for the wire logging, but I actually need the packet captures because I need to use Wireshark's ability to pick apart the NTLM packets. The only thing the wire logging is good for is seeing the headers, and the only thing interesting is this:

        2013/02/01 22:07:44:797 EET [DEBUG] headers - >> Host: 16.77.58.213:81

        Windows domain controllers allow you to set policies describing who can connect to the domain, so often you can't log in unless the host header matches the BIOS name of the machine you are communicating with. But the browser capture has the identical header, so unless you modified one and didn't mention it, that's not the issue here.

        I will set up and Amazon instance and look into this on Sunday.

        Show
        Karl Wright added a comment - Thanks for the wire logging, but I actually need the packet captures because I need to use Wireshark's ability to pick apart the NTLM packets. The only thing the wire logging is good for is seeing the headers, and the only thing interesting is this: 2013/02/01 22:07:44:797 EET [DEBUG] headers - >> Host: 16.77.58.213:81 Windows domain controllers allow you to set policies describing who can connect to the domain, so often you can't log in unless the host header matches the BIOS name of the machine you are communicating with. But the browser capture has the identical header, so unless you modified one and didn't mention it, that's not the issue here. I will set up and Amazon instance and look into this on Sunday.
        Mihai David made changes -
        Attachment browser login with local user.txt [ 12567631 ]
        Mihai David made changes -
        Attachment browser login with local user.txt [ 12567633 ]
        Hide
        Mihai David added a comment -

        I didn't modify the logs.
        I now re-uploaded the browser log because it was saying 'negotiate' instead of ntlm - so i removed 'negotiate' from IIS.

        Thank you for your effort!
        Mihai

        Show
        Mihai David added a comment - I didn't modify the logs. I now re-uploaded the browser log because it was saying 'negotiate' instead of ntlm - so i removed 'negotiate' from IIS. Thank you for your effort! Mihai
        Hide
        Karl Wright added a comment -

        In preparation for reproducing this issue, I hand-unpacked the NTLM packets in the working and non-working examples. There are effectively three differences:

        (1) Flags; two fewer bits set in working Type 3 response (DOMAIN_PRESENT and WORKSTATION_PRESENT are set in the unworking example)
        (2) LM is all zeros in working Type 3 response, vs. calculated value in the non-working one
        (3) Blob in working Type 3 response has values added for field types 6,8,a,9.

        The NTLM implementation from jcifs supposedly works, which does not include blob modifications as described above. This argues for blob additions not being material. Most likely, therefore, is that it's the flag differences that are causing the problem. If I can readily reproduce the problem I can easily try removing these flags to see what happens.

        Show
        Karl Wright added a comment - In preparation for reproducing this issue, I hand-unpacked the NTLM packets in the working and non-working examples. There are effectively three differences: (1) Flags; two fewer bits set in working Type 3 response (DOMAIN_PRESENT and WORKSTATION_PRESENT are set in the unworking example) (2) LM is all zeros in working Type 3 response, vs. calculated value in the non-working one (3) Blob in working Type 3 response has values added for field types 6,8,a,9. The NTLM implementation from jcifs supposedly works, which does not include blob modifications as described above. This argues for blob additions not being material. Most likely, therefore, is that it's the flag differences that are causing the problem. If I can readily reproduce the problem I can easily try removing these flags to see what happens.
        Hide
        Karl Wright added a comment -

        Spent the afternoon setting up Amazon instances, but still haven't been able to join my second instance to the domain established by my first instance, and now I am out of time.

        Instead, I'm going to have to attach a patch proposal to this ticket, which you'll need to apply to httpclient 4.2.3 sources. This patch will suppress the two flags which differ. Please let me know if this works in your environment.

        Show
        Karl Wright added a comment - Spent the afternoon setting up Amazon instances, but still haven't been able to join my second instance to the domain established by my first instance, and now I am out of time. Instead, I'm going to have to attach a patch proposal to this ticket, which you'll need to apply to httpclient 4.2.3 sources. This patch will suppress the two flags which differ. Please let me know if this works in your environment.
        Karl Wright made changes -
        Attachment disable_flags.patch [ 12567786 ]
        Hide
        Karl Wright added a comment -

        Please try out disable_flags.patch. Thanks!

        Show
        Karl Wright added a comment - Please try out disable_flags.patch. Thanks!
        Hide
        Mihai David added a comment -

        The patch doesn't seem to work in my environment..

        Show
        Mihai David added a comment - The patch doesn't seem to work in my environment..
        Hide
        Karl Wright added a comment -

        Ok, trying the second possible fix. Please apply this patch and try it.

        Show
        Karl Wright added a comment - Ok, trying the second possible fix. Please apply this patch and try it.
        Karl Wright made changes -
        Attachment zero_lm.patch [ 12567816 ]
        Hide
        Mihai David added a comment -

        I think the revision you are working with is different then the latest from svn /httpcomponents/httpclient/branches/4.2.x/httpclient/src/main/java/org/apache/http/impl/auth or the one packaged in HttpClient 4.2.3 (GA).
        Can you please tell me what sources should I use?

        Show
        Mihai David added a comment - I think the revision you are working with is different then the latest from svn /httpcomponents/httpclient/branches/4.2.x/httpclient/src/main/java/org/apache/http/impl/auth or the one packaged in HttpClient 4.2.3 (GA). Can you please tell me what sources should I use?
        Hide
        Karl Wright added a comment - - edited

        These diffs are against the httpclient-4.2.x branch. But there should be no difference between that and HttpClient 4.2.3 (GA) at this point.

        Show
        Karl Wright added a comment - - edited These diffs are against the httpclient-4.2.x branch. But there should be no difference between that and HttpClient 4.2.3 (GA) at this point.
        Hide
        Karl Wright added a comment - - edited

        Has there been any further developments here?

        One other point. Since the claim is that jcifs works but httpclient native does not, it occurred to me that we need to know what mode jcifs is operating in. As you may know, you control jcifs by way of a -D switch, which sets its lmcompatibility level emulation. The httpclient code does not need this because it uses the flags sent by the server to determine what mode to use, just like browsers do.

        Please verify that for your jcifs test, you are setting its lmcompatibility level to 5. If you are not setting the switch at all, this too is interesting information in that it means that your jcifs data point is useless. You will need a switch value of 5 to be able to guarantee that jcifs is in fact able to authenticate using the same mechanisms as httpclient is using in this case.

        Show
        Karl Wright added a comment - - edited Has there been any further developments here? One other point. Since the claim is that jcifs works but httpclient native does not, it occurred to me that we need to know what mode jcifs is operating in. As you may know, you control jcifs by way of a -D switch, which sets its lmcompatibility level emulation. The httpclient code does not need this because it uses the flags sent by the server to determine what mode to use, just like browsers do. Please verify that for your jcifs test, you are setting its lmcompatibility level to 5. If you are not setting the switch at all, this too is interesting information in that it means that your jcifs data point is useless. You will need a switch value of 5 to be able to guarantee that jcifs is in fact able to authenticate using the same mechanisms as httpclient is using in this case.
        Hide
        Karl Wright added a comment -

        Another person encountered a problem likely similar to this, so I believe this is now fixed.

        r1447062 (branch)

        Show
        Karl Wright added a comment - Another person encountered a problem likely similar to this, so I believe this is now fixed. r1447062 (branch)
        Hide
        Karl Wright added a comment -

        r1447065 (trunk)

        Show
        Karl Wright added a comment - r1447065 (trunk)
        Karl Wright made changes -
        Resolution Fixed [ 1 ]
        Status Open [ 1 ] Resolved [ 5 ]
        Fix Version/s 4.2.4 [ 12323961 ]
        Fix Version/s 4.3 Alpha2 [ 12323951 ]
        Oleg Kalnichevski made changes -
        Status Resolved [ 5 ] Closed [ 6 ]

          People

          • Assignee:
            Karl Wright
            Reporter:
            Mihai David
          • Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development