Commons Codec
  1. Commons Codec
  2. CODEC-133

Add classes for MD5/SHA1/SHA-512-based Unix crypt(3) hash variants.

    Details

      Description

      The Linux libc6 crypt(3) function, which is used to generate e.g. the password hashes in /etc/shadow, is available in nearly all other programming languages (Perl, PHP, Python, C, C++, ...) and databases like MySQL and offers MD5/SHA1/SHA-512 based algorithms that were improved by adding a salt and several iterations to make rainbow table attacks harder. Thus they are widely used to store user passwords.

      Java, though, has due it's platform independence, no direct access to the libc functions and still lacks an proper port of the crypt(3) function.

      I already filed a wishlist bug (CODEC-104) for the traditional 56-bit DES based crypt(3) method but would also like to see the much stronger algorithms.
      There are other bug reports like DIRSTUDIO-738 that demand those crypt variants for some specific applications so there it would benefit other Apache projects as well.

      Java ports of most of the specific crypt variants are already existing, but they would have to be cleaned up, properly tested and license checked:
      ftp://ftp.arlut.utexas.edu/pub/java_hashes/

      I would be willing to help here by cleaning the source code and writing unit tests etc. but I'd like to generally know if you are interested and if there's someone who can do a code review (it's security relevant after all and I'm no crypto guy)

      bye,

      christian

      1. crypt3-with-utexas-licence.diff
        140 kB
        Christian Hammers
      2. commons-codec-crypt3.diff
        106 kB
        Christian Hammers

        Activity

        Hide
        Gary Gregory added a comment -

        Released in 1.7 today.

        Show
        Gary Gregory added a comment - Released in 1.7 today.
        Hide
        Gary Gregory added a comment -

        In SVN.

        Show
        Gary Gregory added a comment - In SVN.
        Hide
        Gary Gregory added a comment -

        OK, that sounds good.

        Show
        Gary Gregory added a comment - OK, that sounds good.
        Hide
        Christian Hammers added a comment -

        Ok, remove my license.

        What about replacing Pouls license with the sentence:

        Based on the public domain ("beer-ware") C implementation from Poul-Henning Kamp which was found at:
        <pre>
        Source: $FreeBSD: src/lib/libcrypt/crypt-md5.c,v 1.1 1999/01/21 13:50:09 brandon Exp $
        http://www.freebsd.org/cgi/cvsweb.cgi/src/lib/libcrypt/crypt-md5.c?rev=1.1;content-type=text%2Fplain
        </pre>

        I just like to keep a reference to the original source code in case someone want to verify the code and also mentioning which licesence Poul used so that nobody worries that my conversion has any
        licesene issues.

        Show
        Christian Hammers added a comment - Ok, remove my license. What about replacing Pouls license with the sentence: Based on the public domain ("beer-ware") C implementation from Poul-Henning Kamp which was found at: <pre> Source: $FreeBSD: src/lib/libcrypt/crypt-md5.c,v 1.1 1999/01/21 13:50:09 brandon Exp $ http://www.freebsd.org/cgi/cvsweb.cgi/src/lib/libcrypt/crypt-md5.c?rev=1.1;content-type=text%2Fplain </pre> I just like to keep a reference to the original source code in case someone want to verify the code and also mentioning which licesence Poul used so that nobody worries that my conversion has any licesene issues.
        Hide
        Gary Gregory added a comment -

        Here is my POV: before we go ask @legal if this is OK and so on, let's make it as simple as possible and maybe we do not even need to go there.

        Let's start with the simple case: your (Christian) license. My POV is that you granted the ASF license to your port. You contribution should be recorded in the POM as a "Contributor". I'm not sure how to say it correctly (legally or politically but it seems that your public domain license cannot/should not/is irrelevant along side the ASF license. Granting your code to the ASF with the patch having selected the "grant" checkbox takes are of your IP. You do not get to say "I want my license in the source too". Imagine the mess if every patch came with it's own license in the patch text. It would be impossible to deal with. There is one license, the ASF license. Am I making sense?

        The other case is the "beer me" license, which is just as problematic IMO. The Javadoc can talk about the code history, but the only license should be the ASF license IMO.

        Thoughts? From ANYONE?

        Show
        Gary Gregory added a comment - Here is my POV: before we go ask @legal if this is OK and so on, let's make it as simple as possible and maybe we do not even need to go there. Let's start with the simple case: your (Christian) license. My POV is that you granted the ASF license to your port. You contribution should be recorded in the POM as a "Contributor". I'm not sure how to say it correctly (legally or politically but it seems that your public domain license cannot/should not/is irrelevant along side the ASF license. Granting your code to the ASF with the patch having selected the "grant" checkbox takes are of your IP. You do not get to say "I want my license in the source too". Imagine the mess if every patch came with it's own license in the patch text. It would be impossible to deal with. There is one license, the ASF license. Am I making sense? The other case is the "beer me" license, which is just as problematic IMO. The Javadoc can talk about the code history, but the only license should be the ASF license IMO. Thoughts? From ANYONE?
        Hide
        Christian Hammers added a comment -

        Hello

        Both licenses are essentially PUBLIC DOMAIN license statements (although Pouls is funnier than mine).

        Having them in the Apache project next to the Apache License means "the original authors did not care what happend to his intellectual property so we, the Apache Group, took it and re-licensed it under a much stricter Apache License".

        You should definetly retain the FreeBSD cvsweb URL and better the quote from Poul as the FreeBSD project did the same as you are doing now: they took the PUBLIC DOMAIN code and re-licensed it some years later to the FreeBSD license which means that we had problems if I'd took a recend version of crypt-md5.c. But as we can prove that it was once released under PUBLIC DOMAIN in 1999, it's OK to use that old version as base for our Java conversion.

        I'd be glad if my PUBLIC DOMAIN statement could be retained as well as it basically means that everybody
        who finds this Java snippet can use it for whatever he wants. Further modifications will then be subject
        to the Apache License if you put it on top of the file (-> same situation as the mentioned FreeBSD code).
        But if you see problems with that, you may remove my license as the Apache License is reasonably free, too.

        bye,

        christian

        Show
        Christian Hammers added a comment - Hello Both licenses are essentially PUBLIC DOMAIN license statements (although Pouls is funnier than mine). Having them in the Apache project next to the Apache License means "the original authors did not care what happend to his intellectual property so we, the Apache Group, took it and re-licensed it under a much stricter Apache License". You should definetly retain the FreeBSD cvsweb URL and better the quote from Poul as the FreeBSD project did the same as you are doing now: they took the PUBLIC DOMAIN code and re-licensed it some years later to the FreeBSD license which means that we had problems if I'd took a recend version of crypt-md5.c. But as we can prove that it was once released under PUBLIC DOMAIN in 1999, it's OK to use that old version as base for our Java conversion. I'd be glad if my PUBLIC DOMAIN statement could be retained as well as it basically means that everybody who finds this Java snippet can use it for whatever he wants. Further modifications will then be subject to the Apache License if you put it on top of the file (-> same situation as the mentioned FreeBSD code). But if you see problems with that, you may remove my license as the Apache License is reasonably free, too. bye, christian
        Hide
        Gary Gregory added a comment -

        Hello again and thank you for your patience.

        I one file I see:

        • <p>
        • Based on the C implementation from Poul-Henning Kamp which was released under the following licence:
        • <pre>
        • ----------------------------------------------------------------------------
        • "THE BEER-WARE LICENSE" (Revision 42): <phk@login.dknet.dk> wrote this file.
        • As long as you retain this notice you can do whatever you want with this
        • stuff. If we meet some day, and you think this stuff is worth it, you can buy
        • me a beer in return. Poul-Henning Kamp
        • ----------------------------------------------------------------------------
        • Source: $FreeBSD: src/lib/libcrypt/crypt-md5.c,v 1.1 1999/01/21 13:50:09 brandon Exp $
        • http://www.freebsd.org/cgi/cvsweb.cgi/src/lib/libcrypt/crypt-md5.c?rev=1.1;content-type=text%2Fplain
        • </pre>

        I am not sure this is acceptable. IMO this can be removed because we are not shipping the C code, but rather a port of a port.

        And:

        • <p>
        • Conversion to Kotlin and from there to Java in 2012 by Christian Hammers <ch@lathspell.de> and put into the
        • Public Domain.
        • <p>
        • The C style comments are from the original C code, the ones with "//" from me.

        The granting part is also assumed because the license was granted to Apache when the patch was attached with the proper check-box selected.

        Unless someone disagrees, I'll remove the above quoted text from the code and apply soon.

        Thank you,
        Gary

        Show
        Gary Gregory added a comment - Hello again and thank you for your patience. I one file I see: <p> Based on the C implementation from Poul-Henning Kamp which was released under the following licence: <pre> ---------------------------------------------------------------------------- "THE BEER-WARE LICENSE" (Revision 42): <phk@login.dknet.dk> wrote this file. As long as you retain this notice you can do whatever you want with this stuff. If we meet some day, and you think this stuff is worth it, you can buy me a beer in return. Poul-Henning Kamp ---------------------------------------------------------------------------- Source: $FreeBSD: src/lib/libcrypt/crypt-md5.c,v 1.1 1999/01/21 13:50:09 brandon Exp $ http://www.freebsd.org/cgi/cvsweb.cgi/src/lib/libcrypt/crypt-md5.c?rev=1.1;content-type=text%2Fplain </pre> I am not sure this is acceptable. IMO this can be removed because we are not shipping the C code, but rather a port of a port. And: <p> Conversion to Kotlin and from there to Java in 2012 by Christian Hammers <ch@lathspell.de> and put into the Public Domain. <p> The C style comments are from the original C code, the ones with "//" from me. The granting part is also assumed because the license was granted to Apache when the patch was attached with the proper check-box selected. Unless someone disagrees, I'll remove the above quoted text from the code and apply soon. Thank you, Gary
        Hide
        Gary Gregory added a comment -

        I've applied the patch locally, and I will look through it tonight.

        Show
        Gary Gregory added a comment - I've applied the patch locally, and I will look through it tonight.
        Hide
        Christian Hammers added a comment -

        Adds GNU libc compatible crypt() methods.
        Md5Crypt and Sha2Crypt are almoste line-by-line translations of the original (Public Domain / Beerware) C code by me, UnixCrypt comes from another Apache project and was left almost untouched and the other files/testcases are from me.

        Show
        Christian Hammers added a comment - Adds GNU libc compatible crypt() methods. Md5Crypt and Sha2Crypt are almoste line-by-line translations of the original (Public Domain / Beerware) C code by me, UnixCrypt comes from another Apache project and was left almost untouched and the other files/testcases are from me.
        Hide
        Christian Hammers added a comment - - edited

        A new approach: While playing around with the new Kotlin JVM language, I tried to convert the original C sources of MD5 and SHA2 crypt() to Kotlin and after this to Java just to see the differences. The nice benefit of this excersise is that we now have Java implementations that are not only better commented than the ones from UTexas but also sufficiently different to not have any copyright problems. Any resemblance is due to the fact that we both translated the same C code nearly line by line.

        So please accept the attached patch "commons-codec-crypt3.diff"!

        Show
        Christian Hammers added a comment - - edited A new approach: While playing around with the new Kotlin JVM language, I tried to convert the original C sources of MD5 and SHA2 crypt() to Kotlin and after this to Java just to see the differences. The nice benefit of this excersise is that we now have Java implementations that are not only better commented than the ones from UTexas but also sufficiently different to not have any copyright problems. Any resemblance is due to the fact that we both translated the same C code nearly line by line. So please accept the attached patch "commons-codec-crypt3.diff"!
        Hide
        Gary Gregory added a comment -

        Created LEGAL-128 to discuss the legal aspect.

        Show
        Gary Gregory added a comment - Created LEGAL-128 to discuss the legal aspect.
        Hide
        Gary Gregory added a comment -

        Thanks, you can make patche files all-in-one or separate test and main, that's fine by me, it's the extra noise of formatting changes that is the issue.

        Show
        Gary Gregory added a comment - Thanks, you can make patche files all-in-one or separate test and main, that's fine by me, it's the extra noise of formatting changes that is the issue.
        Hide
        Christian Hammers added a comment -

        No problem, I will upload smaller patches once the general legal issues are solved.

        Show
        Christian Hammers added a comment - No problem, I will upload smaller patches once the general legal issues are solved.
        Hide
        Gary Gregory added a comment -

        Howdy,

        I see:

        +
        + Copyright (c) 2008-2010 The University of Texas at Austin.
        +
        + All rights reserved.
        +
        + Redistribution and use in source and binary form are permitted
        + provided that distributions retain this entire copyright notice
        + and comment. Neither the name of the University nor the names of
        + its contributors may be used to endorse or promote products
        + derived from this software without specific prior written
        + permission. THIS SOFTWARE IS PROVIDED "AS IS" AND WITHOUT ANY
        + EXPRESS OR IMPLIED WARRANTIES, INCLUDING, WITHOUT LIMITATION, THE
        + IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A
        + PARTICULAR PURPOSE.
        +
        

        I do not know if this can be used in Apache as is. I'll ask on the ML.

        The patch provided includes a lot of noise due to Javadoc changes that I imagine are not intentional. I would be better to provide a patch without this noise to make it easier to review.

        Show
        Gary Gregory added a comment - Howdy, I see: + + Copyright (c) 2008-2010 The University of Texas at Austin. + + All rights reserved. + + Redistribution and use in source and binary form are permitted + provided that distributions retain this entire copyright notice + and comment. Neither the name of the University nor the names of + its contributors may be used to endorse or promote products + derived from this software without specific prior written + permission. THIS SOFTWARE IS PROVIDED "AS IS" AND WITHOUT ANY + EXPRESS OR IMPLIED WARRANTIES, INCLUDING, WITHOUT LIMITATION, THE + IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A + PARTICULAR PURPOSE. + I do not know if this can be used in Apache as is. I'll ask on the ML. The patch provided includes a lot of noise due to Javadoc changes that I imagine are not intentional. I would be better to provide a patch without this noise to make it easier to review.
        Hide
        Christian Hammers added a comment -

        This is a patch agains svn trunk version 1241181. It includes the neccessary new classes as well as tests:

        M src/main/java/org/apache/commons/codec/digest/DigestUtils.java
        A src/main/java/org/apache/commons/codec/digest/Md5Crypt.java
        M src/main/java/org/apache/commons/codec/digest/package.html
        A src/main/java/org/apache/commons/codec/digest/README.WORK
        A src/main/java/org/apache/commons/codec/digest/Sha256Crypt.java
        A src/main/java/org/apache/commons/codec/digest/Sha512Crypt.java
        A src/main/java/org/apache/commons/codec/digest/UnixCrypt.java
        M src/test/java/org/apache/commons/codec/digest/DigestUtilsTest.java
        A src/test/java/org/apache/commons/codec/digest/Md5CryptTest.java
        A src/test/java/org/apache/commons/codec/digest/Sha256CryptTest.java
        A src/test/java/org/apache/commons/codec/digest/Sha512CryptTest.java
        A src/test/java/org/apache/commons/codec/digest/UnixCryptTest.java

        README.WORK contains some notes to the reviewer and should be deleted then.

        The files already include the Apache license text but the original utexas copyright notice is still left. I leave it for you to sort out the legal stuff with Jonathan Abbey.

        Show
        Christian Hammers added a comment - This is a patch agains svn trunk version 1241181. It includes the neccessary new classes as well as tests: M src/main/java/org/apache/commons/codec/digest/DigestUtils.java A src/main/java/org/apache/commons/codec/digest/Md5Crypt.java M src/main/java/org/apache/commons/codec/digest/package.html A src/main/java/org/apache/commons/codec/digest/README.WORK A src/main/java/org/apache/commons/codec/digest/Sha256Crypt.java A src/main/java/org/apache/commons/codec/digest/Sha512Crypt.java A src/main/java/org/apache/commons/codec/digest/UnixCrypt.java M src/test/java/org/apache/commons/codec/digest/DigestUtilsTest.java A src/test/java/org/apache/commons/codec/digest/Md5CryptTest.java A src/test/java/org/apache/commons/codec/digest/Sha256CryptTest.java A src/test/java/org/apache/commons/codec/digest/Sha512CryptTest.java A src/test/java/org/apache/commons/codec/digest/UnixCryptTest.java README.WORK contains some notes to the reviewer and should be deleted then. The files already include the Apache license text but the original utexas copyright notice is still left. I leave it for you to sort out the legal stuff with Jonathan Abbey.
        Hide
        Christian Hammers added a comment -

        I've clean up the files, wrote a lot of unit tests and added
        wrapper that accept "byte[] plaintext" so that legacy password hashes
        that were generated by other sources using ISO-8859-1 strings can
        somehow be verified.

        The following files would be added:

        M src/main/java/org/apache/commons/codec/digest/DigestUtils.java
        A src/main/java/org/apache/commons/codec/digest/Md5Crypt.java
        A src/main/java/org/apache/commons/codec/digest/README.WORK
        A src/main/java/org/apache/commons/codec/digest/Sha256Crypt.java
        A src/main/java/org/apache/commons/codec/digest/Sha512Crypt.java
        A src/main/java/org/apache/commons/codec/digest/UnixCrypt.java
        M src/test/java/org/apache/commons/codec/digest
        M src/test/java/org/apache/commons/codec/digest/DigestUtilsTest.java
        A src/test/java/org/apache/commons/codec/digest/Md5CryptTest.java
        A src/test/java/org/apache/commons/codec/digest/Sha256CryptTest.java
        A src/test/java/org/apache/commons/codec/digest/Sha512CryptTest.java
        A src/test/java/org/apache/commons/codec/digest/UnixCryptTest.java

        Before I upload them here, I wonder what exactly has to be done regarding the
        licenses. If I remember correctly only source files with the Apache licence
        header are accepted, right?
        Was it enough that Jonathan Abbey, author of the Java ports, statet his
        consent in the issue tracker or do he also have to submit this ICLA fax?
        (I already did so a while ago for another Apache project.)

        Show
        Christian Hammers added a comment - I've clean up the files, wrote a lot of unit tests and added wrapper that accept "byte[] plaintext" so that legacy password hashes that were generated by other sources using ISO-8859-1 strings can somehow be verified. The following files would be added: M src/main/java/org/apache/commons/codec/digest/DigestUtils.java A src/main/java/org/apache/commons/codec/digest/Md5Crypt.java A src/main/java/org/apache/commons/codec/digest/README.WORK A src/main/java/org/apache/commons/codec/digest/Sha256Crypt.java A src/main/java/org/apache/commons/codec/digest/Sha512Crypt.java A src/main/java/org/apache/commons/codec/digest/UnixCrypt.java M src/test/java/org/apache/commons/codec/digest M src/test/java/org/apache/commons/codec/digest/DigestUtilsTest.java A src/test/java/org/apache/commons/codec/digest/Md5CryptTest.java A src/test/java/org/apache/commons/codec/digest/Sha256CryptTest.java A src/test/java/org/apache/commons/codec/digest/Sha512CryptTest.java A src/test/java/org/apache/commons/codec/digest/UnixCryptTest.java Before I upload them here, I wonder what exactly has to be done regarding the licenses. If I remember correctly only source files with the Apache licence header are accepted, right? Was it enough that Jonathan Abbey, author of the Java ports, statet his consent in the issue tracker or do he also have to submit this ICLA fax? (I already did so a while ago for another Apache project.)
        Hide
        Julius Davies added a comment -

        I have a lot of triple-DES stuff in place over here (I'm the author), but I doesn't do crypt() style password files.

        http://juliusdavies.ca/commons-ssl/

        It might be posssible to use some of that code, too. It's all apache licensed.

        Warning: There's a good chance I don't know what I'm talking about... this crypt()-file stuff is mostly new to me, though like anyone I have created .password files to get my personal apache servers to do 1.1 basic auth.

        Show
        Julius Davies added a comment - I have a lot of triple-DES stuff in place over here (I'm the author), but I doesn't do crypt() style password files. http://juliusdavies.ca/commons-ssl/ It might be posssible to use some of that code, too. It's all apache licensed. Warning: There's a good chance I don't know what I'm talking about... this crypt()-file stuff is mostly new to me, though like anyone I have created .password files to get my personal apache servers to do 1.1 basic auth.
        Hide
        Jonathan Abbey added a comment -

        Note that we're also using Damien Miller's java implementation of the OpenBSD 'BCrypt' algorithm in Ganymede (http://www.arlut.utexas.edu/gash2/). It is also BSD licensed.

        See:

        http://git.arlut.utexas.edu/?p=ganymede;a=tree;f=src/ganymede/org/mindrot;hb=HEAD

        for the sources in our Git repo.

        We likewise have Java ports of the original triple-DES crypt, the Samba LANMANHash and NTUNICODEHash, and the salted SHA (SSHA) hash used in LDAP. We don't have the appropriate licensing on the tripe-DES crypt to pass it along, but we could relicense the Samba and SSHA hashes if anyone cares.

        http://git.arlut.utexas.edu/?p=ganymede;a=tree;f=src/ganymede/arlut/csd/crypto;hb=HEAD

        Show
        Jonathan Abbey added a comment - Note that we're also using Damien Miller's java implementation of the OpenBSD 'BCrypt' algorithm in Ganymede ( http://www.arlut.utexas.edu/gash2/ ). It is also BSD licensed. See: http://git.arlut.utexas.edu/?p=ganymede;a=tree;f=src/ganymede/org/mindrot;hb=HEAD for the sources in our Git repo. We likewise have Java ports of the original triple-DES crypt, the Samba LANMANHash and NTUNICODEHash, and the salted SHA (SSHA) hash used in LDAP. We don't have the appropriate licensing on the tripe-DES crypt to pass it along, but we could relicense the Samba and SSHA hashes if anyone cares. http://git.arlut.utexas.edu/?p=ganymede;a=tree;f=src/ganymede/arlut/csd/crypto;hb=HEAD
        Hide
        Jonathan Abbey added a comment -

        We'd be delighted to have these in Apache commons. Christian submitted a patch against our SHA-256/SHA-512 crypt routines that we are committing, and will have up on the ftp site momentarily.

        Let us know if there's anything at all we can do to facilitate this.

        Show
        Jonathan Abbey added a comment - We'd be delighted to have these in Apache commons. Christian submitted a patch against our SHA-256/SHA-512 crypt routines that we are committing, and will have up on the ftp site momentarily. Let us know if there's anything at all we can do to facilitate this.
        Hide
        Julius Davies added a comment -

        The MD5Crypt.java appears to be by Jonathan Abbey based on an original C version by Poul-Henning Kamp. But the licenses look compatible with Apache 2.0.

        (Is that not ironic... can we accept a patch that implements the original Apache HTTPd server's format for encrypting password files?)

        These all seem like great ideas for codec, in my opinion.

        Show
        Julius Davies added a comment - The MD5Crypt.java appears to be by Jonathan Abbey based on an original C version by Poul-Henning Kamp. But the licenses look compatible with Apache 2.0. (Is that not ironic... can we accept a patch that implements the original Apache HTTPd server's format for encrypting password files?) These all seem like great ideas for codec, in my opinion.
        Hide
        Christian Hammers added a comment -

        James Ratcliff, author of the above mentioned Java port just wrote me that he likes the idea of having them in the Apache Commons project so we should get no problems with license issues.

        Show
        Christian Hammers added a comment - James Ratcliff, author of the above mentioned Java port just wrote me that he likes the idea of having them in the Apache Commons project so we should get no problems with license issues.

          People

          • Assignee:
            Unassigned
            Reporter:
            Christian Hammers
          • Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development