What was the proposed solution? I didn't see a difference in the code above vs. what's in git right now. (I could've missed it...)
The overflow happens at the "l += counter" line, yes? The statement above is "this overflow gets lost". Well, it doesn't actually get "lost", since this is integer arithmetic the MAX LONG wraps around to a MIN LONG and keeps going up. So the effects of the counter are constrained to bytes 8-15 of IV, and bytes 0-7 are fixed at the randomly generated value. We are not concerned with re-using IVs by wrapping a long - a long is 2^64 bits, and the counter is for each 16-byte block (or 2^4) so in order to wrap we'd have to have a file that's 2^68 bytes long. Not going to happen.
There actually is no "standard aes-ctr". From http://csrc.nist.gov/publications/nistpubs/800-38a/sp800-38a.pdf,
The Counter (CTR) mode is a confidentiality mode that features the application of the forward
cipher to a set of input blocks, called counters, to produce a sequence of output blocks that are
exclusive-ORed with the plaintext to produce the ciphertext, and vice versa. The sequence of
counters must have the property that each block in the sequence is different from every other
block. This condition is not restricted to a single message: across all of the messages that are
encrypted under the given key, all of the counters must be distinct.
And from Appendix B of that document:
B.2 Choosing Initial Counter Blocks
The initial counter blocks, T1, for each message that is encrypted under the given key must be
chosen in a manner than ensures the uniqueness of all the counter blocks across all the messages.
Two examples of approaches to choosing the initial counter blocks are given in this section.
In the first approach, for a given key, all plaintext messages are encrypted sequentially. Within
the messages, the same fixed set of m bits of the counter block is incremented by the standard
incrementing function. The initial counter block for the initial plaintext message may be any
string of b bits. The initial counter block for any subsequent message can be obtained by
applying the standard incrementing function to the fixed set of m bits of the final counter block
of the previous message. In effect, all of the plaintext messages that are ever encrypted under the
given key are concatenated into a single message; consequently, the total number of plaintext
blocks must not exceed 2^m . Procedures should be established to ensure the maintenance of the
state of the final counter block of the latest encrypted message, and to ensure the proper
sequencing of the messages.
A second approach to satisfying the uniqueness property across messages is to assign to each
message a unique string of b/2 bits (rounding up, if b is odd), in other words, a message nonce,
and to incorporate the message nonce into every counter block for the message. The leading b/2
bits (rounding up, if b is odd) of each counter block would be the message nonce, and the
standard incrementing function would be applied to the remaining m bits to provide an index to
the counter blocks for the message. Thus, if N is the message nonce for a given message, then
the jth counter block is given by Tj = N | [j], for j = 1…n. The number of blocks, n, in any
message must satisfy n < 2^m . A procedure should be established to ensure the uniqueness of the
This recommendation allows other methods and approaches for achieving the uniqueness
property. Validation that an implementation of the CTR mode conforms to this recommendation
will typically include an examination of the procedures for assuring the uniqueness of counter
blocks within messages and across all messages that are encrypted under a given key.
There are two "recommendations", and what's implemented follows the second recommendation. I believe that most CTR implementations (including Java) do something like this. (As an aside, I've seen an openssl implementation that follows the first recommendation.)
Another note is that if this function is altered in any way, it will break the decryption of data that has already been encrypted. I don't know if that's much at all at this point, but one would still have to tread with care.
I recommend that we leave this function alone... unless I misunderstood something about the problem.