[KAFKA-12622] Automate LICENSE file validation

• Type: Task
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Fix Version/s: 3.8.0
• Assignee: Unassigned
• Reporter: John Roesler
• Created: 06/Apr/21 20:30
• Updated: 19/Feb/24 19:33

Description

In https://issues.apache.org/jira/browse/KAFKA-12602, we manually constructed a correct license file for 2.8.0. This file will certainly become wrong again in later releases, so we need to write some kind of script to automate a check.

It crossed my mind to automate the generation of the file, but it seems to be an intractable problem, considering that each dependency may change licenses, may package license files, link to them from their poms, link to them from their repos, etc. I've also found multiple URLs listed with various delimiters, broken links that I have to chase down, etc.

Therefore, it seems like the solution to aim for is simply: list all the jars that we package, and print out a report of each jar that's extra or missing vs. the ones in our `LICENSE-binary` file.

The check should be part of the release script at least, if not part of the regular build (so we keep it up to date as dependencies change).

 

Here's how I do this manually right now:

// build the binary artifacts
$ ./gradlewAll releaseTarGz

// unpack the binary artifact 
$ tar xf core/build/distributions/kafka_2.13-X.Y.Z.tgz
$ cd xf kafka_2.13-X.Y.Z

// list the packaged jars 
// (you can ignore the jars for our own modules, like kafka, kafka-clients, etc.)
$ ls libs/

// cross check the jars with the packaged LICENSE
// make sure all dependencies are listed with the right versions
$ cat LICENSE

// also double check all the mentioned license files are present
$ ls licenses 

Issue Links

• relates to: KAFKA-12602 The LICENSE and NOTICE files don't list everything they should