Hadoop Common
  1. Hadoop Common
  2. HADOOP-7368

Mechanism for providing version info of hadoop jars

    Details

    • Type: Improvement Improvement
    • Status: Open
    • Priority: Major Major
    • Resolution: Unresolved
    • Affects Version/s: 0.23.0
    • Fix Version/s: None
    • Component/s: build
    • Labels:
      None
    • Tags:
      jar version providers

      Description

      In 0.20.x, only one jar (hadoop-core-*.jar) really matters. The o.a.h.util.VersionInfo combined with saveVersion.sh script (generating a package level annotation) served us well. For 0.23+, due to the project split and modularization of mapreduce (currently in MR-279), a lot more essential hadoop jars are created. The potential of mixing up the jars is significantly increased as well. We need a simple way to list the version info (version, branch, source checksum etc.) for all the jars involved. This is essential for QE and tracking down various issues.

      I propose that we use a VersionProvider interface (similar to the current VersionInfo util class) and ServiceLoader to enumerate the version providers in the jars.

      public interface VersionProvider {
        String getJar(); 
        String getPackage();
        String getVersion();
        String getRevision();
        String getBranch();
        String getDate();
        String getUrl();
        String getSourceChecksum();
      }
      

        Activity

        Hide
        Alejandro Abdelnur added a comment -

        IMO, the current mechanism brings obscurity/complexity to the build process.

        And if you open a JAR, replace a file, and reJAR, then you know you are doing something funny. (And you could do that with the annotation class as well)

        Show
        Alejandro Abdelnur added a comment - IMO, the current mechanism brings obscurity/complexity to the build process. And if you open a JAR, replace a file, and reJAR, then you know you are doing something funny. (And you could do that with the annotation class as well)
        Hide
        Allen Wittenauer added a comment -

        If it was a file, I'd be worried about people just replacing the file in the jar to bypass compatibility checks. At least with the class, they have to do a bit more work. (I also have to admit that I like the fact that today it is somewhat non-obvious how this works, at least if one can take the "interesting" version information I've seen from various 3rd party Hadoop distributions as anecdotal evidence).

        Show
        Allen Wittenauer added a comment - If it was a file, I'd be worried about people just replacing the file in the jar to bypass compatibility checks. At least with the class, they have to do a bit more work. (I also have to admit that I like the fact that today it is somewhat non-obvious how this works, at least if one can take the "interesting" version information I've seen from various 3rd party Hadoop distributions as anecdotal evidence).
        Hide
        Luke Lu added a comment -

        less error prone then the reading version.properties.

        ugh, s/then the/than/

        Show
        Luke Lu added a comment - less error prone then the reading version.properties. ugh, s/then the/than/
        Hide
        Luke Lu added a comment -

        In addition, would be possible to store this info in a properties file in the JAR instead calling a script to generate a class?

        Of course, the implementation of the VersionProvider interface is module dependent. The current saveVersion.sh script generates a package level annotation, which is actually simpler and less error prone then the reading version.properties.

        Show
        Luke Lu added a comment - In addition, would be possible to store this info in a properties file in the JAR instead calling a script to generate a class? Of course, the implementation of the VersionProvider interface is module dependent. The current saveVersion.sh script generates a package level annotation, which is actually simpler and less error prone then the reading version.properties.
        Hide
        Alejandro Abdelnur added a comment -

        In addition, would be possible to store this info in a properties file in the JAR instead calling a script to generate a class?

        Then, VersionProvider could be a class that reads a /version.properties' file to get all the values.

        Show
        Alejandro Abdelnur added a comment - In addition, would be possible to store this info in a properties file in the JAR instead calling a script to generate a class? Then, VersionProvider could be a class that reads a /version.properties' file to get all the values.
        Hide
        Todd Lipcon added a comment -

        Nice idea. This will also be handy for things like codec plugins too!

        Show
        Todd Lipcon added a comment - Nice idea. This will also be handy for things like codec plugins too!

          People

          • Assignee:
            Luke Lu
            Reporter:
            Luke Lu
          • Votes:
            0 Vote for this issue
            Watchers:
            7 Start watching this issue

            Dates

            • Created:
              Updated:

              Development