Avro
  1. Avro
  2. AVRO-318

Build.sh cleanup: md5/sha1 on Mac OS X, don't fail hard if build/ exists, and play nice with Forrest

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Blocker Blocker
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 1.3.0
    • Component/s: build
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      I encountered three problems when trying to build Avro today (via "./build.sh dist) to test interop:

      • md5sum and sha1sum do not exist on Mac (md5 is the replacement for the former)
      • If "build/" exists, you fail hard with "mkdir: build: File exists"
      • If you don't have Java 5, Forrest fails with "'java5.home' is not defined. Forrest requires Java 5." and other good stuff.
      1. AVRO-318.patch
        0.5 kB
        Jeff Hammerbacher
      2. AVRO-318.patch
        0.9 kB
        Doug Cutting

        Activity

        Hide
        Jeff Hammerbacher added a comment -

        Blocker for 1.3 release. We need to play nicely with OS X.

        Show
        Jeff Hammerbacher added a comment - Blocker for 1.3 release. We need to play nicely with OS X.
        Hide
        Jeff Hammerbacher added a comment -

        Can someone with more bash skills than me clean up our build.sh so that folks with a Mac can build Avro out of the box without ugly errors?

        Show
        Jeff Hammerbacher added a comment - Can someone with more bash skills than me clean up our build.sh so that folks with a Mac can build Avro out of the box without ugly errors?
        Hide
        Doug Cutting added a comment -

        The Java 5 stuff is just a requirement of Forrest. No way around it.

        Fixing the md5 stuff should be easy, but I don't have a Mac to do it on.

        Show
        Doug Cutting added a comment - The Java 5 stuff is just a requirement of Forrest. No way around it. Fixing the md5 stuff should be easy, but I don't have a Mac to do it on.
        Hide
        Doug Cutting added a comment -

        Is this really a blocker for 1.3?

        Show
        Doug Cutting added a comment - Is this really a blocker for 1.3?
        Hide
        Jeff Hammerbacher added a comment -

        http://raamdev.com/howto-install-md5sum-sha1sum-on-mac-os-x

        "/sbin/md5 -r <filename>" can replace "md5sum" for the Mac; for sha1sum, you'll need openssl, and you can used "openssl sha1 <filename>", though I'm not sure about the format of the output

        Show
        Jeff Hammerbacher added a comment - http://raamdev.com/howto-install-md5sum-sha1sum-on-mac-os-x "/sbin/md5 -r <filename>" can replace "md5sum" for the Mac; for sha1sum, you'll need openssl, and you can used "openssl sha1 <filename>", though I'm not sure about the format of the output
        Hide
        Jeff Hammerbacher added a comment -

        Carl Steinbach notes that "sudo port install coreutils" will get both of these utilities for Mac users.

        Show
        Jeff Hammerbacher added a comment - Carl Steinbach notes that "sudo port install coreutils" will get both of these utilities for Mac users.
        Hide
        Scott Carey added a comment -

        The Java 5 stuff is just a requirement of Forrest. No way around it.

        As in Forrest won't build without Java5 – Java6 breaks it? or as in that env.variable must be set?

        If the latter, just have the script check to see if the env var is set, and if not then set it locally to

        /System/Library/Frameworks/JavaVM.framework/Versions/1.5/Home/
        

        which on all recent Macs should be a symlink to either an actual java 1.5, or 1.6 if that doesn't exist.

        bin/java is contained within.

        On Snow Leopard (OSX 10.6) there is no Java 5, only Zul... err Java 6.

        Snow Leopard's java version mapping directory looks like this:

        $ ls -lh /System/Library/Frameworks/JavaVM.framework/Versions/
        total 64
        lrwxr-xr-x  1 root  wheel     5B Dec 17 09:46 1.3 -> 1.3.1
        drwxr-xr-x  3 root  wheel   102B Jul 20  2009 1.3.1
        lrwxr-xr-x  1 root  wheel    10B Dec 17 09:46 1.4 -> CurrentJDK
        lrwxr-xr-x  1 root  wheel    10B Dec 17 09:46 1.4.2 -> CurrentJDK
        lrwxr-xr-x  1 root  wheel    10B Dec 17 09:46 1.5 -> CurrentJDK
        lrwxr-xr-x  1 root  wheel    10B Dec 17 09:46 1.5.0 -> CurrentJDK
        lrwxr-xr-x  1 root  wheel     5B Dec 17 09:46 1.6 -> 1.6.0
        drwxr-xr-x  8 root  wheel   272B Sep 29 10:06 1.6.0
        lrwxr-xr-x  1 root  wheel     3B Dec 17 09:46 CurrentJDK -> 1.6
        
        Show
        Scott Carey added a comment - The Java 5 stuff is just a requirement of Forrest. No way around it. As in Forrest won't build without Java5 – Java6 breaks it? or as in that env.variable must be set? If the latter, just have the script check to see if the env var is set, and if not then set it locally to /System/Library/Frameworks/JavaVM.framework/Versions/1.5/Home/ which on all recent Macs should be a symlink to either an actual java 1.5, or 1.6 if that doesn't exist. bin/java is contained within. On Snow Leopard (OSX 10.6) there is no Java 5, only Zul... err Java 6. Snow Leopard's java version mapping directory looks like this: $ ls -lh /System/Library/Frameworks/JavaVM.framework/Versions/ total 64 lrwxr-xr-x 1 root wheel 5B Dec 17 09:46 1.3 -> 1.3.1 drwxr-xr-x 3 root wheel 102B Jul 20 2009 1.3.1 lrwxr-xr-x 1 root wheel 10B Dec 17 09:46 1.4 -> CurrentJDK lrwxr-xr-x 1 root wheel 10B Dec 17 09:46 1.4.2 -> CurrentJDK lrwxr-xr-x 1 root wheel 10B Dec 17 09:46 1.5 -> CurrentJDK lrwxr-xr-x 1 root wheel 10B Dec 17 09:46 1.5.0 -> CurrentJDK lrwxr-xr-x 1 root wheel 5B Dec 17 09:46 1.6 -> 1.6.0 drwxr-xr-x 8 root wheel 272B Sep 29 10:06 1.6.0 lrwxr-xr-x 1 root wheel 3B Dec 17 09:46 CurrentJDK -> 1.6
        Hide
        Jeff Hammerbacher added a comment -

        Here's a patch that adds a comment to README.txt about the dependency of the "dist" target on GNU coreutils.

        Show
        Jeff Hammerbacher added a comment - Here's a patch that adds a comment to README.txt about the dependency of the "dist" target on GNU coreutils.
        Hide
        Doug Cutting added a comment -

        Here's a patch that clarifies these requirements in README.txt.

        Show
        Doug Cutting added a comment - Here's a patch that clarifies these requirements in README.txt.
        Hide
        Doug Cutting added a comment -

        Scott, Forrest dies when run with Java 1.5. Forrest may be a dead project. We should probably move the documentation to some other system. But, in the meantime, it's what we're using.

        Show
        Doug Cutting added a comment - Scott, Forrest dies when run with Java 1.5. Forrest may be a dead project. We should probably move the documentation to some other system. But, in the meantime, it's what we're using.
        Hide
        Doug Cutting added a comment -

        Hammer: this dependency is not Python-specific, since these are called directly from the top-level build.sh.

        Show
        Doug Cutting added a comment - Hammer: this dependency is not Python-specific, since these are called directly from the top-level build.sh.
        Hide
        Matt Massie added a comment -

        I'm +1 on Doug's patch to simply document the requirements (e.g. md5sum).

        AFAIK, this issue only effects people who want to build all the release artifacts themselves on MacOS.

        In the future, we can create simple bash functions to make generating md5/sha1 signatures more portable.

        Show
        Matt Massie added a comment - I'm +1 on Doug's patch to simply document the requirements (e.g. md5sum). AFAIK, this issue only effects people who want to build all the release artifacts themselves on MacOS. In the future, we can create simple bash functions to make generating md5/sha1 signatures more portable.
        Hide
        Doug Cutting added a comment -

        I just committed this.

        Show
        Doug Cutting added a comment - I just committed this.
        Hide
        Patrick Angeles added a comment -

        Managed to get Forrest to run on Snow Leopard via the workaround described here:

        http://wiki.oneswarm.org/index.php/OS_X_10.6_Snow_Leopard

        More context:

        http://discussions.apple.com/thread.jspa?threadID=2140099&tstart=0

        It's fugly but it works.

        Show
        Patrick Angeles added a comment - Managed to get Forrest to run on Snow Leopard via the workaround described here: http://wiki.oneswarm.org/index.php/OS_X_10.6_Snow_Leopard More context: http://discussions.apple.com/thread.jspa?threadID=2140099&tstart=0 It's fugly but it works.

          People

          • Assignee:
            Unassigned
            Reporter:
            Jeff Hammerbacher
          • Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development