Uploaded image for project: 'Hadoop Common'
  1. Hadoop Common
  2. HADOOP-12626

Intel ISA-L libraries should be added to the Dockerfile

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Blocker
    • Resolution: Fixed
    • Affects Version/s: 3.0.0-alpha1
    • Fix Version/s: 3.0.0-alpha1
    • Component/s: io
    • Labels:
      None

      Description

      HADOOP-11887 added a compile and runtime dependence on the Intel ISA-L library but didn't add it to the Dockerfile so that it could be part of the Docker-based build environment (start-build-env.sh). This needs to be fixed.

      1. HADOOP-12626-v1.patch
        0.8 kB
        Kai Zheng
      2. HADOOP-12626-v2.patch
        0.8 kB
        Kai Zheng
      3. HADOOP-12626-v3.patch
        0.6 kB
        Kai Zheng

        Issue Links

          Activity

          Hide
          drankye Kai Zheng added a comment -

          Thanks Allen for reporting this. I would help with it and would you help give me some hints or steps do do so? Or is there any example in the code base I can learn about? Thanks!

          Show
          drankye Kai Zheng added a comment - Thanks Allen for reporting this. I would help with it and would you help give me some hints or steps do do so? Or is there any example in the code base I can learn about? Thanks!
          Hide
          aw Allen Wittenauer added a comment -

          The Dockerfile is in dev-support/docker/Dockerfile . Just need to add whatever magic is necessary to get it installed.

          Show
          aw Allen Wittenauer added a comment - The Dockerfile is in dev-support/docker/Dockerfile . Just need to add whatever magic is necessary to get it installed.
          Hide
          drankye Kai Zheng added a comment -

          Thanks Allen for the guide. Let me do it.

          Show
          drankye Kai Zheng added a comment - Thanks Allen for the guide. Let me do it.
          Hide
          drankye Kai Zheng added a comment -

          I've got my docker environment up and am working on it.

          Show
          drankye Kai Zheng added a comment - I've got my docker environment up and am working on it.
          Hide
          drankye Kai Zheng added a comment -

          As the official RPM package is still on the going and isn't available yet, I'm going to create one and host it somewhere first for the downloading and installation for this docker building support. When the official one is ready, it will be easily swapped out. This may be a better option than complicating the Dockerfile with kinds of things that build the ISA-L from the source codes considering there're some tools to be equipped first.

          Show
          drankye Kai Zheng added a comment - As the official RPM package is still on the going and isn't available yet, I'm going to create one and host it somewhere first for the downloading and installation for this docker building support. When the official one is ready, it will be easily swapped out. This may be a better option than complicating the Dockerfile with kinds of things that build the ISA-L from the source codes considering there're some tools to be equipped first.
          Hide
          aw Allen Wittenauer added a comment -

          As the official RPM package is still on the going and isn't available yet, I'm going to create one and host it somewhere first for the downloading and installation for this docker building support. When the official one is ready, it will be easily swapped out. This may be a better option than complicating the Dockerfile with kinds of things that build the ISA-L from the source codes considering there're some tools to be equipped first.

          Definitely not. It's simply not sustainable to host random binaries some where and expect the system to support it. If this is blocked on Intel actually making these libraries usable, then it's blocked. It also makes me REALLY question why this support was committed to Hadoop since it's clearly not ready for production if the binaries aren't out there.

          It's probably also worth pointing out that our Docker image is using Ubuntu, so RPMs won't work anyway. We need something that can be installed with apt.

          Show
          aw Allen Wittenauer added a comment - As the official RPM package is still on the going and isn't available yet, I'm going to create one and host it somewhere first for the downloading and installation for this docker building support. When the official one is ready, it will be easily swapped out. This may be a better option than complicating the Dockerfile with kinds of things that build the ISA-L from the source codes considering there're some tools to be equipped first. Definitely not. It's simply not sustainable to host random binaries some where and expect the system to support it. If this is blocked on Intel actually making these libraries usable, then it's blocked. It also makes me REALLY question why this support was committed to Hadoop since it's clearly not ready for production if the binaries aren't out there. It's probably also worth pointing out that our Docker image is using Ubuntu, so RPMs won't work anyway. We need something that can be installed with apt.
          Hide
          drankye Kai Zheng added a comment -

          Hi Allen,

          Sorry for my misunderstanding. May I clarify? Thanks.
          Intel delivers ISA-L library in 01.org site and provides source package downloading, which I thought is a good practice, considering the customers expect to build from the source codes and install it themselves. It's mature and already widely used (Ceph is an example for open source). The docker building support would desire a binary package as I said above, it's fine and I have already made the request to the ISA-L team. It will be solved anyway, and I just thought before I can get the formal binary package to try the docker building, I'd like to try my work using a temporary resource.

          By the way, I almost got it work. The temporary package is provided here:
          https://github.com/drankye/isal/tree/master/packages

          And my fix is as follows:

          diff --git a/dev-support/docker/Dockerfile b/dev-support/docker/Dockerfile
          index e80df29..121c43d 100644
          --- a/dev-support/docker/Dockerfile
          +++ b/dev-support/docker/Dockerfile
          +
          +######
          +# Install ISA-L library
          +# It uses a temporary resource package, and it will soon be changed to use a formal one.
          +######
          +RUN curl -L https://github.com/drankye/isal/raw/master/packages/isa-l-2.14.0_amd64.deb \
          +         -o /opt/isa-l-2.14.0_amd64.deb && \
          +    dpkg -i /opt/isa-l-2.14.0_amd64.deb
          +
           # Add a welcome message and environment checks.
           ADD hadoop_env_checks.sh /root/hadoop_env_checks.sh
           RUN chmod 755 /root/hadoop_env_checks.sh
          

          Please comment, thanks.

          Show
          drankye Kai Zheng added a comment - Hi Allen, Sorry for my misunderstanding. May I clarify? Thanks. Intel delivers ISA-L library in 01.org site and provides source package downloading, which I thought is a good practice, considering the customers expect to build from the source codes and install it themselves. It's mature and already widely used (Ceph is an example for open source). The docker building support would desire a binary package as I said above, it's fine and I have already made the request to the ISA-L team. It will be solved anyway, and I just thought before I can get the formal binary package to try the docker building, I'd like to try my work using a temporary resource. By the way, I almost got it work. The temporary package is provided here: https://github.com/drankye/isal/tree/master/packages And my fix is as follows: diff --git a/dev-support/docker/Dockerfile b/dev-support/docker/Dockerfile index e80df29..121c43d 100644 --- a/dev-support/docker/Dockerfile +++ b/dev-support/docker/Dockerfile + +###### +# Install ISA-L library +# It uses a temporary resource package , and it will soon be changed to use a formal one. +###### +RUN curl -L https: //github.com/drankye/isal/raw/master/packages/isa-l-2.14.0_amd64.deb \ + -o /opt/isa-l-2.14.0_amd64.deb && \ + dpkg -i /opt/isa-l-2.14.0_amd64.deb + # Add a welcome message and environment checks. ADD hadoop_env_checks.sh /root/hadoop_env_checks.sh RUN chmod 755 /root/hadoop_env_checks.sh Please comment, thanks.
          Hide
          drankye Kai Zheng added a comment -

          Please ref. ISA-L library here, https://01.org/intel%C2%AE-storage-acceleration-library-open-source-version.

          Intel ISA-L team is working on to provide binary packages, before that, please let me know if we need the fix like above as I suggested? Thanks.

          Show
          drankye Kai Zheng added a comment - Please ref. ISA-L library here, https://01.org/intel%C2%AE-storage-acceleration-library-open-source-version . Intel ISA-L team is working on to provide binary packages, before that, please let me know if we need the fix like above as I suggested? Thanks.
          Hide
          drankye Kai Zheng added a comment -

          Uploaded the initial patch as discussed above.

          Show
          drankye Kai Zheng added a comment - Uploaded the initial patch as discussed above.
          Hide
          drankye Kai Zheng added a comment -

          Update: ISA-L has been accepted into Debian unstable https://packages.debian.org/sid/libisal2 and you can pull debs from standard locations https://packages.debian.org/sid/amd64/libisal2/download

          Show
          drankye Kai Zheng added a comment - Update: ISA-L has been accepted into Debian unstable https://packages.debian.org/sid/libisal2 and you can pull debs from standard locations https://packages.debian.org/sid/amd64/libisal2/download
          Hide
          drankye Kai Zheng added a comment -

          Updated the patch to use the formal ISA-L package.

          Show
          drankye Kai Zheng added a comment - Updated the patch to use the formal ISA-L package.
          Hide
          hadoopqa Hadoop QA added a comment -

          A patch to the testing environment has been detected.
          Re-executing against the patched versions to perform further tests.
          The console is at https://builds.apache.org/job/PreCommit-HADOOP-Build/8832/console in case of problems.

          Show
          hadoopqa Hadoop QA added a comment - A patch to the testing environment has been detected. Re-executing against the patched versions to perform further tests. The console is at https://builds.apache.org/job/PreCommit-HADOOP-Build/8832/console in case of problems.
          Hide
          hadoopqa Hadoop QA added a comment -
          +1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 11s Docker mode activated.
          0 shelldocs 0m 4s Shelldocs was not available.
          +1 @author 0m 0s The patch does not contain any @author tags.
          0 mvndep 0m 14s Maven dependency ordering for branch
          0 mvndep 0m 15s Maven dependency ordering for patch
          +1 shellcheck 0m 9s There were no new shellcheck issues.
          +1 whitespace 0m 0s Patch has no whitespace issues.
          +1 asflicense 0m 21s Patch does not generate ASF License warnings.
          1m 22s



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:0ca8df7
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12792452/HADOOP-12626-v2.patch
          JIRA Issue HADOOP-12626
          Optional Tests asflicense shellcheck shelldocs
          uname Linux 9e6f40e3479b 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / 2e040d3
          shellcheck v0.4.3
          modules C: U:
          Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/8832/console
          Powered by Apache Yetus 0.2.0 http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - +1 overall Vote Subsystem Runtime Comment 0 reexec 0m 11s Docker mode activated. 0 shelldocs 0m 4s Shelldocs was not available. +1 @author 0m 0s The patch does not contain any @author tags. 0 mvndep 0m 14s Maven dependency ordering for branch 0 mvndep 0m 15s Maven dependency ordering for patch +1 shellcheck 0m 9s There were no new shellcheck issues. +1 whitespace 0m 0s Patch has no whitespace issues. +1 asflicense 0m 21s Patch does not generate ASF License warnings. 1m 22s Subsystem Report/Notes Docker Image:yetus/hadoop:0ca8df7 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12792452/HADOOP-12626-v2.patch JIRA Issue HADOOP-12626 Optional Tests asflicense shellcheck shelldocs uname Linux 9e6f40e3479b 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 2e040d3 shellcheck v0.4.3 modules C: U: Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/8832/console Powered by Apache Yetus 0.2.0 http://yetus.apache.org This message was automatically generated.
          Hide
          aw Allen Wittenauer added a comment -

          The download needs to happen before the 'YETUS CUT HERE' line if you want to enable ISA-L testing during precommit...

          Show
          aw Allen Wittenauer added a comment - The download needs to happen before the 'YETUS CUT HERE' line if you want to enable ISA-L testing during precommit...
          Hide
          drankye Kai Zheng added a comment -

          Updated the patch according to Allen Wittenauer's review comments. Thanks Allen for the review!

          Show
          drankye Kai Zheng added a comment - Updated the patch according to Allen Wittenauer 's review comments. Thanks Allen for the review!
          Hide
          hadoopqa Hadoop QA added a comment -

          A patch to the testing environment has been detected.
          Re-executing against the patched versions to perform further tests.
          The console is at https://builds.apache.org/job/PreCommit-HADOOP-Build/8839/console in case of problems.

          Show
          hadoopqa Hadoop QA added a comment - A patch to the testing environment has been detected. Re-executing against the patched versions to perform further tests. The console is at https://builds.apache.org/job/PreCommit-HADOOP-Build/8839/console in case of problems.
          Hide
          hadoopqa Hadoop QA added a comment -
          +1 overall



          Vote Subsystem Runtime Comment
          0 reexec 8m 44s Docker mode activated.
          0 shelldocs 0m 3s Shelldocs was not available.
          +1 @author 0m 0s The patch does not contain any @author tags.
          0 mvndep 0m 15s Maven dependency ordering for branch
          0 mvndep 0m 14s Maven dependency ordering for patch
          +1 shellcheck 0m 9s There were no new shellcheck issues.
          +1 whitespace 0m 0s Patch has no whitespace issues.
          +1 asflicense 0m 17s Patch does not generate ASF License warnings.
          9m 50s



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:0ca8df7
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12792808/HADOOP-12626-v3.patch
          JIRA Issue HADOOP-12626
          Optional Tests asflicense shellcheck shelldocs
          uname Linux 8bdcbf31bddb 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / 247a790
          shellcheck v0.4.3
          modules C: U:
          Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/8839/console
          Powered by Apache Yetus 0.2.0 http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - +1 overall Vote Subsystem Runtime Comment 0 reexec 8m 44s Docker mode activated. 0 shelldocs 0m 3s Shelldocs was not available. +1 @author 0m 0s The patch does not contain any @author tags. 0 mvndep 0m 15s Maven dependency ordering for branch 0 mvndep 0m 14s Maven dependency ordering for patch +1 shellcheck 0m 9s There were no new shellcheck issues. +1 whitespace 0m 0s Patch has no whitespace issues. +1 asflicense 0m 17s Patch does not generate ASF License warnings. 9m 50s Subsystem Report/Notes Docker Image:yetus/hadoop:0ca8df7 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12792808/HADOOP-12626-v3.patch JIRA Issue HADOOP-12626 Optional Tests asflicense shellcheck shelldocs uname Linux 8bdcbf31bddb 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 247a790 shellcheck v0.4.3 modules C: U: Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/8839/console Powered by Apache Yetus 0.2.0 http://yetus.apache.org This message was automatically generated.
          Hide
          drankye Kai Zheng added a comment -

          Hi Allen Wittenauer or Colin P. McCabe, would you help review and commit this? I did a local test as well, with the docker environment set up with this change, I built successfully with ISA-L enabled. With this, I thought we can make a simple change in YETUS-222 to trigger the building with ISA-L. Thanks.

          Show
          drankye Kai Zheng added a comment - Hi Allen Wittenauer or Colin P. McCabe , would you help review and commit this? I did a local test as well, with the docker environment set up with this change, I built successfully with ISA-L enabled. With this, I thought we can make a simple change in YETUS-222 to trigger the building with ISA-L. Thanks.
          Hide
          cmccabe Colin P. McCabe added a comment -

          +1.

          Show
          cmccabe Colin P. McCabe added a comment - +1.
          Hide
          cmccabe Colin P. McCabe added a comment -

          Committed to 3.0, thanks!

          Show
          cmccabe Colin P. McCabe added a comment - Committed to 3.0, thanks!
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-trunk-Commit #9475 (See https://builds.apache.org/job/Hadoop-trunk-Commit/9475/)
          HADOOP-12626. Intel ISA-L libraries should be added to the Dockerfile (cmccabe: rev fbe3e86970cda0390eaf3c4820539a6dcb957926)

          • dev-support/docker/Dockerfile
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-trunk-Commit #9475 (See https://builds.apache.org/job/Hadoop-trunk-Commit/9475/ ) HADOOP-12626 . Intel ISA-L libraries should be added to the Dockerfile (cmccabe: rev fbe3e86970cda0390eaf3c4820539a6dcb957926) dev-support/docker/Dockerfile
          Hide
          drankye Kai Zheng added a comment -

          Thanks Colin P. McCabe for committing this!

          Show
          drankye Kai Zheng added a comment - Thanks Colin P. McCabe for committing this!

            People

            • Assignee:
              drankye Kai Zheng
              Reporter:
              aw Allen Wittenauer
            • Votes:
              0 Vote for this issue
              Watchers:
              6 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development