Uploaded image for project: 'Solr'
  1. Solr
  2. SOLR-14064

remove some hadoop brain-damage from build environment

Details

    • Bug
    • Status: Closed
    • Major
    • Resolution: Fixed
    • None
    • 8.5
    • None
    • None

    Description

      Some permissions and build hacks were made on behalf of hadoop. These were most definitely hacks on top of hacks.

      The background is that the hadoop code is a true nightmare to deal with, if you want to sandbox code with SecurityManager.

      Now that krisden has wrestled it (at least mostly?) to the ground, let's remove the hacks from solr security policy and lucene build that I added. We need to be strict: ensure things are really working (otherwise we get SecurityException). This also makes the configuration simpler.

      Attachments

        1. SOLR-14064.patch
          2 kB
          Robert Muir

        Issue Links

          Activity

            romseygeek Alan Woodward added a comment -

            Bulk close for 8.5.0 release

            romseygeek Alan Woodward added a comment - Bulk close for 8.5.0 release

            Commit a6e7c770c27f6467d8aa717d4b07c8682d09ebc8 in lucene-solr's branch refs/heads/gradle-master from Robert Muir
            [ https://gitbox.apache.org/repos/asf?p=lucene-solr.git;h=a6e7c77 ]

            SOLR-14064: remove some hadoop brain damage from build environment

            Some permissions and build hacks were made on behalf of hadoop: hacks on
            top of hacks. Now that the major problems such as classpath pollution and
            hadoop test code are fixed, so we can remove hacks built on top of them.

            jira-bot ASF subversion and git services added a comment - Commit a6e7c770c27f6467d8aa717d4b07c8682d09ebc8 in lucene-solr's branch refs/heads/gradle-master from Robert Muir [ https://gitbox.apache.org/repos/asf?p=lucene-solr.git;h=a6e7c77 ] SOLR-14064 : remove some hadoop brain damage from build environment Some permissions and build hacks were made on behalf of hadoop: hacks on top of hacks. Now that the major problems such as classpath pollution and hadoop test code are fixed, so we can remove hacks built on top of them.
            rcmuir Robert Muir added a comment -

            Thanks Kevin!

            rcmuir Robert Muir added a comment - Thanks Kevin!

            Commit 1761e48e534225bec234ea2d0fd223f70c962d5a in lucene-solr's branch refs/heads/branch_8x from Robert Muir
            [ https://gitbox.apache.org/repos/asf?p=lucene-solr.git;h=1761e48 ]

            SOLR-14064: remove some hadoop brain damage from build environment

            Some permissions and build hacks were made on behalf of hadoop: hacks on
            top of hacks. Now that the major problems such as classpath pollution and
            hadoop test code are fixed, so we can remove hacks built on top of them.

            jira-bot ASF subversion and git services added a comment - Commit 1761e48e534225bec234ea2d0fd223f70c962d5a in lucene-solr's branch refs/heads/branch_8x from Robert Muir [ https://gitbox.apache.org/repos/asf?p=lucene-solr.git;h=1761e48 ] SOLR-14064 : remove some hadoop brain damage from build environment Some permissions and build hacks were made on behalf of hadoop: hacks on top of hacks. Now that the major problems such as classpath pollution and hadoop test code are fixed, so we can remove hacks built on top of them.

            Commit a6e7c770c27f6467d8aa717d4b07c8682d09ebc8 in lucene-solr's branch refs/heads/master from Robert Muir
            [ https://gitbox.apache.org/repos/asf?p=lucene-solr.git;h=a6e7c77 ]

            SOLR-14064: remove some hadoop brain damage from build environment

            Some permissions and build hacks were made on behalf of hadoop: hacks on
            top of hacks. Now that the major problems such as classpath pollution and
            hadoop test code are fixed, so we can remove hacks built on top of them.

            jira-bot ASF subversion and git services added a comment - Commit a6e7c770c27f6467d8aa717d4b07c8682d09ebc8 in lucene-solr's branch refs/heads/master from Robert Muir [ https://gitbox.apache.org/repos/asf?p=lucene-solr.git;h=a6e7c77 ] SOLR-14064 : remove some hadoop brain damage from build environment Some permissions and build hacks were made on behalf of hadoop: hacks on top of hacks. Now that the major problems such as classpath pollution and hadoop test code are fixed, so we can remove hacks built on top of them.
            krisden Kevin Risden added a comment -

            Full test runs on Windows and Mac look good. +1 from me

            krisden Kevin Risden added a comment - Full test runs on Windows and Mac look good. +1 from me
            krisden Kevin Risden added a comment -

            Yea agreed. Created SOLR-14078. I don't think its specific to that test actually. Will comment more over there.

            krisden Kevin Risden added a comment - Yea agreed. Created SOLR-14078 . I don't think its specific to that test actually. Will comment more over there.
            rcmuir Robert Muir added a comment -

            Thanks for testing!

            For the issue you see in package store tests, we should make a separate issue: it is wrong for a test to be modifying the source code tree! I'm not familiar with the specific test but it seems like it should be using a temp directory.

            rcmuir Robert Muir added a comment - Thanks for testing! For the issue you see in package store tests, we should make a separate issue: it is wrong for a test to be modifying the source code tree! I'm not familiar with the specific test but it seems like it should be using a temp directory.
            krisden Kevin Risden added a comment -

            HDFS / Hadoop test runs on Mac and Windows look good. Running full test next.

            BTW not related to this change but saw this pop up. It doesn't cause a failure but seems wrong:

              2> 23044 WARN  (SUITE-TestRecoveryHdfs-seed#[AC80F05AAAD3A298]-worker) [     ] o.a.s.f.DistribPackageStore Unable to create [/Users/risdenk/repos/apache/lucene-solr/solr/core/src/test-files/solr/filestore] directory in SOLR_HOME [/Users/risdenk/repos/apache/lucene-solr/solr/core/src/test-files/solr].  Features requiring this directory may fail.
              2>           => java.security.AccessControlException: access denied ("java.io.FilePermission" "/Users/risdenk/repos/apache/lucene-solr/solr/core/src/test-files/solr/filestore" "write")
              2> 	at java.base/java.security.AccessControlContext.checkPermission(AccessControlContext.java:472)
              2> java.security.AccessControlException: access denied ("java.io.FilePermission" "/Users/risdenk/repos/apache/lucene-solr/solr/core/src/test-files/solr/filestore" "write")
              2> 	at java.security.AccessControlContext.checkPermission(AccessControlContext.java:472) ~[?:?]
              2> 	at java.security.AccessController.checkPermission(AccessController.java:897) ~[?:?]
              2> 	at java.lang.SecurityManager.checkPermission(SecurityManager.java:322) ~[?:?]
              2> 	at java.lang.SecurityManager.checkWrite(SecurityManager.java:752) ~[?:?]
              2> 	at java.io.File.mkdir(File.java:1323) ~[?:?]
              2> 	at java.io.File.mkdirs(File.java:1355) ~[?:?]
              2> 	at org.apache.solr.filestore.DistribPackageStore.ensurePackageStoreDir(DistribPackageStore.java:476) ~[java/:?]
              2> 	at org.apache.solr.filestore.DistribPackageStore.<init>(DistribPackageStore.java:65) ~[java/:?]
              2> 	at org.apache.solr.filestore.PackageStoreAPI.<init>(PackageStoreAPI.java:77) ~[java/:?]
            
            krisden Kevin Risden added a comment - HDFS / Hadoop test runs on Mac and Windows look good. Running full test next. BTW not related to this change but saw this pop up. It doesn't cause a failure but seems wrong: 2> 23044 WARN (SUITE-TestRecoveryHdfs-seed#[AC80F05AAAD3A298]-worker) [ ] o.a.s.f.DistribPackageStore Unable to create [/Users/risdenk/repos/apache/lucene-solr/solr/core/src/test-files/solr/filestore] directory in SOLR_HOME [/Users/risdenk/repos/apache/lucene-solr/solr/core/src/test-files/solr]. Features requiring this directory may fail. 2> => java.security.AccessControlException: access denied ( "java.io.FilePermission" "/Users/risdenk/repos/apache/lucene-solr/solr/core/src/test-files/solr/filestore" "write" ) 2> at java.base/java.security.AccessControlContext.checkPermission(AccessControlContext.java:472) 2> java.security.AccessControlException: access denied ( "java.io.FilePermission" "/Users/risdenk/repos/apache/lucene-solr/solr/core/src/test-files/solr/filestore" "write" ) 2> at java.security.AccessControlContext.checkPermission(AccessControlContext.java:472) ~[?:?] 2> at java.security.AccessController.checkPermission(AccessController.java:897) ~[?:?] 2> at java.lang. SecurityManager .checkPermission( SecurityManager .java:322) ~[?:?] 2> at java.lang. SecurityManager .checkWrite( SecurityManager .java:752) ~[?:?] 2> at java.io.File.mkdir(File.java:1323) ~[?:?] 2> at java.io.File.mkdirs(File.java:1355) ~[?:?] 2> at org.apache.solr.filestore.DistribPackageStore.ensurePackageStoreDir(DistribPackageStore.java:476) ~[java/:?] 2> at org.apache.solr.filestore.DistribPackageStore.<init>(DistribPackageStore.java:65) ~[java/:?] 2> at org.apache.solr.filestore.PackageStoreAPI.<init>(PackageStoreAPI.java:77) ~[java/:?]
            krisden Kevin Risden added a comment -

            This looks good on the surface. I'll run the tests on it in a bit on Windows/Mac.

            FYI created SOLR-14077 to remove the need to the hadoop metrics user.home lines.

            krisden Kevin Risden added a comment - This looks good on the surface. I'll run the tests on it in a bit on Windows/Mac. FYI created SOLR-14077 to remove the need to the hadoop metrics user.home lines.
            rcmuir Robert Muir added a comment -

            There were two major problems causing these hacks-on-top-of-hacks:

            • stacktrace inspection of hadoop "required" process executions: they use $PATH hence require <<ALL FILES>> EXECUTE. Also means overriding SecurityManager methods without other setup. This breaks java security model, it falls apart unless SecurityManager has AllPermission. But, <<ALL FILES>> EXECUTE. gotta do what you gotta do.
            • classpath pollution of the ant build runtime into the tests classpath (LUCENE-9090). This caused me to add initial permissions to allow crazy shit that was happening from solr tests -> hadoop -> jetty -> (scan of total classpath loading classes, maybe running some clinit, how fun could this get, etc)
            rcmuir Robert Muir added a comment - There were two major problems causing these hacks-on-top-of-hacks: stacktrace inspection of hadoop "required" process executions: they use $PATH hence require <<ALL FILES>> EXECUTE. Also means overriding SecurityManager methods without other setup. This breaks java security model, it falls apart unless SecurityManager has AllPermission. But, <<ALL FILES>> EXECUTE. gotta do what you gotta do. classpath pollution of the ant build runtime into the tests classpath ( LUCENE-9090 ). This caused me to add initial permissions to allow crazy shit that was happening from solr tests -> hadoop -> jetty -> (scan of total classpath loading classes, maybe running some clinit, how fun could this get, etc)
            rcmuir Robert Muir added a comment -

            only tested on OS X so far.

            rcmuir Robert Muir added a comment - only tested on OS X so far.

            People

              rcmuir Robert Muir
              rcmuir Robert Muir
              Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: