Solr
  1. Solr
  2. SOLR-7458

Expose HDFS Block Locality Metrics

    Details

    • Type: Improvement Improvement
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 5.3, 6.0
    • Component/s: SolrCloud
    • Labels:

      Description

      We should publish block locality metrics when using HDFS.

      1. Data Locality.png
        74 kB
        Mike Drob
      2. SOLR-7458.part-2.patch
        3 kB
        Mike Drob
      3. SOLR-7458.patch
        18 kB
        Mike Drob
      4. SOLR-7458.patch
        14 kB
        Mike Drob
      5. SOLR-7458.patch
        13 kB
        Mike Drob
      6. SOLR-7458.patch
        11 kB
        Mike Drob
      7. SOLR-7458.patch
        9 kB
        Mike Drob

        Issue Links

          Activity

          Hide
          Mike Drob added a comment -

          I'm working on a patch for this, but it will require the changes in SOLR-7457 first.

          Show
          Mike Drob added a comment - I'm working on a patch for this, but it will require the changes in SOLR-7457 first.
          Hide
          Mike Drob added a comment -

          Patch against trunk.

          Show
          Mike Drob added a comment - Patch against trunk.
          Hide
          Mark Miller added a comment -

          Looks like directories is never set to non null.

          Show
          Mark Miller added a comment - Looks like directories is never set to non null.
          Hide
          Mark Miller added a comment -

          What do we know about the expense of this? For a monitoring tool that might be pounding JMX on a dozen large collections every 30-60 seconds, I'm a little worried about iterating through all the files and making server calls for each one.

          Show
          Mark Miller added a comment - What do we know about the expense of this? For a monitoring tool that might be pounding JMX on a dozen large collections every 30-60 seconds, I'm a little worried about iterating through all the files and making server calls for each one.
          Hide
          Mark Miller added a comment -

          Adding a bit of history: We used to get the size of a directory in hdfs by running through the files and adding up the size for each. On a large cluster with many cores, this was a huge performance issue. We had to switch to a single call that was more like a df command.

          That makes me think this impl will have the same type of problems. We may have to look into doing cached responses or something.

          Show
          Mark Miller added a comment - Adding a bit of history: We used to get the size of a directory in hdfs by running through the files and adding up the size for each. On a large cluster with many cores, this was a huge performance issue. We had to switch to a single call that was more like a df command. That makes me think this impl will have the same type of problems. We may have to look into doing cached responses or something.
          Hide
          Mike Drob added a comment -

          Looks like directories is never set to non null.

          Oops, thanks. I had to redo this file when splitting the patch from other work and looks like I missed this. Will fix in the next version of the patch.

          That makes me think this impl will have the same type of problems. We may have to look into doing cached responses or something.

          I know that HBase has done something similar, I'll check with some folks over there to see if they have battle tested experience.

          Show
          Mike Drob added a comment - Looks like directories is never set to non null. Oops, thanks. I had to redo this file when splitting the patch from other work and looks like I missed this. Will fix in the next version of the patch. That makes me think this impl will have the same type of problems. We may have to look into doing cached responses or something. I know that HBase has done something similar, I'll check with some folks over there to see if they have battle tested experience.
          Hide
          Mike Drob added a comment -

          Here's a version that caches the results for each file. We still have to do a directory listing each time we get a JMX request, but that seems not too bad.

          An alternative approach would be to update the cache only when we open a file (i.e. new HdfsIndexInput()), but that had more intricate object life-cycle implications so I decided to avoid it unless somebody else thought it might be worthwhile.

          Show
          Mike Drob added a comment - Here's a version that caches the results for each file. We still have to do a directory listing each time we get a JMX request, but that seems not too bad. An alternative approach would be to update the cache only when we open a file (i.e. new HdfsIndexInput() ), but that had more intricate object life-cycle implications so I decided to avoid it unless somebody else thought it might be worthwhile.
          Hide
          Mark Miller added a comment -

          This looks like a good approach.

          localhost = System.getProperty("solr.host", "localhost");

          This looks off to me. For one, I'm wary of using 'localhost' vs 127.0.0.1. Are we sure that is the right approach?

          Also, 'solr.host' is not any kind of official system property. There is a host property in solr.xml.

          private final Map<HdfsDirectory,Map<FileStatus,BlockLocation[]>> cache

          This cache needs to be thread safe.

          Show
          Mark Miller added a comment - This looks like a good approach. localhost = System.getProperty("solr.host", "localhost"); This looks off to me. For one, I'm wary of using 'localhost' vs 127.0.0.1. Are we sure that is the right approach? Also, 'solr.host' is not any kind of official system property. There is a host property in solr.xml. private final Map<HdfsDirectory,Map<FileStatus,BlockLocation[]>> cache This cache needs to be thread safe.
          Hide
          Mike Drob added a comment -

          This looks off to me. For one, I'm wary of using 'localhost' vs 127.0.0.1. Are we sure that is the right approach?

          Not at all sure. I'm open to suggestions.

          I saw the property in solr.xml, but could not figure out how to access it. A pointer here would be helpful.

          This cache needs to be thread safe.

          I guess there could be a JMX call happening while a new shard is being created? Do I need to plan for multiple concurrent JMX calls?

          Show
          Mike Drob added a comment - This looks off to me. For one, I'm wary of using 'localhost' vs 127.0.0.1. Are we sure that is the right approach? Not at all sure. I'm open to suggestions. I saw the property in solr.xml, but could not figure out how to access it. A pointer here would be helpful. This cache needs to be thread safe. I guess there could be a JMX call happening while a new shard is being created? Do I need to plan for multiple concurrent JMX calls?
          Hide
          Mark Miller added a comment -

          Not at all sure. I'm open to suggestions.

          I'm not yet sure myself. My main concern is that whatever this host property gets assigned ends up matching what hdfs returns as the host. This seems tricky to ensure offhand.

          I saw the property in solr.xml, but could not figure out how to access it. A pointer here would be helpful.

          It's SolrXmlConfig or something along those lines, accessible at the CoreContainer level. I've been noodling how you might make this CoreContainer level scope rather than static scope so you have access to that host config. Again, looks a bit tricky.

          I guess there could be a JMX call happening while a new shard is being created? Do I need to plan for multiple concurrent JMX calls?

          Right, SolrCores can be created at any time and will use new directories. I'm not so worried about staleness, but gets on a hashmap with concurrent get/updates can also hang or other bad stuff. Prob best to just use a ConcurrentHashMap.

          Show
          Mark Miller added a comment - Not at all sure. I'm open to suggestions. I'm not yet sure myself. My main concern is that whatever this host property gets assigned ends up matching what hdfs returns as the host. This seems tricky to ensure offhand. I saw the property in solr.xml, but could not figure out how to access it. A pointer here would be helpful. It's SolrXmlConfig or something along those lines, accessible at the CoreContainer level. I've been noodling how you might make this CoreContainer level scope rather than static scope so you have access to that host config. Again, looks a bit tricky. I guess there could be a JMX call happening while a new shard is being created? Do I need to plan for multiple concurrent JMX calls? Right, SolrCores can be created at any time and will use new directories. I'm not so worried about staleness, but gets on a hashmap with concurrent get/updates can also hang or other bad stuff. Prob best to just use a ConcurrentHashMap.
          Hide
          Mike Drob added a comment -

          Did some digging with HDFS folks and it looks like BlockLocation::host is generally a hostname (and not an ip, with caveat that your cluster is configured reasonably). The general solution to determining a hostname for a machine is very difficult, since any given server could have multiple interfaces with multiple names for each alias, etc. We probably just have to rely on some well known one that we can get, and not spend too much effort worrying about if "localhost" is good enough.

          Well look at SolrXmlConfig. Will add in ConcurrentHashMap.

          Show
          Mike Drob added a comment - Did some digging with HDFS folks and it looks like BlockLocation::host is generally a hostname (and not an ip, with caveat that your cluster is configured reasonably). The general solution to determining a hostname for a machine is very difficult, since any given server could have multiple interfaces with multiple names for each alias, etc. We probably just have to rely on some well known one that we can get, and not spend too much effort worrying about if "localhost" is good enough. Well look at SolrXmlConfig. Will add in ConcurrentHashMap.
          Hide
          Mike Drob added a comment -

          So I might be able to do something like

              String solrHome = SolrResourceLoader.locateSolrHome();
              ConfigSolr cfg = ConfigSolr.fromSolrHome(new SolrResourceLoader(solrHome), solrHome);
              String host = cfg.getHost();
          

          But that feels incredibly fragile. Will continue to look for alternative solutions.

          Show
          Mike Drob added a comment - So I might be able to do something like String solrHome = SolrResourceLoader.locateSolrHome(); ConfigSolr cfg = ConfigSolr.fromSolrHome( new SolrResourceLoader(solrHome), solrHome); String host = cfg.getHost(); But that feels incredibly fragile. Will continue to look for alternative solutions.
          Hide
          Mark Miller added a comment -

          What if we added DirectoryFactory to awareCompatibility in SolrResourceLoader and make HdfsDirectoryFactory implement SolrCoreAware.

          That get's us access to the SolrCore from inform which will fire before init. From the SolrCore we can access the CoreContainer and the solr.xml config that get us the configured host name.

          The general solution to determining a hostname for a machine is very difficult, since any given server could have multiple interfaces with multiple names for each alias, etc.

          Right. That is my concern.

          That's why we use the following process for Solr:

          • If no host is specified, guess which to use.
          • Allow a specific host override to be specified.

          If we don't happen to match up, won't we report the wrong results about what is local?

          Show
          Mark Miller added a comment - What if we added DirectoryFactory to awareCompatibility in SolrResourceLoader and make HdfsDirectoryFactory implement SolrCoreAware. That get's us access to the SolrCore from inform which will fire before init. From the SolrCore we can access the CoreContainer and the solr.xml config that get us the configured host name. The general solution to determining a hostname for a machine is very difficult, since any given server could have multiple interfaces with multiple names for each alias, etc. Right. That is my concern. That's why we use the following process for Solr: If no host is specified, guess which to use. Allow a specific host override to be specified. If we don't happen to match up, won't we report the wrong results about what is local?
          Hide
          Mike Drob added a comment -

          What if we added DirectoryFactory to awareCompatibility in SolrResourceLoader and make HdfsDirectoryFactory implement SolrCoreAware.

          Yes, I think this will work. New patch incoming shortly.

          Show
          Mike Drob added a comment - What if we added DirectoryFactory to awareCompatibility in SolrResourceLoader and make HdfsDirectoryFactory implement SolrCoreAware. Yes, I think this will work. New patch incoming shortly.
          Hide
          Mike Drob added a comment -

          Haven't had a chance to throw this up onto a real install and check the output, but wanted to make the patch available for review.

          Show
          Mike Drob added a comment - Haven't had a chance to throw this up onto a real install and check the output, but wanted to make the patch available for review.
          Hide
          Mike Drob added a comment -

          An update on my status here: It seems to work on startup, but there is an issue when refreshing the data. Will continue looking into it. This is probably a good indicator that I needed more logging, anyway...

          Show
          Mike Drob added a comment - An update on my status here: It seems to work on startup, but there is an issue when refreshing the data. Will continue looking into it. This is probably a good indicator that I needed more logging, anyway...
          Hide
          Mark Miller added a comment -

          Any progress on this issue Mike?

          Show
          Mark Miller added a comment - Any progress on this issue Mike?
          Hide
          Mike Drob added a comment -

          Not anything significant. I'll upload my latest patch that includes some log statements.

          Show
          Mike Drob added a comment - Not anything significant. I'll upload my latest patch that includes some log statements.
          Hide
          Mike Drob added a comment -

          So... It turns out that my problem where I thought it was not updating was completely unrelated.

          Attached is a screenshot for a 4.10.3 based patch showing what it looks like there. Will be the same in trunk, naturally, this is the one I happened to have available.

          I've got a new patch that also adds some testing, so I think we are getting close to wrapped up here.

          Show
          Mike Drob added a comment - So... It turns out that my problem where I thought it was not updating was completely unrelated. Attached is a screenshot for a 4.10.3 based patch showing what it looks like there. Will be the same in trunk, naturally, this is the one I happened to have available. I've got a new patch that also adds some testing, so I think we are getting close to wrapped up here.
          Hide
          Mark Miller added a comment -

          Should we call this attribute hdfs-locality instead of just locality?

          Show
          Mark Miller added a comment - Should we call this attribute hdfs-locality instead of just locality?
          Hide
          Mike Drob added a comment -

          Should we call this attribute hdfs-locality instead of just locality?

          That makes sense.

          Worth noting that if you are not using HDFS, then these metrics won't show up at all, so I think there is less potential for confusion.

          Show
          Mike Drob added a comment - Should we call this attribute hdfs-locality instead of just locality? That makes sense. Worth noting that if you are not using HDFS, then these metrics won't show up at all, so I think there is less potential for confusion.
          Hide
          ASF subversion and git services added a comment -

          Commit 1684711 from Mark Miller in branch 'dev/trunk'
          [ https://svn.apache.org/r1684711 ]

          SOLR-7458: Expose HDFS Block Locality Metrics via JMX

          Show
          ASF subversion and git services added a comment - Commit 1684711 from Mark Miller in branch 'dev/trunk' [ https://svn.apache.org/r1684711 ] SOLR-7458 : Expose HDFS Block Locality Metrics via JMX
          Hide
          ASF subversion and git services added a comment -

          Commit 1684720 from Mark Miller in branch 'dev/branches/branch_5x'
          [ https://svn.apache.org/r1684720 ]

          SOLR-7458: Expose HDFS Block Locality Metrics via JMX

          Show
          ASF subversion and git services added a comment - Commit 1684720 from Mark Miller in branch 'dev/branches/branch_5x' [ https://svn.apache.org/r1684720 ] SOLR-7458 : Expose HDFS Block Locality Metrics via JMX
          Hide
          Mark Miller added a comment -

          Thanks Mike!

          Show
          Mark Miller added a comment - Thanks Mike!
          Hide
          Mark Miller added a comment -

          Hmm...I've noticed this jenkins fail twice now:

          Build: https://builds.apache.org/job/Lucene-Solr-Tests-trunk-Java8/72/
          
          1 tests failed.
          REGRESSION:  org.apache.solr.core.HdfsDirectoryFactoryTest.testLocalityReporter
          
          Error Message:
          expected:<0> but was:<1>
          
          Stack Trace:
          java.lang.AssertionError: expected:<0> but was:<1>
                  at __randomizedtesting.SeedInfo.seed([FA6B1344CA602824:B4769CBC9B5059C9]:0)
                  at org.junit.Assert.fail(Assert.java:93)
                  at org.junit.Assert.failNotEquals(Assert.java:647)
                  at org.junit.Assert.assertEquals(Assert.java:128)
                  at org.junit.Assert.assertEquals(Assert.java:147)
                  at org.apache.solr.core.HdfsDirectoryFactoryTest.testLocalityReporter(HdfsDirectoryFactoryTest.java:217)
          
          Show
          Mark Miller added a comment - Hmm...I've noticed this jenkins fail twice now: Build: https://builds.apache.org/job/Lucene-Solr-Tests-trunk-Java8/72/ 1 tests failed. REGRESSION: org.apache.solr.core.HdfsDirectoryFactoryTest.testLocalityReporter Error Message: expected:<0> but was:<1> Stack Trace: java.lang.AssertionError: expected:<0> but was:<1> at __randomizedtesting.SeedInfo.seed([FA6B1344CA602824:B4769CBC9B5059C9]:0) at org.junit.Assert.fail(Assert.java:93) at org.junit.Assert.failNotEquals(Assert.java:647) at org.junit.Assert.assertEquals(Assert.java:128) at org.junit.Assert.assertEquals(Assert.java:147) at org.apache.solr.core.HdfsDirectoryFactoryTest.testLocalityReporter(HdfsDirectoryFactoryTest.java:217)
          Hide
          Mike Drob added a comment -

          Strange. I tried running the suggested ant command and was unable to reproduce locally. The command output on Jenkins isn't terribly useful either. I can submit a patch to add error messages to the asserts if we think that will help long term maintainability.

          The failure looks like the test runner on Jenkins is able to count the written blocks as local despite no hostname having been set. Maybe something in the environment affects how MiniDfsCluster operates. We could set an explicit bogus hostname to ensure that the data is "not local." I'm not sure how the test will interact with 127.0.0.1 (the next assert) on Jenkins though.

          Show
          Mike Drob added a comment - Strange. I tried running the suggested ant command and was unable to reproduce locally. The command output on Jenkins isn't terribly useful either. I can submit a patch to add error messages to the asserts if we think that will help long term maintainability. The failure looks like the test runner on Jenkins is able to count the written blocks as local despite no hostname having been set. Maybe something in the environment affects how MiniDfsCluster operates. We could set an explicit bogus hostname to ensure that the data is "not local." I'm not sure how the test will interact with 127.0.0.1 (the next assert) on Jenkins though.
          Hide
          Mark Miller added a comment -

          I can submit a patch to add error messages to the asserts if we think that will help long term maintainability.

          +1

          Show
          Mark Miller added a comment - I can submit a patch to add error messages to the asserts if we think that will help long term maintainability. +1
          Hide
          Mike Drob added a comment -

          New JIRA or just attach to this one?

          Show
          Mike Drob added a comment - New JIRA or just attach to this one?
          Hide
          Mark Miller added a comment -

          This one is fine since we are just improving the test messages.

          Show
          Mark Miller added a comment - This one is fine since we are just improving the test messages.
          Hide
          Mike Drob added a comment -

          Addendum patch against trunk.

          Show
          Mike Drob added a comment - Addendum patch against trunk.
          Hide
          ASF subversion and git services added a comment -

          Commit 1685847 from Mark Miller in branch 'dev/trunk'
          [ https://svn.apache.org/r1685847 ]

          SOLR-7458: Improve test assert messages.

          Show
          ASF subversion and git services added a comment - Commit 1685847 from Mark Miller in branch 'dev/trunk' [ https://svn.apache.org/r1685847 ] SOLR-7458 : Improve test assert messages.
          Hide
          ASF subversion and git services added a comment -

          Commit 1685848 from Mark Miller in branch 'dev/branches/branch_5x'
          [ https://svn.apache.org/r1685848 ]

          SOLR-7458: Improve test assert messages.

          Show
          ASF subversion and git services added a comment - Commit 1685848 from Mark Miller in branch 'dev/branches/branch_5x' [ https://svn.apache.org/r1685848 ] SOLR-7458 : Improve test assert messages.
          Hide
          Shalin Shekhar Mangar added a comment -

          Bulk close for 5.3.0 release

          Show
          Shalin Shekhar Mangar added a comment - Bulk close for 5.3.0 release

            People

            • Assignee:
              Mark Miller
              Reporter:
              Mike Drob
            • Votes:
              1 Vote for this issue
              Watchers:
              6 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development