I'm working on a patch for this, but it will require the changes in SOLR-7457 first.

Patch against trunk.

Looks like directories is never set to non null.

What do we know about the expense of this? For a monitoring tool that might be pounding JMX on a dozen large collections every 30-60 seconds, I'm a little worried about iterating through all the files and making server calls for each one.

Adding a bit of history: We used to get the size of a directory in hdfs by running through the files and adding up the size for each. On a large cluster with many cores, this was a huge performance issue. We had to switch to a single call that was more like a df command.

That makes me think this impl will have the same type of problems. We may have to look into doing cached responses or something.

Looks like directories is never set to non null.

Oops, thanks. I had to redo this file when splitting the patch from other work and looks like I missed this. Will fix in the next version of the patch.

That makes me think this impl will have the same type of problems. We may have to look into doing cached responses or something.

I know that HBase has done something similar, I'll check with some folks over there to see if they have battle tested experience.

Here's a version that caches the results for each file. We still have to do a directory listing each time we get a JMX request, but that seems not too bad.

An alternative approach would be to update the cache only when we open a file (i.e. new HdfsIndexInput()), but that had more intricate object life-cycle implications so I decided to avoid it unless somebody else thought it might be worthwhile.

This looks like a good approach.

localhost = System.getProperty("solr.host", "localhost");

This looks off to me. For one, I'm wary of using 'localhost' vs 127.0.0.1. Are we sure that is the right approach?

Also, 'solr.host' is not any kind of official system property. There is a host property in solr.xml.

private final Map<HdfsDirectory,Map<FileStatus,BlockLocation[]>> cache

This cache needs to be thread safe.

This looks off to me. For one, I'm wary of using 'localhost' vs 127.0.0.1. Are we sure that is the right approach?

Not at all sure. I'm open to suggestions.

I saw the property in solr.xml, but could not figure out how to access it. A pointer here would be helpful.

This cache needs to be thread safe.

I guess there could be a JMX call happening while a new shard is being created? Do I need to plan for multiple concurrent JMX calls?

Not at all sure. I'm open to suggestions.

I'm not yet sure myself. My main concern is that whatever this host property gets assigned ends up matching what hdfs returns as the host. This seems tricky to ensure offhand.

I saw the property in solr.xml, but could not figure out how to access it. A pointer here would be helpful.

It's SolrXmlConfig or something along those lines, accessible at the CoreContainer level. I've been noodling how you might make this CoreContainer level scope rather than static scope so you have access to that host config. Again, looks a bit tricky.

I guess there could be a JMX call happening while a new shard is being created? Do I need to plan for multiple concurrent JMX calls?

Right, SolrCores can be created at any time and will use new directories. I'm not so worried about staleness, but gets on a hashmap with concurrent get/updates can also hang or other bad stuff. Prob best to just use a ConcurrentHashMap.

Did some digging with HDFS folks and it looks like BlockLocation::host is generally a hostname (and not an ip, with caveat that your cluster is configured reasonably). The general solution to determining a hostname for a machine is very difficult, since any given server could have multiple interfaces with multiple names for each alias, etc. We probably just have to rely on some well known one that we can get, and not spend too much effort worrying about if "localhost" is good enough.

Well look at SolrXmlConfig. Will add in ConcurrentHashMap.

So I might be able to do something like

    String solrHome = SolrResourceLoader.locateSolrHome();
    ConfigSolr cfg = ConfigSolr.fromSolrHome(new SolrResourceLoader(solrHome), solrHome);
    String host = cfg.getHost();

But that feels incredibly fragile. Will continue to look for alternative solutions.

What if we added DirectoryFactory to awareCompatibility in SolrResourceLoader and make HdfsDirectoryFactory implement SolrCoreAware.

That get's us access to the SolrCore from inform which will fire before init. From the SolrCore we can access the CoreContainer and the solr.xml config that get us the configured host name.

The general solution to determining a hostname for a machine is very difficult, since any given server could have multiple interfaces with multiple names for each alias, etc.

Right. That is my concern.

That's why we use the following process for Solr:

• If no host is specified, guess which to use.
• Allow a specific host override to be specified.

If we don't happen to match up, won't we report the wrong results about what is local?

What if we added DirectoryFactory to awareCompatibility in SolrResourceLoader and make HdfsDirectoryFactory implement SolrCoreAware.

Yes, I think this will work. New patch incoming shortly.

Haven't had a chance to throw this up onto a real install and check the output, but wanted to make the patch available for review.

An update on my status here: It seems to work on startup, but there is an issue when refreshing the data. Will continue looking into it. This is probably a good indicator that I needed more logging, anyway...

Any progress on this issue Mike?

Not anything significant. I'll upload my latest patch that includes some log statements.

So... It turns out that my problem where I thought it was not updating was completely unrelated.

Attached is a screenshot for a 4.10.3 based patch showing what it looks like there. Will be the same in trunk, naturally, this is the one I happened to have available.

I've got a new patch that also adds some testing, so I think we are getting close to wrapped up here.

Should we call this attribute hdfs-locality instead of just locality?

Should we call this attribute hdfs-locality instead of just locality?

That makes sense.

Worth noting that if you are not using HDFS, then these metrics won't show up at all, so I think there is less potential for confusion.

Commit 1684711 from Mark Miller in branch 'dev/trunk'
[ https://svn.apache.org/r1684711 ]

SOLR-7458: Expose HDFS Block Locality Metrics via JMX

Commit 1684720 from Mark Miller in branch 'dev/branches/branch_5x'
[ https://svn.apache.org/r1684720 ]

SOLR-7458: Expose HDFS Block Locality Metrics via JMX

Thanks Mike!

Hmm...I've noticed this jenkins fail twice now:

Build: https://builds.apache.org/job/Lucene-Solr-Tests-trunk-Java8/72/

1 tests failed.
REGRESSION:  org.apache.solr.core.HdfsDirectoryFactoryTest.testLocalityReporter

Error Message:
expected:<0> but was:<1>

Stack Trace:
java.lang.AssertionError: expected:<0> but was:<1>
        at __randomizedtesting.SeedInfo.seed([FA6B1344CA602824:B4769CBC9B5059C9]:0)
        at org.junit.Assert.fail(Assert.java:93)
        at org.junit.Assert.failNotEquals(Assert.java:647)
        at org.junit.Assert.assertEquals(Assert.java:128)
        at org.junit.Assert.assertEquals(Assert.java:147)
        at org.apache.solr.core.HdfsDirectoryFactoryTest.testLocalityReporter(HdfsDirectoryFactoryTest.java:217)

Strange. I tried running the suggested ant command and was unable to reproduce locally. The command output on Jenkins isn't terribly useful either. I can submit a patch to add error messages to the asserts if we think that will help long term maintainability.

The failure looks like the test runner on Jenkins is able to count the written blocks as local despite no hostname having been set. Maybe something in the environment affects how MiniDfsCluster operates. We could set an explicit bogus hostname to ensure that the data is "not local." I'm not sure how the test will interact with 127.0.0.1 (the next assert) on Jenkins though.

I can submit a patch to add error messages to the asserts if we think that will help long term maintainability.

+1

New JIRA or just attach to this one?

This one is fine since we are just improving the test messages.

Addendum patch against trunk.

Commit 1685847 from Mark Miller in branch 'dev/trunk'
[ https://svn.apache.org/r1685847 ]

SOLR-7458: Improve test assert messages.

Commit 1685848 from Mark Miller in branch 'dev/branches/branch_5x'
[ https://svn.apache.org/r1685848 ]

SOLR-7458: Improve test assert messages.

Bulk close for 5.3.0 release