Details
Minor Description
I have been testing a few Spark jobs we have at my company which work off of TableSnapshotInputFormat to read directly from the filesystem snapshots created on another EMR/Hbase cluster and stored in S3. During performance testing I found various small changes which would greatly enhance peformance. Right now we are running our jobs linked with a patched version of HBase 1.4.0 in which I made these changes, and I am hoping to submit my patch for review and eventual acceptance into the main codebase.
The list of changes are :
1. a flag to control whether the snapshot restore uses a UUID based random temp dir in the specified restore directory. We use the flag to turn this off so that we can benefit from a AWS S3 specific bucket partitioning scheme we have provisioned. The way S3 partitioning works, you have to give a fixed path prefix and a pattern of files after that, and AWS can then partition on the paths after the fixed prefix into different resources to get more parallelization. We were advised by AWS that we could only get this good partitioning behavior if we didn't have that rancom directory in there.
2. a flag to turn off the code that tries to compute locality information for the splits. This is useless when dealing with S3 since the files are not on the cluster so there is no use in computing locality; and worse yet, it uses a single thread in the driver to iterate over all the files in the restored snapshot. For a very large table this was taking hours and hours iterating through S3 objects just to list them (about 360,000 of them for the our specific table).
3. a flag to override the column family schema setting to prefetch regions on open. This was causing the main executor thread on which a Spark task was running, which was trying to read through HFile's for its scan, compete for a lock on the underlying EMRFS stream object with prefetch threads trying to read the same file, so most tasks in the Spark stage would finish but the last few would linger half an hour or more competing with the prefetch threads alternately for a lock on an EMRFS stream object. This is the only change that had to be outside the mapreduce package as it directly affects the prefetch behavior in CacheConfig.java
4. a flag to turn off maintenance of Scan metrics. this was also causing a major slowdown, getting rid of this sped things up 4-5 times. What I observed in the thread dumps was that every call to update scan metrics was trying to get some HBase counter object and deep underneath was trying to access some Java resource bundle, and throwing an exception that it wasn't found. The exception was never visible at the application level and was swallowed underneath but whatever it was doing was causing a major slowdown. So we use this flag to avoid collecting those metrics because we never used them
I am polishing my patch a bit more and hopefully will attach it tomorrow. One caveat, I tried but struggled with how to write any useful unit/component tests for these as these are invisible behaviors that do not affect the final result at all. And I am not that familiar with the HBase testing standards, so for now I am looking for guidance on what to tests.
Would appreciate any feedback plus guidance on writing tests, provided of course there is interest in incorporating my patch into the main codebase.
Cheers.
----Saad