I came across a case where a cached query is referenced on both sides of a join, but the InMemoryRelation is inserted on only one side. This case occurs only when the cached query uses a (Hive-style) view.
Consider this example:
Note that the optimized plan does not use an InMemoryRelation for the right side, but instead just uses a Relation:
The fragment does not match the cached query because AliasViewChild adds an extra projection under the View on the right side (see #2 below).
AliasViewChild adds the extra projection because the exprIds in the View's output appears to have been renamed by Analyzer$ResolveReferences (#1 below). I have not yet looked at why.
In a larger query (where cachedata may be referred on either side only indirectly), this phenomenon can create certain oddities, as the fragment is not replaced with InMemoryRelation, and the fragment is present when the plan is optimized as a whole.
In Spark 2.1.3, Spark uses InMemoryRelation on both sides.