I'm seeing some bad looking behavior in GroupJoin.
In the constructor, there is:
_outerReferenceTracker = new ReferenceCountTracker<TOuter>(outer);
If the 'outer' list is non-empty, the first line fill _outerReferenceTracker. Then, RebuildAll re-adds everything in _outer. It does not clear _outerReferenceTracker first. The end result is that every item has a reference count of 2.
An unrelated problem is in RemoveOuter, where there's the line:
List<OuterEntry> outersMatchingKey = _keyToOuterLookup[outerEntry.Key];
It appears to me that _keyToOuterLookup is not being updated when an outer item's key changes. I encountered this when I changed the key of an outer item, and later Removed that outer item. RemoveOuter tries to find the key in _keyToOuterLookup, it's not
there, and an exception is thrown.
Even if someone's code doesn't encounter this exception, the fact that items have an extra reference count and key changes are not tracked leads to _outerReferenceTracker and _keyToOuterLookup containing no-longer-used information.