In LateralJoin currently the copying of data from left and right inputs is not very efficient. Basically for the number of rows seen in right side (let say n), it does copy row by row across all vectors in left and right inputs. Whereas if it copies each vectors (n times) and then move to next vector that can help in improving the performance.
Also today MemoryManager::update can be called in lateral multiple times for left and right incoming. These calls are found to be very expensive hence make sure it's called only once for every new incoming from both left and right.