Perhaps instead of only modifying player positions you modify the positions of all moving objects?