What you are talking about is called the Cush-drive.
The heads of the allen-bolts fit exactly in the rubbers of the cush-drive, allowing for some shock absorption.
Consequently, these bolts only face shear-forces, no pushing or pulling.
The fact that the nut was not up to spec is not good, but could have caused very little damage.
Perhaps that offers some peace of mind