Missing mortar, bricks and tiles allow access for cunning nesting creatures, which can cause various problems even if only due to noise.
If you need to exclude mammals, birds or insects, please try to fit artificial habitat to mitigate the loss.
I don’t use poison or kill anything.Active wasp and bee nests are best referred to a specialist.