Health risk
As well as being highly irritating to horse and rider, they also cause difficult behaviour as your horse tries to escape them. Flies also present a health risk as vectors of disease and infections can quickly spread through a yard where flies will go from horse to horse. Fly strike is possible from eggs laid on open wounds then hatching into maggots.
The reality is, once you need to start using fly repellents, there are already flies breeding at your yard.