I have come across this very same situation with my bluebird houses. I even watched a pair of house sparrows "gettin busy" on top of the house. Too bad for them they were evicted from the house several times, before they finally got the hint. Eventually they moved on, and the bluebirds moved in.
"I thought that because fewer wolves meant more deer, that no wolves would mean hunters' paradise. But after seeing the green fire die, I sensed that neither the wolf nor the mountain agreed with such a view."
Aldo Leopold