Yes tree swallows are one of the first to migrate back(soon as insects appear) and I highly doubt they will be evicted unless a starling desides to TRY to fit in the box(which are made too small for them on purpose )
What gear were you using, because with a tripod and around 500mm you can fill the frame(I have years ago) with patience and keeping motionless![]()