Am I crazy to think that it makes sense to use the same size filter and then use a step-down / step-up ring for different lenses?

Any pro's/cons?