Erik I think you would be better served if you used m4/3rd lenses and get rid of the 4/3rd ones. But. If I were you I would consider the E5. When I had mine it seemed the images were sharper and almost 3D looking compared to the m4/3rd cameras and also the IS seemed to work better. Used bodies are everywhere and prices have dropped. The E5 is rugged and would last you many years. Unless the size is an issue.
Either way I would put all my eggs in one basket and use the lenses made for the system without adapters.