Every manufacturer has pros/cons.
The biggest drawbacks to Oly cameras are:
1. Less dynamic range compared to others
2. More noise at high ISOs (which by the way aflicts all the sony cameras except the 700 as well)
3. Less affective focus tracking than competition and less accurate in low light (without focus assist).
4. Greater depth-of-field at similar Field of View (realize though this is relative and only impacts certain styles of shots)
5. Less lens options - not as many as other brands BUT enough to cover all the bases and the Oly lenses are some of the best in the business so quality is there.
The only reason to worry about the above is if you really take a lot of photos where the above limitations come into play. For certain every other manufacturer has their downsides. I would only recommend a switch to another system if you're photography is adversely affected by the above. For instance sports shooting (focus / high ISO)or wedding work / large scale landscape (dynamic range). Switching to another system is expensive. And you'll find the kit lenses from the other systems aren't as good as the Oly lenses - you can get high quality lenses but you'll have to pay more.
What do you shoot? That will help determine if another system is significantly better for that style of shooting.