How much of an issue it is depends on how much you need to have to have pictures without visible dust. Most times, it can be removed in post-processing by a bit of cloning, being most visible is areas of sky or similar low detail parts of pic. It can be quite annoying to find that after a lens change, the sensor picked up a speck right where you don't need it. I find that it is more prevalent in the winter when humidity is very low. I hesitate to swap lenses then.
This doesn't ease my fears. I thought one of the nice things about digital vs. film was you didn't have to deal with dust and hairs getting on your prints. I may never take my lens off.