Well most of my lenses are grey market, they're definitely cheaper than your regular warrantied lenses. I usually check it thoroughly, the manual focus, the AF performance, scratches/junk inside, etc before buying. I've been quite happy so far.
They're usually from the same factory. Nothing to worry about the build quality. A grey market lens is almost always the exact same as the warrantied one you'd pick up at a retail outlet.