Macrophotography generally presumes a 1:1 magnification ratio, which would project a full size image onto the image sensor. It at least would require a 1:2 magnification ratio, which would project a half-size image.
What matters is how much of a person's mouth you want to appear in each frame. My guess is that, at most, you'd want a single tooth accompanied by the image of half a tooth either side. But for general shooting, you'd want 3 or 4 teeth, possibly including one or more dental instruments, and even one or more of the dentist's hands.
Historically, this type of photography was accomplished with a
35mm film SLR and a 100mm lens, but it would be operated by the dentist, not by a third party observer.
I'd say that you'd need a high resolution image sensor, at least 20MP for an APS-C body, or 40MP for a 'Full Frame' body, in order to capture the detail you want. You'd also need a long macro lens. Sigma has a 150mm macro lens with a magnification ration of 1:1 and a minimum focus distance of 15 inches. It's also stabilized which will help when shooting hand-held. (The longest lenses available that are capable of 1:1 magnification are Tamron's and Sigma's 180mm lenses, but they're not stabilized.) You should also get a ring light or ring flash to supplement the available light, or you'll need to use large apertures which will limit the depth-of-field you'll get. If you need to get farther away, you could use a 1.4X teleconverter, but I would avoid using a 2X teleconverter if you want sharp photos. I'd also avoid using close-up lenses for the same reason.
All this gear is available for rent at
LensRentals.com, so you can rent it all to see how it works for you.