Adulterer (or adulteress) can be used for anyone who has a sexual relationship with a person who is married to another, but again does not always imply the specific social situational that mistress does.

What do you call a guy dating a married woman?

In vernacular, words often used for the illicit lover of a married man include “side chick,” “shack,” “shackup,” and “side piece.” Literary terms for the partner of a married man include “paramour,” “moll,” “courtesan,” “best girl,” “inamorata,” “chatelaine,” and “beau.”