Ok, so a few points to make here:
Regarding performance:
You're likely to see a performance loss here for the following reasons:
-
Browsers will cache the images from Gravatar.com or YourServer.com so anything you do will only affect first load
- Gravatar.com itself is cached downstream by ISPs so there's a chance you won't even need to pull all the way from the Gravatar servers
- Gravatar itself has good performance - many large sites use it without performance issues
- Loading the images from another domain (in this case gravatar.com) will be done in parallel to calls on your local domain by most modern browsers, so you might actually slow things down by adding more calls to your server which are done serially after a certain number of connection slots are taken (8+ depending on your browser).
It's pretty hard to imagine seeing a performance gain from this. Maybe if you tiled a big collection of gravatar images into one larger png and used CSS to manipulate which ones showed, you'd have fewer server calls and might see a small performance gain, but man that sounds like a huge complexity hit for marginal gains at best.
Regarding SEO:
There shouldn't be an SEO issue with this, since search engines won't know that the image you serve locally is actually a Gravatar image. Though I suppose some search engines might consider using Gravatar a bit of a bonus as it could "legitimize" you in some way, though this is entirely speculative and should not be a factor in your decision making.