I'm having the same problem here with a Google Apps email but should work exactly as a Gmail account. I have successfully configured the same email address in the Menu->My Settings->Control Panel and I have verified that I can send and receive emails with the account via the user interface. However, when I try to use the same credentials in the Menu->Administrator->Mail Server Settings then save the settings I do not get any server notifications.
The way I have been testing it is to create a different user with a different email address and trying to have that account do a password recovery. When I do that I get "SMTP Error: Could not authenticate. SMTP Error: Could not authenticate." on the top of the page. I looked in the data/CRM_Roundcube/log directory for logs and the only entry I see is the successful email that was sent from the User email test I did previously. It doesn't look like the server itself is logging in and sending the emails correctly whereas the Roundcube client works just fine. I have tried both smtp.gmail.com:587 and smtp.gmail.com:465 and received the same error above. I have telnetted into the smtp server address and did EHLO and STARTTLS (in the port 587 case) and get good results (ready to start TLS).
Is there another place to search for logs? Does the server 'Mail Server Settings' take a different path when trying to send out notification emails?
I appreciate any help you can give. I've been trying a bunch of different combinations all day today but I'm happy to work with you guys if there's something funky going on here. The server is a VPS and I have full root access to the machine so if I need to tweak anything it's not a problem.
-Alan