It seems you have fixed the problem - a good result, with some logic behind it.
I have a small supplemental question, though I suspect the answer may be lost in the detail of the changes that were tried.
Just my curiosity.
The starting pattern is clear, as you've described.
With the firewall rule turned off (email notifications working) -
- Camera sends SYN
- Google responds with SYN, ACK
- Camera sends ACK
With the firewall rule turned on (email notifications not working) -
- Camera sends SYN
- (no SYN, ACK reply)
- Two instances of retry with no reply
In your wireshark interpretations above the good result would be following on from the camera doing a DNS lookup to find the Google IP address.
Do you have a recollection if the camera was using your router IP address as the DNS setting, or was it set for an internet IP address?
Is your new 'DNS Allow' rule set to use TCP or UDP (or both)?
In the bad result above - do you know if the SYN that was not acknowledged was to the Google IP address, or something else?