How to configure DNS so a mail server and the email it sends pass anit-spam tests

First, make sure that any domain names used in return addresses resolve to an MX record or an A record. For example, if the mail server sends out all mail addressed from user@foo.example, make sure foo.example owns at least an MX record:

foo.example.    IN    MX    10 mail.foo.example.

Next, make sure that the IP address that the mail server sends mail from reverse-maps to a domain name (that is, that the domain name in in-addr.arpa that corresponds to the address has a PTR record attached):

2.0.168.192.in-addr.arpa.    IN    PTR    mail.foo.example.

Check that the domain name that the mail server's address maps to in turn maps back to that address (that is, the domain name has an A record with the same address on the right side):

mail.foo.example.    IN    A    192.168.0.2

Finally, check that the domain name your mail software uses in the HELO or EHLO (extended HELLO) SMTP commands is either the same as the domain name you just checked (mail.foo.example), or else passes the same forward- and reverse-mapping checks. For example, if your mail server announces itself as smtp.foo.example, make sure smtp.foo.example maps to an address, and that address maps back to smtp.foo.example.

Not all mail software performs all of these checks when receiving email, but ensuring that a mail server passes them will help guarantee that the mail it sends won't be refused as spam by the stricter mail servers on the Internet.