As someone who thought it'd be a good idea to run their own mail server, I found that without SPF/DKIM/DMARC your mail gets identified as spam a lot. Having DMARC just means you get told about it.
The problem comes if the sender has a DMARC reject policy, and he or she sends e-mail to a mailing list which then reflects the mail to a recipient which honors DMARC, the recipient will bounce the e-mail message, and never see it. In some cases this will cause the recipient to be automatically unsubscribed from the mailing list.
If you're not using DMARC, and you're not running a mailing list, then yes, it won't affect you. But it's no urban legend.
Running without DKIM/DMARC exposes your domain to a suite of spoofing attacks that aren't mitigable by SPF alone. A spammer can send mail "from" your domain and over time this activity may lessen the reputation of your domain and/or IP/netblock—even if it's just Gmail users manually marking such messages as spam—making your subsequent outbound, legitimate messages more likely to be flagged. If nothing else, the backscatter gets annoying! DKIM, DMARC, and ADSP are worth setting up for these reasons, and the reports provided by e.g. Postmark[0] are invaluable for understanding how your domains are being abused.
Are you on IPv4 or IPv6? I've found that IPv6 is much more likely to have problems with deliverability than IPv4. When my own server was IPv4-only, I never had problems, even with only an SPF record. When I enabled IPv6, I discovered that I couldn't get through to Comcast users without DMARC, and I also had to request my own /64 from the VPS provider, since the default single address supplied was part of a blacklisted /64 thanks to others. Unless you can get a squeaky-clean /64, forget about it.
