I found this article by Dale Cole in the IBM Lotus Notes and Domino wiki.
“In order to enable RSS for mail databases based on the mail 8 template, you need to add ini parameters to the Domino server notes.ini where the mail databases reside. I did this in 8.0.2.”