Double new email and calendar notifications on Windows 8

For some of my new emails, I receive multiple new email alerts. These alerts come at the same time or within seconds from each other.

When I check in Outlook, I can see that I only received the message once. I also have no rules configured in Outlook to display a New Mail Alert for certain messages.

How can I get rid of these double alerts?