Receiving new mail alerts when Outlook is closed

We often see questions regarding getting new mail notifications when Outlook is closed. Ironically sometimes they are requests about how to get them and sometimes they are complaints about how to get rid of them.

So, how do you get these new mail alerts or how do you get rid of them?