I’ve configured Outlook with my Outlook.com account and even though my Email and Calendar gets synched just fine, my contacts don’t.

I’ve got plenty of contacts in the People section of Outlook.com but only a few of them actually sync to Outlook.

How can I get all my contacts to sync with Outlook?