Yea not a good idea; simply because Microsoft Account uses it in password resets. Or for two factor auth. And two factor auth is a good idea.
Plus, the only thing it's used for is clearly stated;
Complete the SMS challenge required during the purchasing process. We do not use mobile phone numbers collected during the SMS challenge for any other purpose unless you choose to submit that mobile phone number as part of your contact information.
You don't have to look for an option to turn it off, you are the one choosing to turn them on.
And contact information doesn't have phone number as a mandatory field. Oh, AWS does exactly the same thing during signup.
If you want to check how your profile is configured, you can do so here