Totally understand, once you get into one of their ecosystems it's hard to break out of that for something different. It also makes sense to turn off mics on devices you're not using.
As far as the fan goes, I recently installed a ceiling fan and it has a remote and all but I never use it, for devices like that I just replace the light switch with a smart switch, then just add a voice control command for it. I do this for all of my lights as well, just replacing the switches so I don't ever have to fuss with bulbs, etc.