As pointed out, you have to be registered to vote to receive a ballot. Registration is what gets you on the mailing list. If you get purged from the list, no ballot gets sent. Honestly, the biggest difference is that instead of someone at your polling place checking for your name on the list, it's happening when the ballots are being prepared to be mailed.
Also as pointed out, signatures are used to avoid things like double-counting, as are other measures. I wish I kept the link, but there was a recent incident of someone who got two ballots because they updated their mailing address right around the time ballots were being sent out. They reported on social media suggesting that this was insecure and could let someone vote twice, and a worker for their county pointed out that each of those ballots were marked by batch. When an updated ballot is sent like this, the previous ballot is invalidated for that voter and cannot be used to vote. Not only does this avoid double-voting, but it also means that if someone gets that first ballot other than the intended recipient, any attempt to cast a vote with it wouldn't work.
You've got researchers analyzing various voting schemes, and offering suggestions on how to make them more secure. So people are thinking about this more than you might realize. That said, it's not a guarantee that various states/counties will listen, as things like electronic voting are still viewed with skepticism in many research circles, yet here we are. I actually prefer the paper ballots our state uses that are machine scannable. The tech is mature, reasonably cheap, and easy to audit.
In WA, it's possible for me to confirm if my vote was tabulated, which adds transparency to the process. From WA's FAQ on how our mail-in ballots are tabulated:
