Microsoft’s Recall

[dada_dave]

As @Nycturne said in the AI thread this topic could derail the AI thread as it has so many different paths so I thought I’d create a new thread specifically about it:

Recall: Stealing everything you’ve ever typed or viewed on your own Windows PC is now possible.

Photographic memory comes to Windows, and is the biggest security setback in a decade.

doublepulsar.com

The overwhelmingly negative reaction has probably taken Microsoft leadership by surprise. For almost everybody else, it won’t have. This was like watching Microsoft become an Apple Mac marketing department.

Ouch.

And an earlier post by longhorn:

Recall: random notes about the security model – random blog

threedots.ovh

 

[casperes1996]

But at the same time it’s not like windows has historically tried that hard to isolate process from each other anyway and at leash it used to be fairly trivial to read memory from another process

 

[dada_dave]

But at the same time it’s not like windows has historically tried that hard to isolate process from each other anyway and at leash it used to be fairly trivial to read memory from another process

Absolutely, but I think the key point is here:

But if a hacker gains access to run code on your PC, it’s already game over!

A. If you run something like an info stealer, at present they will automatically scrape things like credential stores. At scale, hackers scrape rather than touch every victim (because there are so many) and resell them in online marketplaces.

Recall enables threat actors to automate scraping everything you’ve ever looked at within seconds.

During testing this with an off the shelf infostealer, I used Microsoft Defender for Endpoint — which detected the off the shelve infostealer — but by the time the automated remediation kicked in (which took over ten minutes) my Recall data was already long gone.

It’s the ease with which hackers are able to scape all this data that’s the issue. Sure more complicated methods can get often get the same info but now everything is all packaged neatly up for them. Nice little bow and all.

 

[Nycturne]

Yes. The approach taken with Windows’ security model of “If they get user level permissions, oh well, sorry” is depressing. I know folks scoff at Apple’s seemingly arbitrary approaches to security, but at least Apple seems to get that we’re in a world today where local apps shouldn’t be run in trusted contexts. It’s pretty reasonable to expect modern apps on macOS to be fully sandboxed and not be able to see or touch anything outside it’s container without user interaction. It’s not perfect coverage, because it’s still possible to ship apps without it enabled which is where you get your trojans, but it helps.

And to top it off, UAC bypasses are still very much a thing. Windows keeps riding that line between “this legacy thing shouldn’t be too noisy, so it won’t notify about elevation”, and “we wouldn’t want you to mistakenly install that weird thing you just downloaded”.

But yes, fundamentally my issues with Recall boil down to these things:

1) It creates a central repository of screen captures and annotations generated from those captures using ML models. Very tasty target that persists long enough to be at least as valuable as scraping your browser cache. This may be the new top target for scraping as it rolls out to users.
2) Microsoft’s behavior around Windows of late is user ambivalent at best, and getting quite hostile on average. Expecting good behavior 3-5 years from now is a stretch, IMO. At some point, I wouldn’t be surprised if engineering gets asked to find a way to train the ML on the user data to improve the features because of XYZ reason, creating more concerns in the process.
3) Microsoft talks about security, but then trips over itself trying to explain all the caveats around the type of data this captures which could be problematic (and which makes it an even more tempting target), on top of the general security issues Windows already has baked in.

That said, I think how Edge might be blocking screen captures when using a private tab is by marking the private tab as “DRM’d” somehow (i.e. marking the content as requiring HDCP). I wonder if other browsers can do the same? Maybe apps in general can sabotage the feature?

 

[casperes1996]

That said, I think how Edge might be blocking screen captures when using a private tab is by marking the private tab as “DRM’d” somehow (i.e. marking the content as requiring HDCP). I wonder if other browsers can do the same? Maybe apps in general can sabotage the feature?

If so that would limit the abilities to use specific monitors/cables and screen casting with private Edge tabs.... That'll go down well

 

[Nycturne]

If so that would limit the abilities to use specific monitors/cables and screen casting with private Edge tabs.... That'll go down well

It's a guess. It's either that or some new private API that Edge uses just for private tabs. Seeing as sites that require HDCP also triggers it, it's reasonable to guess that they used the same mechanism for both to save the time of creating the API.

Is HDCP still a big compatibility problem on Windows? About the only place where HDCP even matters to me is on the Apple TV, so I am a bit out of the loop on the landscape there.

 

[casperes1996]

It's a guess. It's either that or some new private API that Edge uses just for private tabs. Seeing as sites that require HDCP also triggers it, it's reasonable to guess that they used the same mechanism for both to save the time of creating the API.

Is HDCP still a big compatibility problem on Windows? About the only place where HDCP even matters to me is on the Apple TV, so I am a bit out of the loop on the landscape there.

It's a good guess. It apparently also works for other Chromium browsers, not sure if they do the same.

It's not that long since I last heard someone complain that Netflix didn't work properly on their PC and it was because of HDCP.

 

[Nycturne]

It's a good guess. It apparently also works for other Chromium browsers, not sure if they do the same.

That makes me think there very well could be another mechanism then, as I can’t find any reference to Incognito mode or the like triggering HDCP. Now I wonder what it actually is.

EDIT: It might actually be the WindowDisplayAffinity setting, since that can be used to prevent screenshots of your window from being taken: https://github.com/akinbicer/screen-capture-protector

It's not that long since I last heard someone complain that Netflix didn't work properly on their PC and it was because of HDCP.

Fair enough. I’m not a fan of it, but it is what it is.

Although with laptops being more popular these days, and the display chain being fully integrated there unless you are hooking up to a TV or the like, things should be less twitchy on the first machines getting this feature.

 

[dada_dave]

A positive step:

Microsoft backtracks on new Recall feature — enhancing Recall's security and making it an opt-in decision

Lock it down, the public said of Recall, and Microsoft listened.

www.tomshardware.com

I mean it’s a little odd that this wasn’t the case from the start, but this is better.

 

[dada_dave]

It’s baaaack!

Microsoft will try the data-scraping Windows Recall feature again in October

Initial Recall preview was lambasted for obvious privacy and security failures.

arstechnica.com

 

[casperes1996]

It’s baaaack!

Microsoft will try the data-scraping Windows Recall feature again in October

Initial Recall preview was lambasted for obvious privacy and security failures.

arstechnica.com

Guide to introducing something people aren't going to like.

Step 1)
Announce a version of the feature that has even more things in it that people will hate than what you actually want to do

Step 2)
React to all the negative criticism by pulling back and saying "We made a mistake. We'll go back to the drawing board"

Step 3)
Re-introduce the feature, this time the way you initially wanted it to work, removing the additional "features" you added to cause intensified anger, making it seem like a huge improvement.

Step 4)
Hope the above steps took long enough that people think it's boring to keep hearing about and that the "It's better than before at least" drowns out the "But it's still bad".

 

[Nycturne]

Guide to introducing something people aren't going to like.

Step 1)
Announce a version of the feature that has even more things in it that people will hate than what you actually want to do

Step 2)
React to all the negative criticism by pulling back and saying "We made a mistake. We'll go back to the drawing board"

Step 3)
Re-introduce the feature, this time the way you initially wanted it to work, removing the additional "features" you added to cause intensified anger, making it seem like a huge improvement.

Step 4)
Hope the above steps took long enough that people think it's boring to keep hearing about and that the "It's better than before at least" drowns out the "But it's still bad".

I think this grants more foresight to the folks working on this feature than they actually have.

Microsoft historically has been more of a bottom-up engineering org, and I think the first version was absolutely the thing they wanted to ship. It’s more that the designers and project managers involved couldn’t see why this feature is something people don’t want, mixed with a mandate to do AI features.

My time at the larger tech orgs showed me that there’s a lot of weird culture at these orgs which can really shape the minds of college hires in ways that I don’t think are great. You also get types like a recent boss of mine who was on board with NFTs and Metaverse because of the opportunity to make it a personal side hustle. Also was looking at ways to make ChatGPT into a sort of tutor for kids which is kinda scary knowing how prone LLMs are to bad output. It’s been a weird couple of years for me being in one of these orgs, not gonna lie.

I honestly think and economics + ethics classes should be mandatory when getting a tech degree. Give folks better tools to smell a grift when it’s right in front of them.

 

[Yoused]

Announce a version of the feature that has even more things in it that people will hate

I think you misspelled "bug" there.

 

[dada_dave]

I think you misspelled "bug" there.

Speaking of bugs, the option to uninstall Recall entirely is apparently a bug!

Microsoft says its Recall uninstall option in Windows 11 is just a bug

Recall won’t be available until October for Windows testers.

www.theverge.com

 

[throAU]

Yes. The approach taken with Windows’ security model of “If they get user level permissions, oh well, sorry” is depressing. I know folks scoff at Apple’s seemingly arbitrary approaches to security, but at least Apple seems to get that we’re in a world today where local apps shouldn’t be run in trusted contexts. It’s pretty reasonable to expect modern apps on macOS to be fully sandboxed and not be able to see or touch anything outside it’s container without user interaction. It’s not perfect coverage, because it’s still possible to ship apps without it enabled which is where you get your trojans, but it helps.

So much this.

For all the bitching about the iOS security model and how it is becoming more of a thing on macOS as well, at least Apple are aware of and acknowledging the fact that we connect to a very hostile internet, full of people who's sole purpose in life to try and rip you off.

Microsoft seems to be more driven by "look at this shiny new feature!" with ZERO thought as to how much of a honeypot it may be, or how to secure it. This is a company culture that appears to go back decades at this point, and I'm not sure it is fixable now, especially now their product has become dependent on the additional revenue stream generated by advertising inside of paid products.

I'm out - I'm done.

The only copies of Windows I run now are inside purpose built virtual machines purely for windows Active Directory domain administration. They get none of my personal data at all. e.g., all my home Windows VM has in it is a VPN connection to work and a saved RDP connection.

 

[dada_dave]

Microsoft's Recall not exactly opt-in since disabling breaks the interface:

Removing Windows Recall breaks File Explorer in latest 24H2 update

It looks like Microsoft's Recall push is back in full swing with Windows 24H2. A recent technical analysis revealed that disabling Recall also disables the modern File Explorer interface. Currently, it is unclear whether this is a bug or intended behaviour, but it's indicative of Microsoft's...

www.notebookcheck.net

 

[Cmaier]

Microsoft's Recall not exactly opt-in since disabling breaks the interface:

Removing Windows Recall breaks File Explorer in latest 24H2 update

It looks like Microsoft's Recall push is back in full swing with Windows 24H2. A recent technical analysis revealed that disabling Recall also disables the modern File Explorer interface. Currently, it is unclear whether this is a bug or intended behaviour, but it's indicative of Microsoft's...

www.notebookcheck.net

It’s such a microsoft thing to have multiple different interfaces at the same time.

 

[Yoused]

It’s such a microsoft thing to have multiple different interfaces at the same time.

That is, i think, at least in part because Windows was glued together from a parts bucket of clutter, and while they have striven to make it coherent, it just cannot seem to escape its eclectic heritage. Also, the will not let up on using UI colors designed to make your eyes bleed.

 

[dada_dave]

Microsoft Recall screenshots credit cards and Social Security numbers, even with the "sensitive information" filter enabled

Despite promising to filter personal data out, Recall still captures it.

www.tomshardware.com

While not surprising giving the difficulty level, Recall cannot censor all sensitive information in places a user might enter it still making the Recall database a juicy target for hacking even if the filters are on.