TL;DR
- Artificial intelligence (AI) and machine learning (ML) features are being implemented on devices
- They could allow the leakage of information through accidental interaction
- They can cause a security control to change its attack surface and risk score.
Introduction
I’m quite a curmudgeon about whether we should report some of the traditional security findings because of a lack of actual risk, something I have written about and even given a talk about. Inside our tech team, I raise up what I call “vulnerabilities that aren’t” where there is little risk in most circumstances.
Sometimes though, things change and where there was little risk before, one comes along. This is a story about an accidental screen press highlighting something to me.
What I thought before AI
Traditionally on mobile devices a screenshot of the active screen of an app will be taken by the window manager and used as a thumbnail in the task manager. Similarly, the user can choose to take a screenshot, where it will be saved to the device’s storage along with photos. It won’t usually get uploaded to the cloud backup, but will be accessible to other apps with the right permissions.
This setting is managed along with other facilities, such as screen mirroring, through either casting to a device or copying the screen buffer, at a frontend task level. On Android this is implemented through the use of window and surface layout flags (WindowManager.LayoutParams.FLAG_SECURE and SurfaceControl.SECURE, hereafter referred to as FLAG_SECURE) that can be applied to a WindowManager Layout which tells the WindowManager that the screen cannot be replicated by an insecure task. This has existed since the very first version of Android.
TL;DR on Android, if FLAG_SECURE is set then it usually cannot be saved as a screenshot.
For the majority of apps this flag is very mehworthy: it really doesn’t matter if a screenshot or thumbnail is taken, or the app is cast to a TV screen. This could even be deemed to be desirable. But, for some apps, like financial or some media, this could cause the leakage of sensitive or copyright material. It is common for financial apps to use FLAG_SECURE on their main layouts, especially if bank number or card number details are seen.
I’ve pretty much always advocated this case and don’t generally regard FLAG_SECURE being missing as a vulnerability, unless the app is sensitive, or it is a requirement.
The story that changed my opinion about screenshots
I use a Pixel 8 as my personal phone, because I prefer the Android ecosystem and UX. Google have jumped on the AI bandwagon hard and updated the assistant to Gemini.
Recently the “Circle to Search” facility has been added into modern Android devices, including the Pixel 8 and Samsung S24. This is a facility where you can long tap the home icon / the swipe up bar and it will go into a Gemini overlay. At this point Gemini will look for interesting stuff on the screen and allow direct interaction, e.g. if it sees what looks like a phone number it will allow the user to make calls or add it to contacts. It can select text and allow a Google search. If you tap on the unused section of the screen, it will then upload the whole screen to Google search.
This is an excellent UX decision – it now makes any Window usable to pick features out of and create shortcuts. This feature does not require the app owners to create shortcuts, as it generates them automatically.
But what if we don’t want the screen uploaded to Google search. What if I accidentally do the long hold, panic and then tap when I’m in the wrong app? This, my dear reader is what happened to me last weekend.
I have a Monzo account that I use as a supplementary account for out of band payments, such as going down the pub, the odd coffee, or music albums. Anyway, I was out and about and just wanted to make sure I had enough funds, so I loaded up the app and unlocked it. My thumb then hovered a tad too long at the bottom of the screen, causing Circle to Search to load. Trying to dismiss it, caused a misplace tap and suddenly I have Gemini telling me about my balance and where my last purchase was.
It was enough to make me think twice about these features. This got me thinking about some other implications this could have on other apps that don’t implement screen protection on sensitive screens.
What does Google say?
According to Google, when you use Circle to Search or similar AI features like Gemini, a screenshot of your screen can be sent to Google for processing. They state that these screenshots aren’t saved to your account or search history unless you actively choose to use the feature (e.g. tapping a result or performing a follow-up action). The screen may still be analysed in order to extract objects, text, and context.
Is this data used to train AI models?
Google claims that personal data is not used for training unless you explicitly opt in via settings (usually through “Help improve Assistant” or similar toggles). That said, it’s not always transparent where your data goes after leaving your device, so the privacy implications aren’t entirely clear.
What about other apps?
Let’s expand this a bit. Imagine that someone doesn’t use a password manager and just keeps their passwords in a spreadsheet in Google Sheets. Now a couple of random taps results in an accidental Google search with the contents of their password spreadsheet in them.
What information could be accessed by the phone’s use? Let’s start with what we can’t do:
- This will not be stored in Gemini, so you can’t ask it to read back information it’s highlighted through Circle to Search
- The screenshot itself will not be visible in your Google history
What we can see:
- The text accidentally searched for will be available in your Google history, which is, by default, recorded.
Fortunately, this isn’t placed in Gemini’s store about you, but it will show in your search history. Image searches aren’t stored in your history so the screenshot of my bank balance will stay on my local phone, but it will show in your search history.
It’s suddenly become a whole lot easier to accidentally spread around information that really shouldn’t be spread around. Now we’re depending on Google’s privacy controls to not distribute that sensitive information to researchers or Google’s admins.
To reiterate from above, Circle to Search will work on any screen that does not have the FLAG_SECURE attribute applied to it, this will include Google docs, contacts, and photos.
Circle to Search on the Android Photos app:
As the device’s user, you have only one option to protect yourself: to disable Circle to Search, which you can do from the Settings app, Display and Touch->Navigation Mode.
Conclusion
Is this Google’s fault? Not really, they have added an effective improvement to their UX; but like any improvement, it can link data in ways that needs to be thought out fully.
This doesn’t mean that this information is automatically sent to Google, but it really makes it a lot easier for a couple of accidental keypresses to send data to places where it really should be.
Is it Monzo’s (or similar app) fault? Yes, and no, I would certainly say that Monzo should use FLAG_SECURE on many more screens. But I’ve used the thumbnail in the past to copy my card details across between different apps on the phone; something I’ve fought with when using other banking apps. This means any fix would have a UX impact, which needs to be a business decision.
Certainly, I’ve rethought when and where I could raise findings about allowing screenshots beyond the old “I’ve accidentally cast my passwords to the office TV screen”.
Risks can and do change over the lifetime of a risk, this may mean something could be considered a risk, then changes and mitigations change the risk level, which is then changed again. This is one of the reasons why revisiting testing is important.
