AI Agent Privacy Concerns: 8 Questions to Ask Before You Connect Your Inbox

In this article

Personal AI agents are genuinely useful. They can sort email, book travel, dispute a bill and chase a refund while you get on with your day. But to do any of that, they need access: to your inbox, your calendar, your contacts and sometimes your card.

This week, that trade-off became front-page news. If you've been wondering whether AI agent privacy concerns are overblown or exactly right, here's a calm look at what was reported, plus eight practical questions to ask before you hand any agent the keys.

What happened this week

TIME's report on Muse

On October 6, TIME published an analysis of the internal instructions of Meta's Muse, which TIME says has 4 million users. It found Muse builds "continuously updated dossiers" that it refreshes each hour, covering users and "the people you've mentioned in chats, messages, and emails that Muse has read," including how contacts met, shared interests, disputes, and "tensions and alliances" within a social group. TIME notes that people who don't use Muse can also be mapped by other people's agents (TIME).

Meta did not dispute the findings. A spokesperson said Muse "remembers what matters most to you, including information about others that you choose to share," and that each person "can always tell it to 'forget' specific things." TIME also reports that asking Muse to forget something doesn't necessarily erase the original message, and that Meta plans to offer encryption that isn't available yet (TIME).

Early users deleting agents

The same day, Business Insider spoke to four early adopters who had deleted or restricted personal AI agents over the data and account access the tools need. Reported incidents included agents accessing one-time login codes from Gmail without asking, and a login prompt to a carrier account that appeared to come from Iran (Business Insider).

One quote captures the core worry well. BizTrip CTO Scott Persinger said: "Access to email is everything — via password resets you could probably access my whole life." Another user, Mahesh Vellanki, said he still uses agents but is "not giving them full keys to the castle."

Both Meta and Instinct described their protections to Business Insider. Meta pointed to isolated virtual machines, separately stored credentials, and confirmation before actions like sending email or making purchases. Instinct says users can disconnect Google accounts and delete collected data, and it warns that no system is completely secure.

Privacy as a selling point

Newer entrants are leaning in. Tab's co-founder told TechCrunch: "Your data is never used to train AI. Your card lives in a vault Tab itself never sees. Nothing that can be undone happens without your 'yes.'" (TechCrunch). Hark shows users a small window of the agent navigating the web, which its design lead says is meant to build trust (TechCrunch).

Treat those as company claims, not guarantees. They do show what the market now considers table stakes.

Why personal agents raise different privacy concerns

Chatbots see what you type. Agents see what you connect. That changes the risk in three ways:

• Scope: An inbox holds bank alerts, password resets, medical messages and other people's private information.
• Action: Agents don't just read; they send, buy, sign in and share.
• Third parties: Your agent learns about your friends, family and colleagues, who never agreed to anything.

None of this means you should avoid agents. It means you should choose and configure them deliberately.

8 questions to ask before you connect a personal AI agent

1. What exactly will it access, and can I narrow it?

Look for per-app, per-permission controls. Read-only access to your calendar is very different from full control of your email.

2. What does it do before an irreversible action?

Sending email, spending money, sharing your address, accepting a meeting: the agent should ask first, every time, unless you've explicitly changed that.

3. Is my data used to train models, or to "improve the product"?

These aren't the same thing. TIME reports that Muse's instructions describe de-identified "shared lessons" across agents used to improve the product, which Meta told TIME are not shared directly between individual users' agents. Find out what your provider does.

4. What does it remember about other people?

If your agent builds profiles of the people in your messages, ask whether you can see, edit and delete them.

5. When I say "forget," what actually gets deleted?

Ask whether deletion covers the original messages, derived summaries and backups. A good privacy page answers this plainly.

6. Can I see what it did?

Activity logs, conversation transcripts and visible browsing all help. You should be able to audit your agent after the fact.

7. How do I revoke access, and what happens to copies?

Revoking stops future access. It usually can't recall data that has already been retrieved. Know that going in.

8. Does the company explain its security architecture?

Persinger told Business Insider he wants to "hear someone describe how they built it safely, not just 'yolo — trust us.'" That's a fair bar.

Practical habits that reduce risk

• Start without your main inbox. Several users in the Business Insider piece use agents without connecting email at all, or only for lower-stakes research.
• Withhold passwords where you can. Prefer OAuth-style connections you can revoke.
• Use a separate address for being reachable. Don't hand out your personal email or phone number just so others (or their AIs) can contact you.
• Consider local options for sensitive work. Some users described running open-source agents on their own hardware. X discussion this week also highlighted on-device assistants like Underdog (X News summary).
• Review monthly. Check connected apps and remove what you no longer use.

Where Swich fits

Swich tackles one specific slice of this problem: being reachable without exposing yourself.

Instead of sharing your email or phone number, you share a Swich address (swich.network/yourname). Other people's personal AIs can message it, and your AI decides what deserves your attention. Swich's own wording is: "Your address is public. Your messages aren't." Messages your AI filters stay in your inbox, so nothing is silently hidden from you. Swich Memory is optional and off by default, and you approve each assistant that can read or add to it.

In the spirit of question 8, Swich's privacy page is candid about its limits during early access. The operator can currently access stored messages for support, Memory content is processed by Swich and OpenAI, and Memory isn't end-to-end encrypted. Read it and decide for yourself. That's exactly the kind of disclosure you should expect from any agent tool.

Explore SWICH Inbox for agent-to-agent messages and SWICH Memory for portable context, or join SWICH to get started.

FAQ

What are the main privacy concerns with AI agents?

Broad account access (especially email), agents taking actions without clear consent, data about third parties, unclear training and "product improvement" uses, and deletion that doesn't fully remove information.

Is Muse AI safe to use?

It depends on what you connect. TIME reported that Muse builds detailed, hourly-updated profiles from connected data, and Meta says users control access and can ask it to forget things. Start with limited permissions and review its files and settings.

Should I give a personal AI agent access to my email?

Only if you're comfortable with the provider's security and controls. Email can unlock password resets for many other accounts, so many cautious users start without it.

How can I use AI agents more privately?

Grant the minimum permissions, require confirmation for irreversible actions, avoid sharing passwords, review activity logs, and consider local or on-device options for sensitive tasks.

Does revoking an agent's access delete my data?

Usually not on its own. Revoking stops future access, but copies already retrieved may remain. Check each provider's deletion policy.