A friend once asked me, only half joking, whether Netflix was listening through his TV microphone. It isn’t — but the truth is both more boring and more interesting. Recommendation algorithms don’t need your microphone. They have something better: a complete record of what you do. Every pause, skip, replay, and 2am binge is a data point, and the systems built on those data points know your taste better than your friends do.
So what exactly are they collecting? This guide is the full inventory — the behavioral signals, the content features, the contextual data, and the inferences you’d never guess. Not to scare you, but to inform you: you can’t manage what you can’t see. The general mechanics are in how recommendation algorithms work.
Table of Contents
- The Short Answer: Your Behavior, Mostly
- Explicit Signals: What You Tell Them Directly
- Implicit Signals: What Your Behavior Reveals
- Negative Signals: What You Reject
- Content Data: What the Items Themselves Contribute
- Contextual Data: When, Where, and How You Consume
- Inferred Data: What They Figure Out About You
- What They Mostly Don’t Use
- What You Can Actually Control
- Frequently Asked Questions
The Short Answer: Your Behavior, Mostly
The single biggest input to every recommendation system is behavioral data — what you did on the platform. Not what you said about yourself in a profile, not your demographics, but the revealed preferences in your actions: what you clicked, watched, bought, skipped, finished, replayed, saved, and abandoned.
This is a deliberate design philosophy. Stated preferences (“I like documentaries”) are unreliable — people describe their aspirational selves. Revealed preferences (you watched six hours of reality TV last weekend) are honest. The systems learned long ago to trust the behavior over the biography.
Everything below is a refinement of that principle. The data falls into a few big buckets, and understanding the buckets tells you both how the systems work and where your leverage is.
Explicit Signals: What You Tell Them Directly
The rarest and most valuable data — you, deliberately instructing the system:
Ratings. Thumbs up/down, star ratings, hearts. Netflix replaced stars with thumbs specifically because binary signals are cleaner — stars invited grade inflation and cultural differences in what “3 stars” means. A thumbs-down is unambiguous.
Saves and lists. Watchlists, save-for-later, favorites, My List. Adding something to a list is a strong declaration of intent — stronger than a like, because it implies future action.
Follows and subscriptions. Following a creator, subscribing to a channel, following an artist. This is you curating your own feed sources — the system treats it as a durable preference, not a momentary impulse.
Search queries. What you search for reveals intent directly. Searching “sourdough troubleshooting” tells the system more in four words than a month of passive browsing. Search data is also how systems detect emerging interests before behavior confirms them.
Onboarding choices. The “pick three artists you like” screens. Crude but effective seed data for new users.
Explicit signals are gold because they’re intentional — but they’re scarce. Most users rate almost nothing, save occasionally, and search sporadically. The systems can’t survive on explicit data alone, which is why the implicit layer below does most of the work.
Implicit Signals: What Your Behavior Reveals
This is the bulk of the data — everything you do without thinking about the algorithm:
Consumption depth. Did you finish? How much did you consume — 10%, 90%, all of it? Completion is the strongest implicit signal across nearly every platform. A finished 40-minute video outweighs ten clicked-and-abandoned ones.
Dwell time. How long you lingered, even without explicit interaction. On feeds, pause duration on an item predicts interest. You didn’t like it, didn’t save it — you just looked at it three seconds longer than the others. Noted.
Replays and revisits. Watching something twice, replaying a section, returning to a product page three times. Repetition signals strong interest — it’s the behavioral equivalent of underlining.
Velocity and recency. When you did it matters. Yesterday’s binge outweighs last year’s phase. Systems apply time decay to behavioral data — recent actions count more — because taste drifts and the model should drift with it.
Session patterns. Binge sessions vs. sampling sessions, session length, time between sessions. The rhythm of your usage is itself a signal about engagement depth and content preferences.
Scroll and navigation. How far you scrolled, what you hovered over, what you clicked past. On e-commerce, the path through categories reveals intent even when nothing is purchased.
The unsettling part isn’t any single signal — it’s the completeness. The systems don’t sample your behavior; they record it exhaustively. Your taste profile is a high-resolution behavioral portrait, updated continuously.
Negative Signals: What You Reject
What you don’t want is as informative as what you do — sometimes more:
Skips and quick abandons. Skipping a song in ten seconds, closing a video after thirty seconds, bouncing from an article immediately. Fast rejection is a loud signal.
“Not interested” and hides. The explicit negative — rare, because most users never bother, but extremely informative when given. Systems weight these heavily precisely because they’re deliberate.
Scroll-past patterns. Consistently ignoring a category, creator, or topic. The system tracks non-engagement per content cluster — your indifference is mapped.
Smart users weaponize negative signals deliberately — aggressive “not interested” tapping is the fastest way to reshape a feed. The systems are designed to listen; most people just never speak.
Content Data: What the Items Themselves Contribute
Recommendations need data about the items too, not just the users:
Metadata. Titles, descriptions, categories, genres, tags, creators, publishers, dates. The structured facts about each item.
Content features. The actual substance: for text, the words and topics; for music, audio features like tempo and energy; for video, visual and audio analysis plus transcripts. Modern systems extract rich feature representations from content itself.
Engagement history. How the item performed with others: view counts, completion rates, like ratios, velocity. An item’s track record with the crowd informs its treatment with you.
Relationship data. Which items are frequently consumed together, which creators collaborate, which products are bought together. The item graph — how things connect — is itself valuable data.
Freshness. Publication or release dates. New items get exploration boosts; the system needs behavioral data on them, so it shows them to likely-interested users to learn.
This is why a new song with zero plays can still be recommended: the system knows what it sounds like (content features) even before it knows who likes it (behavioral data).
Contextual Data: When, Where, and How You Consume
The same user wants different things at different times, and the systems know it:
Time. Hour of day, day of week, season. Morning news vs. late-night comedy, weekday workouts vs. weekend movies. Temporal patterns are among the most predictive contextual features.
Device. Phone vs. TV vs. desktop vs. smart speaker. Phone sessions skew short-form; TV skews long-form; each device gets tuned recommendations. The device is a proxy for context — couch vs. commute.
Location. Country (for catalog and language), sometimes city-level for local relevance. Location also determines what’s available — licensing differs by region, so the candidate set itself is location-dependent.
Session state. How long you’ve been active, what you’ve already consumed this session, whether you’re binging or browsing. The fifth video in a binge gets different recommendations than the first video of the day.
Network and technical. Connection speed (affects what quality/format to serve), app version, OS. Mostly operational, occasionally informative.
Contextual data is why your feed feels different at midnight than at noon. The system isn’t just modeling you — it’s modeling you-right-now.
Inferred Data: What They Figure Out About You
Here’s the layer most people don’t know about: systems infer attributes you never provided, from patterns in your behavior.
Taste clusters. The system assigns you to latent preference groups — not “likes comedy” but positions in a high-dimensional taste space. These clusters are discovered, not predefined, and they capture subtleties no questionnaire could. They’re also where what AI bias is gets concrete: clusters learned from skewed behavioral data reproduce the skew.
Life patterns. Sleep schedules (from usage hours), work routines, relationship status changes (sudden shifts in viewing patterns), even mood — inferred from what you consume when. None of this is labeled in a profile you can see; it exists as model parameters.
Predicted behavior. Propensity models estimate what you’ll do next — likelihood to churn, subscribe, or purchase — and these predictions feed back into what you’re shown.
This inferred layer is the least transparent and least controllable. You can’t edit an embedding; you can only change the behavior that generates it.
What They Mostly Don’t Use
Myth-busting, because the folklore outruns the facts:
Microphone audio. The major platforms consistently deny using microphone data for recommendations, and the technical and legal costs would be enormous. Your behavior is cheaper and more reliable than your conversations.
Demographic targeting as primary signal. Age, gender, and similar attributes are weak predictors compared to behavior. The systems learned this empirically — two demographically identical users can have nothing in common taste-wise.
What you tell surveys. Platforms occasionally survey users, but stated preferences consistently underperform behavioral data. The systems trust what you do.
The pattern: the systems use what’s predictive and cheap to collect. Behavior is both. Everything else is secondary.
What You Can Actually Control
You have more leverage than it feels like:
Curate explicitly. Rate, save, follow, use “not interested.” Explicit signals are scarce and therefore powerful — a few deliberate actions outweigh weeks of passive behavior.
Manage your history. Review and delete watch/search history periodically. Your history is the training data; pruning it is the most direct way to reshape your profile.
Separate contexts. Different profiles or accounts for different parts of your life — kids, work, guilty pleasures. Mixed signals produce confused recommendations.
Use private/incognito modes for out-of-character browsing. What the system never sees, it can’t learn from.
Check privacy dashboards. Every major platform has one — Google’s My Activity, for example. Review what’s collected, delete what you don’t want kept, adjust ad personalization. And learn how to spot a phishing email while you’re at it — data hygiene extends beyond the platforms themselves.
Remember the exchange. The recommendations are the product; your behavior is the price. That’s not necessarily a bad deal — but it should be a conscious one.
Frequently Asked Questions
Primarily behavioral data: what you watch, click, buy, skip, finish, replay, and save — plus when, on what device, and for how long. They also use content features, contextual signals like time and location, and inferred attributes like taste clusters derived from your behavior patterns.
No — major platforms consistently deny it, and behavioral data is cheaper and more reliable than audio. They don’t need your conversations when they have a complete record of your actions. Every pause, skip, and replay is a data point.
Completion and consumption depth — whether you finished something and how much of it you consumed. Explicit actions like saves, ratings, and “not interested” taps are rarer but weighted heavily. Recent behavior counts more than old behavior due to time decay.
Raw interaction logs typically age out over months to a couple of years, but derived taste profiles persist for the account’s lifetime and patterns learned from your data remain in models indefinitely. Deletion requests under privacy laws remove identifiable data, though removing data from trained models remains technically difficult.
Yes, substantially: use explicit signals (ratings, saves, “not interested”), periodically delete watch and search history, separate contexts with different profiles, use private modes for out-of-character browsing, and review platform privacy dashboards. Your behavior is the training data — curate it deliberately.
