New Tools for Studying Digital Interactions
How can researchers better study what people do online—not just what they say they do?
Join us on May 27th at 3 p.m. Eastern for a workshop on new tools for studying digital interactions. You can register here.
Co-hosted by Boys & Men Online and the Psychology of Technology Institute, the webinar will feature four short presentations followed by audience Q&A.
Speakers include:
Alexandra Rodman & Varun Mishra, Northeastern University — UbiWell Connect, a custom passive data collection tool developed for the longitudinal Connect Study.
Fiona Baker, SRI — The ABCD Study’s approach to digital data collection, including the EARS app and Fitbit integration.
Matthew Brown, University of Chicago — The use of SharpSports to measure sports betting spending and lessons learned about collecting data on web browsing.
Richard Landers, University of Minnesota — QUAIL, an open-source tool that collects data from LLM conversations inside Qualtrics surveys.
The trouble with asking people what they do online
How many times did you pick up your phone yesterday?
I guessed 30. My screen time app says 84. Yikes.
I’m not the only one off by a wide margin. In one study tracking 23 users over two weeks, participants estimated 36 pickups per day. The actual average was 85.
And it’s not just young people. In another study, over half of parents of young children underestimated their phone use. Among parents who said they checked their phone fewer than 10 times a day, the real average was 52.
You might assume we all underestimate, but a 2021 meta-analysis of 49 comparisons between self-reported and logged use found that discrepancies were evenly split between underreporting and overreporting. Rarely were they accurate; only three of the 49 studies had mean self-reported use within 5% of the logged mean. And the association was even weaker when researchers asked about problematic use.
It’s hard to accurately recall and estimate phone use, especially when we’re asked about use that can invoke shame, like gambling and pornography.
And yet, much of the influential research on social media, online gambling, gaming, and pornography still relies on self-reported use. Media coverage of these studies rarely distinguishes between what people say they do and what they actually do. Much of what we think we know about technology use is still based on asking people to remember and report their behavior.
Everybody Lies
Surveys are blunt instruments, and humans are unreliable narrators. In Everybody Lies, Seth Stephens-Davidowitz catalogs several examples where survey responses don’t match actual behavior, including:
Self-reported sex frequency and condom use imply billions of condom uses per year. Actual condom sales tell a different story.
Surveys and Facebook self-disclosure put the share of gay men at 2–3%, with sharp differences between tolerant and intolerant states. Google and pornography search data put the figure closer to 5%, with the state differences largely disappearing.
People routinely mislead pollsters and researchers. They especially lie in surveys about sex, drugs, and money. There is no reason to think they suddenly become honest when asked about their phones.
Surveys remain essential for understanding attitudes, beliefs, intentions, and self-perception. But when the question is behavioral — what people actually do online — we need better tools that allow researchers to go beyond mere exposure. Two hours on YouTube could be guitar lessons, outrage bait, help fixing an appliance, or falling into a sports betting rabbit hole. The same app, duration, and device can tell completely different stories.
Better tools for measuring what people actually do
A new generation of research tools is making it easier for researchers to observe digital behavior more directly, and to pair that observation with well-timed questions about motivation, mood, and context.
Some tools measure basic patterns of phone and app use: pickups, duration, app switching, notifications, and time of day. Others go further. Apps like EARS can log keyboard activity, allowing researchers to study not only how long someone spends in an app, but the content and tone of digital interactions. Other systems monitor traffic through an encrypted VPN, giving researchers a fuller picture of browsing and app use, including sessions in private browsing mode.
Most of these tools support ecological momentary assessments (EMAs): short surveys delivered at the moment they are most relevant. A researcher can trigger a survey every time a participant spends two consecutive hours on a given app. Or a survey might be triggered by a specific word pattern detected through keyboard logging. Instead of asking someone on Friday how they felt while scrolling on Monday night, researchers can ask closer to the moment itself.
AI interviews and richer qualitative data
Large language models also open new possibilities with AI-led interviews and focus groups. Platforms like Conveo and Glaut allow researchers to design in-depth video interviews conducted by an AI agent with a large number of participants. These tools can elicit more nuanced responses than a standard survey and also analyze the participant’s voice, expression, and emotional tone.
The AI agent can ask follow-up questions, show images or videos, and explore why a participant responded in a particular way. These platforms do not replace human qualitative research, but they allow researchers to conduct nuanced interviews at a scale that was previously impractical.
Better measurement, new risks
These tools promise better data, more precise measurements, and a clearer understanding of how digital technologies shape behavior and well-being.
They also present new ethical considerations, including serious questions about privacy, consent, disclosure, data security, and the responsibilities of institutional review boards.
Monitoring web traffic, logging keystrokes, or triggering surveys based on sensitive behavior can produce valuable research. It can also present ethical risks if participants do not fully understand what is being collected and how it will be used.
On May 27 at 3 p.m. Eastern, we will discuss the opportunities, challenges, and real-world lessons from researchers using these tools to study digital interactions. I hope you can join us by registering here.