How Information-Seeking Behavior Has Changed in 22 Years

The web has changed dramatically over the last two decades. To understand how user behavior has been affected by these changes, we replicated a 1997 study conducted at Xerox PARC. We asked people to describe a situation where online information significantly impacted their decisions or actions.

We found that, compared to 22 years ago, a larger proportion of current critical internet activities involved finding answers and gathering information in order to better understand a topic. A fair amount of information was acquired in a passive way, without looking for it, during browsing. And often, during critical activities, users turned to other people, asking for their help or opinion.

Methodology

Original Study (1997)

In 1997, Julie Morrison, Peter Pirolli, and Stuart Card conducted a large-scale survey with 3,292 respondents in which they asked people to answer a single question:

Please try to recall a recent instance in which you found important information on the World Wide Web, information that led to a significant action or decision. Please describe that incident in enough detail so that we can visualize the situation.

This question was intended to study important web use, rather than all web use. It’s an example of a UX-research method called the critical-incident technique: participants are asked to recall a significant event that has a certain quality. In this case, the question focuses on web use that influenced actions or decisions that people considered important — such as making a purchase, interacting with a company, or changing an opinion.

The Xerox PARC researchers organized the responses in three different taxonomies:

• the purpose of the activity reported in the incident: Find, Compare/Choose, and Understand
• the method used to find information: Find, Collect, Explore, and Monitor
• the domain of the content that was consumed: Business, Education, Finance, Job search, Medical, Miscellaneous, News, People, Product Info & Purchase, and Travel

Replication Study (2019)

Since 1997, there were at least three major web-related changes likely to influence usage patterns:

1. Many more people access the internet today than in 1997. According to ITU, only 2% of the world’s population had internet access in 1997. This proportion was estimated to be 54% in 2019. In the United States, internet access is even more widespread: according to Statista, 87% of people in the US had access to the internet in 2019, compared with 36% in 1997 (according to Pew).
2. Today, people access the web on a variety of devices — mobile phones and tablets being among the most notable. In 2019, We Are Social reported that people spent 48% of their online time on mobile phones. In contrast, in 1997, virtually all web use was on desktop computers (in our first study of mobile web use in 2000, most users said that phones were not ready for the internet).
3. There are many more services available on the internet today than in 1997 (between the two studies, the number of websites grew from about 1 million to about 183 million — to say nothing of the millions of apps available in the Apple and Android app stores).

These changes mean that the internet is more readily accessible today than in 1997, both in terms of audience size and where and when people go online. It’s only natural to expect that that important information-seeking behavior may have changed as well.

In order to test this hypothesis, we decided to replicate and expand the 1997 Xerox PARC study. We started with the same question that was asked in that study, but, after several rounds of pilot testing, we had to slightly modify the wording of the original question. In 1997, people performed relatively few online information-seeking tasks in their recent past, so it was easy for them to choose a single instance. Today, our respondents were performing hundreds of such tasks in a single day, so isolating one was difficult.

Here’s the question that we used in the final survey:

Please try to recall a recent instance in which you found important information online, information that led to a significant action or decision. Please describe that incident in enough detail so that we can visualize the situation.

If you can recall several such instances, please describe the one that was the most important to you.

(Research purists will note that modifying the wording means that our study wasn’t a true replication of the original. However, an exact replica of the old study wasn’t interesting, because the usual goal of research replication is irrelevant in this case. Usually, studies are replicated to assess the validity of the findings. But, our goal was not to confirm or refute the original — an impossible feat given the 22 years between the studies — but rather to assess the current state of internet use. It’s also possible — even though highly unlikely — that any differences between the two studies are due to the slight variation in wording instead of the time distance between them.)

We followed up this core question with several other questions meant to clarify the context of the reported information-seeking task:

• Satisfaction scores and comments: How satisfied respondents were with the websites or the applications they had used and what changes they would have liked to make, if any.
• Context: When and where the reported instance happened, what device(s) was used, and whether the user interacted with anyone else.
• Importance: How much their decision was influenced by the information they had found online. (We removed respondents who selected Not at all because we wanted to capture only influential information acquisition.)

Our online survey received 750 responses. We collected the data over two separate periods: weekdays and weekend. We split the data collection to avoid time-related biases. (For example, it’s possible that during the week most important tasks are work- or school-related.)

We substantially overrecruited, anticipating that many of the responses would not be detailed enough for coding. We ended up including 498 of the responses in our analysis.

We strictly coded the answers based on the criteria set by the Xerox researchers and came up with new categories only when the answers didn’t fit any existing ones, to ensure that the two studies were comparable. One researcher coded the full dataset and a second researcher coded 100 randomly selected responses. The definitions of new categories were discussed afterwards, and the first researcher recoded all the responses based on the discussion.

We also came up with two additional taxonomies (Social Interaction and Device) that reflected the new web landscape and practices in which web users engage today.

In what follows, we discuss all these different taxonomies (old and new) separately.

Purposes of Web Activities: More Understand Activities

In the original study, researchers grouped important information-seeking behaviors in 3 categories, reflecting the purpose of the activity:

• Compare/Choose: The user evaluates multiple products or information sources to make a decision. For example, one participant compared the prices and features of several pieces of specialized scientific equipment in order to decide which one to purchase.
• Understand: The user gains understanding of some topic. For instance, one participant reported that he decided against starting the Keto diet based on the information that he had found online.
• Acquire: The user looks for a fact, finds product information, or downloads something. For example, one participant looked up the steps to perform CPR. (The original study called this category Find. We chose to rename it, because the Xerox PARC researchers used the same label again elsewhere with a different meaning.)

Our data indicates that today’s web users engage more frequently in Understand information-finding activities and less in Compare/Choose activities than before. In the 1997 study, only 24% of the activities fell in the Understand category; in contrast, in our study this category received most responses (40%). (The difference was statistically significant, p < 0.001).

In 1997, the most popular category, accounting for more than half of the responses (51%), was Compare/Choose. In our study, only 36% of the responses were classified as Compare/Choose. (This difference was statistically significant, p < 0.001.) The third category, Acquire, comprised 24% of responses (similar to 1997, which was 25%).

The increase in Understand responses may be due to the fact that online content has become comprehensive and easy to access due to search improvements. Back in 1997, if you wanted to learn about a topic, you would probably have gone to the library, but today most people would just look it up online.

Why did the proportion of Choose/Compare activities decrease since 1997? This may be an artifact of our coding approach. We found that our survey participants often engaged in activities that fell in multiple categories; for example, a user reported “I researched what could help with psoriasis and then I bought a cream that I found online.” Though the primary purpose was to research the topic, it is likely that he also compared and chose from several options, though a possible comparison between different creams was not mentioned explicitly. Our code captured only the primary purpose that users stated clearly and had a critical influence on their decisions or actions.

So it isn’t that the proportion of Compare/Choose activities has necessarily decreased, but rather that the Understand activities have become more common than in 1997 and that Compare/Choose behaviors are now often an immediate consequence of Understand.

Content Types: Wider Variety of Important Online Content

In agreement with our finding that Understand activities have become more prevalent, we also found that the content that people viewed as critical had become broader, falling into 13 distinct content categories, compared to only 10 in the original study (which were transformed into 9 categories in our classification, after combining 2 of the original categories). In our study, 14% of the responses fell into the 4 new content categories that we identified.

The new content categories were Entertainment, Hobbies & Interests, Home & Families, and Pets. We combined the original Business and Job Search categories into a new category, Work. We also broadened and renamed the Medical category as Health and the Travel category as Planning.

Content Taxonomy and Examples

Overall, Product Information & Purchase was the type of content most often mentioned (30% of all responses), followed by Health (19%). The results didn’t change significantly compared with the 1997 study, when 30% of the responses fell into Product Information & Purchase and 18% of the responses were classified as Medical. Though the Internet has grown a lot, it’s remarkable that these two main content categories still were involved in almost half of reported critical incidents.

In contrast, the Work and People categories saw major drops from the old study to the new. It’s probably not the case that respondents work less today but Work as a percentage of all critical internet use has dropped because there are now so many more options for nonwork use. The drop in People from 13% to 3% is partly (but only partly) compensated by the 7% of cases falling into the new Home & Family category, since some of the family-oriented use might previously have been considered people-oriented.

We observed that certain content types tended to be associated with specific purpose categories. For example, for Product Information & Purchase, 47% of the reported activities fell into the Compare/Choose category, but 75% of the Health responses belonged in the Understand category.

How the Information Was Acquired: The Growth of Passive Information Acquisition

The 1997 study reported 4 different ways in which the information was gathered: two categories (Find, Collect) were active and usually driven by an explicit information need that the user had (e.g., a specific question) and the other two categories (Explore, Monitor) were passive, meaning that the user was not actively seeking out the information. Explore activities referred to finding the information accidentally while browsing the web, without actively looking for it (e.g., a user found out about graffiti on Venice Beach while reading news and went out to clean it), while Monitor activities involved repeatedly going to a website in order to check for new or updated information.

In both studies, the majority of the information was accumulated through active methods, but we found that a larger proportion of the reported instances in the new study involved passive information acquisition (14% versus only 4% in 1997 — the difference was statistically significant, p < 0.05).

We found that Monitor activities were virtually nonexistent in 2019 — possibly because people did not provide us with enough information in order to determine whether they had visited the site repeatedly or not.

Instead, we identified another type of passive information acquisition: notifications (grouped in the new category Notified). Activities in this category were triggered by notifications — received through text and email messages or push notifications on smartphones (e.g., a user reported “Fandango sent me an alert saying that they had an early showing for Shazam, so I told my friends, and we immediately got tickets”). However, Explore activities still had the largest share among the passive information acquisition (83%); only 17% of the reported passive incidents fell into Notified.

Social-Interaction Patterns

In our study, we also asked users whether they interacted with anyone during the reported incident. We divided responses into 6 different categories; this taxonomy was not present in the original PARC study.

Social-Interaction Taxonomy and Examples

28% of respondents reported accompanying social-interaction behaviors while making important decisions based on online information. Inquire was the most reported social-interaction activity (15%); the other types of social interaction were relatively low (under 6%).

Social-interaction patterns also showed a relationship with the type of content involved in the activity. For Product Info & Purchase decisions, though only 20% of instances had accompanying social-interaction behavior, 67% of their social interaction was Inquire. This makes sense — someone trying to make a decision about a purchase may have specific questions for a company representative or an existing customer.

In contrast, for News & Politics, 39% of responses reported accompanying social-interaction activities, but all of them were Share. For example, a respondent saw a blizzard warning in her city and shared the information with her mom: “I talked to my mom on the phone who lives in a different state to let her know not to worry since it was all over the news.”

Device

Another taxonomy that was new to the 2019 study referred to the device used during the reported incident. We defined 4 categories: desktop/laptop computers, smartphones, tablets, and multiple devices.

We found that most of the critical incidents (42%) occurred when people were using their smartphones — critical smartphone use was higher than critical desktop use (p < 0.05), which was somewhat surprising. (This result seems to contradict our finding that the activities done on mobile are rated as less important than those carried out on computers; the reason for this difference likely lies in the fact that smartphones are available at all times and tend to be used more than bigger devices — thus, it’s possible that it was easier for people to remember activities performed on a smartphone than activities performed on a computer.)

Only 6% of respondents relied on tablets to get online information to make important decisions. Multiple-device usage was also common, with 20% of our respondents using 2 or more devices during their incidents.

Different content types tended to be associated with different devices. For example, for the Product Info & Purchasecontent type, the distribution of device usage was relatively even: 39% smartphone, 29% desktop/laptop computers, and 24% multiple devices. In contrast, for the content type Health, 54% of the decisions were based on the information found on mobile phones. Most Work activities (more than 50%) were carried out on desktop/laptop computers.

Conclusion

Information technology has changed dramatically since 1997. But has user behavior? And in particular, has the impact of the internet on our lives changed in any ways? Our study has determined that the Internet has become a primary, influential source of information — the majority of the activities that people carry out on the internet are related to gathering knowledge and understanding a topic. 22 years ago, most important, memorable Internet-related activities involved comparing and choosing among different products or sources of information in order to make a decision. Even though these types of activities are still very present today, they are often a byproduct of educating ourselves and gathering knowledge about a topic. Also interestingly, a good chunk of the information we gather on the internet is passive — we’re not even looking for it, it’s simply delivered to us by virtue of notifications or discovered accidentally during browsing. (The passive information gain is possibly related to the Vortex — as we’re spending more and more time on the internet, jumping from one site to the next, we inherently discover more information that we’re not looking for.)

We also discovered that today, critical-internet usage is often a social activity — it involves more than one person. That’s likely due to the ubiquity of smartphones: everyone is always connected, within one-tap reach, and can quickly chime in with an opinion or with a piece of information whenever needed. (Also, there are many more services now for connecting with other people through technology.) Last, but not least and not surprisingly, many, many critical activities are carried out on phones today. It’s not the case that people will do every type of task on a small screen, but many of the important and memorable tasks are done (or at least partially done) on their phone. The ability to start an activity on one device and seamlessly continue it on another is, thus, crucial for a good user experience.

Given the growing importance of information-gathering activities, a big implication is that sites need to go beyond merely providing alternatives to users and focus on providing good content that educates them. Content is why users come to your site. Even on an ecommerce site, appealing visual design won’t persuade users to purchase — but comprehensive content that clearly and understandably explains product benefits will. Good and credible content can help users gather the information they want quickly and further increase their awareness of your brand.

A final point relates to the concept of critical use, as opposed to average use. Many methods to analyze what users do on a website will be dominated by less-important use cases, since average problems tend to occur more frequently than truly important ones. Thus, if you only go by pure statistics, you may end up designing to support people’s less important needs and provide poor (or nonexistent) support for their most crucial needs. We strongly recommend that you conduct research to gain an understanding of your customers’ most important needs and design with those in mind, even if they are not as frequent.

Reference

Morrison, J.B., Pirolli, P., and Card, S.K. (2001): "A Taxonomic Analysis of What World Wide Web Activities Significantly Impact People's Decisions and Actions." Interactive poster, presented at the Association for Computing Machinery's Conference on Human Factors in Computing Systems, Seattle, March 31 – April 5, 2001. (Warning: link leads to a PDF file.)