POPROX Brings Recommender-Systems Experiments Out of the Offline Lab (and News Recommendations to Users)

In 2023, we reported on a $2 million NSF project led by Joseph Konstan at the University of Minnesota. The project aimed to build shared infrastructure that would enable academic researchers to conduct recommender-system experiments with real users. Three years later, this infrastructure has developed into a functioning service with a user base, research publications, and a fitting name: POPROX.ai.

POPROX stands for the Platform for OPen Recommendation and Online eXperimentation. From a user perspective, it operates as a personalized news newsletter. US subscribers receive a daily selection of Associated Press stories, with recommendations adapted to stated topic interests and reading behavior. From a research perspective, the newsletter provides the interface to an experimental infrastructure on which researchers can deploy recommender algorithms and interface variants to actual readers. The platform has been live since January 2025.

From Offline Evaluation to Online Experiments

A substantial part of recommender-systems research still relies on offline evaluation. Researchers select a dataset such as MovieLens, MIND, or Amazon, create training and test partitions, train a model, and compare performance using metrics such as NDCG or Recall. This workflow has clear advantages in terms of scalability and reproducibility. Its ability to provide evidence about user behavior and long-term effects is more limited. POPROX addresses this gap by providing infrastructure for online experimentation. The project team notes that online experiments and user studies remain comparatively rare in news recommendation, while predictive accuracy continues to dominate evaluation.

The service provides much of the infrastructure that makes online research difficult for individual academic groups. POPROX maintains a pool of consenting participants and obtains current news through the Associated Press. It manages subscriptions, accounts, consent, behavioral logging, and surveys. Researchers can assign participants to experimental recommenders and receive anonymized interaction and survey data. Weekly questionnaires measure dimensions of user experience and news consumption using validated instruments. The project documentation specifies an eventual enrollment target of approximately 10,000 subscribers, although a current figure for active participation does not appear to be publicly available.

Researchers remain responsible for the experimental recommender itself. They provide their own recommendation code and computing resources, expose a POPROX-compatible endpoint, and obtain human-subjects approval at their institution. The current user interface is the daily newsletter, and the platform presently supports relatively simple experimental designs. Access is also primarily oriented toward researchers in the United States. These design choices define the current scope of the platform and allow POPROX to focus on the shared components that are particularly costly to implement independently: recruitment, consent, content delivery, instrumentation, and longitudinal data collection.

Some Familiar Names Behind POPROX

The researchers behind POPROX are particularly noteworthy. The six principal investigators listed by the project are Joseph Konstan and Daniel Kluver from the University of Minnesota, Bart Knijnenburg from Clemson University, Edward Malthouse from Northwestern University, Michael Ekstrand, now at Drexel University, and Robin Burke from the University of Colorado Boulder. The advisory board includes Casey Fiesler, Peter Brusilovsky, Jan Golbeck, Chirag Shah, Julian McAuley, and Ullas Gargi. Together, the group represents substantial expertise in recommender systems, personalization, human-computer interaction, journalism, and information access.

Several of these researchers have played central roles in the development of recommender-systems research. Konstan is one of the main researchers behind GroupLens and MovieLens and a co-founder of the ACM Conference on Recommender Systems. In 2026, he received the ACM SIGCHI Lifetime Research Award. His group has conducted recommender-system experiments with users for several decades. Ekstrand maintains LensKit, one of the established open-source toolkits for recommender-systems research, and has worked extensively on evaluation, reproducibility, and fairness. Burke has contributed to research on hybrid recommendation, multi-stakeholder recommendation, and fairness. Knijnenburg’s work on user-centric evaluation has contributed methods for studying recommender systems beyond offline predictive performance.

Building an Operational News Recommender

This combination of expertise is reflected in the design of the platform. POPROX provides more than an interface for supplying a ranking function. Its reference recommender uses a pipeline built on LensKit. The current system combines signals from reading histories with explicit topic preferences. Its developers started from NRMS trained on Microsoft’s MIND dataset and adapted the model to continuously arriving AP content. This development process exposed several limitations of existing news-recommendation research. Models based heavily on static item identifiers are difficult to apply when most articles are new. Public datasets often provide different metadata from production news feeds. Explicit preferences collected during onboarding have also received comparatively limited attention in the research literature.

The POPROX team discusses these issues in a paper titled “What News Recommendation Research Did (But Mostly Didn’t) Teach Us About Building A News Recommender.” The paper is particularly relevant because it documents the transition from benchmark-oriented research to an operational news recommender. Building a live system required the researchers to address onboarding, preference elicitation, cold-start behavior, longitudinal engagement, survey participation, and interactions between recommendation components. These aspects are difficult to represent in static offline benchmarks. POPROX therefore serves both as experimental infrastructure and as a source of empirical evidence about the requirements of operational recommender systems.

There is already evidence that the infrastructure can support longitudinal research. A study presented at SIGIR 2026 used POPROX for a five-week experiment with 120 US news readers. The researchers studied topic and locality calibration together with personalized news-preview nudges. Algorithmic interventions increased both exposure diversity and consumption diversity, while the effects of LLM-generated presentation nudges were more mixed. The study illustrates the type of evaluation that POPROX can support: controlled algorithmic interventions, repeated exposure over time, behavioral measurements, subjective measurements, and sufficient duration to observe changes in user behavior.

Related Work: From Software Libraries to Living Labs

POPROX is part of a much longer effort to make recommender-systems research easier to conduct under realistic conditions. It is useful, however, to distinguish several types of infrastructure. As discussed in our collections of recommender-system software libraries, Recommendations-as-a-Service systems, and competitions, shared tasks, and living labs, these systems remove quite different barriers. A software library helps implement an algorithm. A RaaS helps operate a recommender. A dataset supports reproducible offline evaluation. A living lab provides access to interactions with actual users. POPROX is interesting because it combines several of these layers.

Software Libraries and Simulation

Software libraries have substantially lowered the engineering cost of recommender-systems research. LensKit (Paper; Website; GitHub) and RecBole (Paper; Website; GitHub) provide implementations, data handling, evaluation procedures, and experimental tooling. POPROX itself uses LensKit in its reference recommendation pipeline. Such libraries make it much easier to develop and compare algorithms, but the resulting experiments usually remain offline unless researchers separately operate a service and recruit users.

Simulation provides another approach. Google’s RecSim and RecSim NG (RecSim Paper; RecSim NG Paper; RecSim GitHub; RecSim NG GitHub) model sequential interactions between recommender systems and simulated users. This is useful when studying reinforcement learning, feedback loops, long-term effects, or ecosystem dynamics that are difficult to evaluate with static datasets. The price of simulation is the user model itself: conclusions depend on how well simulated responses approximate human behavior. POPROX approaches the same general problem from the opposite direction by providing access to observed behavior from actual users.

MovieLens: A Research Recommender with Real Users

One important predecessor is also closely connected to the POPROX team. MovieLens (Paper 1; Paper 2; Website) has operated as a non-commercial movie recommender at the University of Minnesota for decades. MovieLens is famous for its datasets, but its importance as an experimental system is sometimes overshadowed by the popularity of those datasets. GroupLens has used the live service for field studies on recommendation algorithms, interfaces, explanations, user control, and many other questions. One study, for example, allowed MovieLens users to select among different recommender algorithms and observed their choices in the operational system.

MovieLens is highly relevant to POPROX because it demonstrates the scientific value of maintaining a service that users genuinely want to use. It also illustrates the difficulty of reproducing such an environment elsewhere. MovieLens is operated by GroupLens and its experiments are primarily conducted by GroupLens researchers and collaborators. It is not a generic living-lab endpoint to which any researcher can independently submit a recommender. POPROX attempts to make part of this type of experimental capability available as shared infrastructure.

Docear: Research-Paper Recommendations in a Production Application

Docear (Paper 1; Paper 2; Website; GitHub) followed a related model in the scholarly domain. The academic literature suite included a research-paper recommender that generated recommendations for its actual users. The system supported experiments with recommendation algorithms and produced several datasets containing recommendation and interaction data. The Docear desktop application was open source, although the recommendation server itself was operated by the project and was not released as an open research platform for arbitrary third-party experiments.

Docear is therefore another example of a researcher-operated production system. It demonstrated that substantial recommender-system experiments can be embedded into software that provides independent value to its users. The limitation, from the perspective of the wider community, was institutional: access to the users and experimental infrastructure remained largely with the team operating the service.

CLEF NewsREEL and plista

One of the most relevant predecessors in the news domain was CLEF NewsREEL (Paper 1; Paper 2; Website). From 2014 to 2017, its NewsREEL Live task allowed research teams to provide recommendations to users of actual news websites through plista’s Open Recommendation Platform. Incoming requests, newly published articles, recommendations, and subsequent interactions were exchanged in near real time. NewsREEL also offered a Replay task based on recorded interaction streams, thereby connecting online and offline evaluation within the same infrastructure.

NewsREEL is probably the closest historical predecessor to POPROX in the news domain. Both give academic researchers the possibility of evaluating algorithms against interactions with users of an operational news service rather than only against a static dataset. There are also substantial differences. NewsREEL was organized as a shared evaluation campaign around publisher traffic supplied through plista. Participation and live evaluation were tied to the campaign infrastructure. After the CLEF editions ended, NewsREEL continued at MediaEval with offline evaluation, while the live component disappeared. POPROX instead aims to maintain a persistent participant population and experimental service.

RecSys Challenge 2017: Recommendations Sent to More Than 900,000 XING Users

The ACM RecSys Challenge 2017 (Paper; Website) provides another instructive example. The challenge, organized with XING, studied job recommendation under cold-start conditions. It first ran a conventional offline phase. The best-performing teams could then participate in an online phase in which submitted recommendations were delivered through XING’s production system. Eighteen teams eventually rolled out recommendations to more than 900,000 users.

This was unusually strong experimental access for an academic recommender-systems competition. It was also temporary and selective. The online phase was offered to the top 25 teams from the offline phase rather than to every researcher who wished to conduct an experiment. The challenge shows both the scientific value of industry-supported online evaluation and the difficulty of making such access persistent. POPROX attempts to institutionalize the latter rather than negotiate it separately for each competition.

RecSys Challenge 2024 and EB-NeRD: Production Data Without a Living Lab

The ACM RecSys Challenge 2024 is useful as a more recent comparison. The challenge centered on news recommendation using Ekstra Bladet’s EB-NeRD dataset (Challenge Paper; Challenge Website; Dataset Website; GitHub). EB-NeRD contains data from more than 2.7 million users, more than 600 million impression logs, and more than 120,000 news articles. It provides unusually rich data from a real production news environment.

The evaluation itself was nevertheless offline. Participants predicted clicks for logged impressions and submitted rankings for a fixed test set. This distinction is useful when positioning POPROX. Access to large-scale industrial interaction logs greatly improves offline research, but it does not allow a researcher to change the recommender and observe what subsequently happens. POPROX provides a smaller experimental population but supports interventions. These two forms of evidence answer different questions.

Mr. DLib and the RARD Datasets

Related ideas were explored in academic recommendation through Mr. DLib (Paper 1; Paper 2; Website; GitHub). Mr. DLib started as an open-source Recommendations-as-a-Service system for scholarly literature and later added a living lab. Experimental recommenders could be connected through a REST API and evaluated with real users of the JabRef reference manager. Mr. DLib’s A/B engine determined whether a request was handled by its internal recommender or by an external research partner. During the first 16 months, the living lab delivered more than 1.8 million recommendations.

Mr. DLib was explicitly open to external research partners, although participation was not an anonymous self-service process. A research partner needed to operate a compatible REST API, return suitable documents within the required latency, and contact the project team for integration. This is an important distinction when discussing “open” living labs: the opportunity was available to researchers, but access still depended on technical compatibility and coordination with the infrastructure operator.

Mr. DLib is also interesting because its online operation produced reusable offline resources. The first Related-Article Recommendation Dataset, RARD (Paper; Dataset Description), contained 57.4 million delivered recommendations. RARD II (Paper) expanded this to approximately 94 million recommendations across an item space of 24 million documents. The datasets preserved recommendation logs, algorithm configurations, and interaction data from an operational recommender. They illustrate a useful lifecycle for research infrastructure: an online system can produce experimental results while it is operating and valuable offline datasets after the live service has ended.

CORE Recommender: Recommendations-as-a-Service for Scholarly Repositories

CORE Recommender (Paper 1; Paper 2; Website) provides another relevant model. CORE operates a recommender for open-access scholarly literature that can be embedded into repositories, journals, and other services. By 2023, more than 70 repository systems had registered to use the recommender, and CORE recommendations had also been deployed on arXiv. The service therefore demonstrates how a central recommendation infrastructure can provide recommendations across many independent information systems.

CORE is closer to Recommendations-as-a-Service than to a general living lab. Repository operators can integrate the CORE recommender, but arbitrary researchers do not thereby gain the ability to upload a new algorithm and expose it experimentally to CORE’s distributed user population. Its relevance to POPROX lies in the service architecture and in the benefits of centralizing corpus management and recommendation infrastructure. POPROX adds an explicit experiment-broker role.

Darwin & Goliath and the RaaS Perspective

A related service-oriented approach was Darwin & Goliath (Paper; Website), presented at RecSys 2019. It proposed a white-label Recommendations-as-a-Service architecture in which multiple recommendation frameworks and algorithms could be deployed for different client organizations. Automated algorithm selection was intended to identify suitable configurations for each client and even individual recommendation requests. Darwin & Goliath was less directly an academic living lab than NewsREEL or Mr. DLib, but it addressed a related infrastructure problem: recommender systems are costly to build, deploy, monitor, and evaluate repeatedly, and much of this engineering can be centralized.

Our RaaS overview contains several other services, including Recombee, bX, BibTip, and commercial e-commerce recommenders. They are useful context, but they are less direct predecessors of POPROX. A conventional RaaS allows an organization to add recommendations to its own product. A research living lab additionally has to expose experimental control, assign treatments, collect scientifically useful feedback, manage participants or production traffic, and support research governance. The latter requirements are the more unusual part of POPROX.

LL4IR, TREC OpenSearch, STELLA I, and STELLA II

The living-lab idea also developed within information retrieval. CLEF Living Labs for Information Retrieval, LL4IR (Paper), and the subsequent TREC OpenSearch track (Website; GitHub/Data) connected experimental ranking systems with real search services. The basic concept was similar to online recommendation experiments: participating systems supplied rankings, while the production environment supplied queries and user interactions.

This line of work was subsequently developed further by STELLA – Infrastructures for Living Labs (Paper 1; Paper 2; Website; GitHub), developed by TH Köln with GESIS and ZB MED. STELLA supports experimental retrieval and recommendation systems packaged as microservices and provides mechanisms such as A/B testing and interleaving. At CLEF LiLAS 2021, the infrastructure was used for live evaluation in LIVIVO and GESIS Search. STELLA II continued the infrastructure with an emphasis on continuous evaluation and deployment at additional information providers.

STELLA remains particularly relevant in 2026. The open-source infrastructure continues to be developed, and STELLA is being used for recommendations in GESIS Search and for live evaluation in services such as the MLentory machine-learning model registry. A July 2026 study used STELLA for continuous online comparison of recommendation strategies in GESIS Search and found that semantic-similarity recommendations performed better than the tested lexical and session-based alternatives. STELLA therefore demonstrates that persistent online evaluation infrastructure can survive beyond a single shared-task campaign.

Here again, open-source infrastructure should not be confused with unrestricted access to production users. Anyone can inspect and deploy the STELLA software, but experiments with users of GESIS Search, LIVIVO, MLentory, or another integrated information service depend on the corresponding provider and experimental arrangement. The software layer is open; access to a particular population is a separate resource.

arXivDigest

Another close example is arXivDigest (Paper; Website; GitHub), introduced at CIKM 2020 as a living lab for personalized scientific-literature recommendation. Users receive personalized recommendations for recent arXiv papers based on interest profiles and can provide explicit feedback. The back end operates as a broker between users and experimental recommendation systems. External systems can access the living-lab API and upload recommendations.

Conceptually, arXivDigest is close to POPROX because the experimental infrastructure is attached to a useful recommendation service. Users have a reason to participate beyond contributing to an experiment: they receive recommendations for new research papers. Researchers obtain a setting in which recommendation methods can be evaluated online. The main differences are the domain, scale, instrumentation, and the additional participant-management and survey infrastructure provided by POPROX.

Informfully

More recently, Informfully (Paper 1; Paper 2; Website; GitHub) has developed into an open-source platform for conducting recommender-system user studies through mobile and web interfaces. Researchers can distribute algorithmically selected text, images, audio, and video, configure experimental conditions, record detailed interaction histories, and embed questionnaires. A 2026 article in ACM Transactions on Recommender Systems describes Informfully explicitly as a free research resource for conducting online studies of how recommendations affect consumption behavior.

Informfully has already supported longitudinal news studies and has also released the IDEA dataset containing articles, recommendations, interaction histories, ratings, survey responses, and data from hundreds of study participants. Its division of responsibilities differs from POPROX in an important respect. Informfully provides the software and user-facing research application, while researchers normally recruit and enroll their own participants and provide the content for their study. POPROX additionally maintains a participant pool and supplies a continuously updated Associated Press content stream.

Access to Software Is Not the Same as Access to Users

The history of these systems also illustrates that “open” can describe several different properties. LensKit, RecBole, RecSim, Informfully, and STELLA provide open-source software. EB-NeRD and RARD provide research data. NewsREEL and LL4IR provided access through specific evaluation campaigns. In the RecSys Challenge 2017, only the best-performing teams from the offline phase were invited to the XING online evaluation. Mr. DLib accepted external research partners whose systems satisfied its API and latency requirements, but integration required coordination with the operators. MovieLens supports extensive online experimentation, but primarily for GroupLens and its collaborators. STELLA’s software is public, while access to live users depends on the participating information provider. Informfully can be used freely by researchers, but researchers generally bring their own participants.

This distinction is central when assessing the practical value of POPROX. Code for an A/B-testing framework is relatively easy to distribute. A persistent population of consenting users is much harder to distribute. So are licensed content, ethics procedures, account management, reliable delivery infrastructure, experimental randomization, longitudinal logging, and validated surveys. A platform may therefore be entirely open source while still leaving most of the difficult work of an online user study to the researcher.

Where POPROX Fits

Seen in this broader context, POPROX combines ideas that have previously appeared in different forms. MovieLens and Docear demonstrated the scientific value of researcher-operated recommendation services with genuine users. NewsREEL and the 2017 RecSys Challenge demonstrated that external algorithms can be exposed to production traffic. Mr. DLib combined Recommendations-as-a-Service with a scholarly living lab and subsequently released large interaction datasets. CORE showed how recommendation infrastructure can be shared across many independent information services. STELLA developed reusable infrastructure for integrating experimental systems into operational search and recommendation environments. arXivDigest combined a user-facing recommendation service with a broker for external experimental systems. Informfully provides much of the software required for longitudinal user studies.

POPROX combines a persistent participant pool, licensed and continuously updated content, longitudinal recommendation, behavioral logging, survey instruments, and externally supplied experimental recommenders in one service. This combination is uncommon. Many earlier infrastructures have disappeared, changed scope, existed only for the duration of a shared task, restricted live experiments to selected teams, or require researchers to recruit their own users. Maintaining the participant population may therefore be at least as important as maintaining the technical API. Without users, even a well-designed living lab eventually becomes another offline dataset.

Methodological Boundaries

The platform also has methodological boundaries that should be considered when interpreting results. POPROX users are US adults who voluntarily subscribe to an experimental news newsletter. Newsletter-based news consumption represents a specific interaction setting and differs from environments such as TikTok, Google Discover, newspaper homepages, or streaming services. AP content also represents a particular news ecosystem. The POPROX team has reported low engagement with weekly surveys during the initial deployment. Shared infrastructure improves comparability across studies, but it also creates common assumptions and sampling conditions. Results obtained through POPROX should therefore be interpreted within the context of this specific real-world environment.

POPROX as a News Service for Users

The service also has value beyond its role as research infrastructure. For participants, POPROX provides a personalized mechanism for navigating a continuous stream of news. Such systems have become increasingly relevant as the volume of available information has grown far beyond what individuals can inspect directly. Recommender systems now mediate access to news, entertainment, products, scientific literature, and many other forms of information. A system such as POPROX provides an opportunity to study how this mediation can account for relevance while also considering diversity, user control, explicit interests, and long-term information consumption.

News recommendation is particularly relevant in the current information environment. Readers face a continuous supply of stories from news websites, social platforms, aggregators, newsletters, podcasts, and video services. Selecting a useful subset is itself an information-retrieval problem. A personalized news recommender can reduce this selection effort while helping users follow topics that match their interests. At the same time, news recommendation raises questions about exposure diversity, calibration, exploration, and the balance between established interests and information outside a user’s usual consumption patterns. POPROX provides both a service to readers and an experimental setting in which these questions can be examined empirically.

Why POPROX Matters for Recommender-Systems Research

For the recommender-systems community, the research value is substantial. Large technology companies can routinely conduct randomized experiments involving hundreds of thousands or millions of users. Most university researchers cannot. This asymmetry influences which research questions are feasible in academia. Offline datasets make the development and comparison of ranking models relatively inexpensive. Research on satisfaction, control, diversity, fairness, interface design, longitudinal effects, and changes in user behavior requires considerably more infrastructure. POPROX reduces this asymmetry by making controlled online experimentation available to a broader group of researchers.

The potentially most important contribution of POPROX may therefore be methodological rather than algorithmic. Moving from statements such as “our model improves NDCG by 0.7%” toward evidence that compares recommendation strategies through repeated interaction with real users would strengthen the empirical basis of recommender-systems research. Offline evaluation will remain essential for model development and reproducible comparison. Large production datasets such as MIND and EB-NeRD add another important layer of evidence. Simulation provides yet another. Infrastructure such as POPROX complements these approaches with the possibility of controlled interventions and direct observation of their effects on users.

The POPROX website summarizes the motivation rather succinctly: “Can’t let the industry platforms have all the fun.” For recommender-systems research, providing academics with comparable opportunities for controlled online experimentation is a useful objective. For users, the same infrastructure provides a personalized way to access news. The combination is what makes POPROX particularly interesting.