From reproducible evidence to safer and more useful recommendations: nine new ACM TORS articles

We are pleased to announce nine newly accepted articles in ACM Transactions on Recommender Systems. The collection reflects a field that is broadening its understanding of progress. Recommendation accuracy remains central, but it is increasingly studied alongside reproducibility, user control, data quality, conversational interaction, multimodal evidence, diversity, and deployment conditions.

Newly accepted papers

[1] “Towards Mitigating Unwanted Recommendations with Risk-Controlling Recommender Systems”
Giovanni De Toni, Erasmo Purificato, Emilia Gómez, Andrea Passerini, Bruno Lepri, Cristiann Consonni.

[2] “Reproducibility in Recommender Systems: A Survey”
Alan Said, Alejandro Bellogín.

[3] “Illoominate – Data Shapley Values for Debugging Neighborhood-based Recommender Systems”
Barrie Kersbergen, Olivier Sprangers, Bojan Karlas, Maarten de Rijke, Sebastian Schelter.

[4] “Heterogeneous Graph Transfer Learning for Diversity-enhanced Cross-Domain Sequential Recommendation”
Gengyu Zhan, Zitao Xu, Xiaoqing Chen, Xueqi Li, Weike Pan, Zhong Ming.

[5] “Recommender Systems in Industry: Applications, Challenges, and the Academia–Industry Gap”
Dietmar Jannach, Li Chen.

[6] “Conversational Recommender Systems Using Generative Models (Gen-CRS): A Literature Review”
Ahmadou Wagne, Thomas Kolb, Ashmi Banerjee, Fatemeh Nazary, Julia Neidhardt, Yashar Deldjoo.

[7] “RAG-VisualRec: An Open Resource for Vision- and Text-Enhanced Retrieval-Augmented Generation in Recommendation”
Ali Tourani, Fatemeh Nazary, Yashar Deldjoo.

[8] “Informfully Research Platform – An Open Source Project for Conducting Empirical Research with Recommender Systems”
Lucien Heitz, Julian A. Croci, Madhav Sachdeva, Abraham Bernstein.

[9] “Exploring Personality-Aware Explanations for Recommender Systems”
Gabrielle Alves, Dietmar Jannach, Luan Soares de Souza, Marcelo Manzato.

Reproducibility across algorithms, resources, and user studies

Reproducibility is the direct subject of the survey by Said and Bellogín [2], but it is also embedded in the resources and research practices presented by Kersbergen and colleagues [3], Tourani and colleagues [7], and Heitz and colleagues [8]. Together, these papers show that reproducibility requires more than publishing source code. It also depends on explicit protocols, inspectable data transformations, reusable infrastructure, documented evaluation settings, and sufficient information to understand how each result was produced.

Said and Bellogín examine all 51 papers accepted to the ACM RecSys Reproducibility Track between 2020 and 2025 [2]. They describe how the track developed from an initial focus on reproduction and replication into a broader venue that also includes benchmarking studies, research resources, methodological contributions, and reflective work [2]. Participation grew substantially over this period: the track accepted two papers in 2020 and fourteen in 2024 [2].

The survey documents stronger practices around artifact availability, transparency, and experimental reporting [2]. It also identifies opportunities for further methodological expansion because reproducibility studies still tend to draw on a concentrated set of datasets, algorithms, metrics, and evaluation protocols [2]. Many studies do not merely repeat an earlier experiment unchanged; instead, they test the original claims with additional models, criteria, datasets, or application settings [2]. Reproducibility therefore emerges as a cumulative scientific activity through which results are examined under carefully documented variations.

Several papers in this collection put these principles into practice. Tourani, Nazary, and Deldjoo release a configurable pipeline for multimodal retrieval-augmented recommendation [7]. Their resource includes interaction data, generated movie descriptions, trailer-derived representations, fusion procedures, retrieval settings, quality-control results, and optional language-model re-ranking [7]. This makes it possible to inspect the intermediate stages of the experiment rather than seeing only the final recommendation scores.

Tourani and colleagues also address a challenge that is becoming increasingly important in generative recommender systems: generated content may itself become part of the experimental data [7]. Their LLMGenQC component audits synthetic descriptions for semantic alignment with available metadata, internal consistency, safety, and basic sanity conditions [7]. The corresponding scores and acceptance labels are released with the generated artifacts [7]. Reproducing the study therefore involves examining not only the ranking pipeline but also the synthetic evidence supplied to it.

RAG-VisualRec covers 9,724 MovieLens titles, of which 3,751 were aligned with available trailers [7]. It supports textual and visual features, as well as optional audio information [7]. In the reported experiments, canonical-correlation-based fusion improved Recall@10 by as much as 27% over unimodal or simple-concatenation alternatives and almost doubled catalogue coverage [7]. LLM-based re-ranking generally improved nDCG by refining the retrieved candidate pool, particularly when textual evidence was limited [7]. Because the individual stages are exposed, later studies can investigate whether gains originate from metadata enrichment, multimodal fusion, retrieval depth, user-profile construction, or re-ranking.

Heitz and colleagues address reproducibility at another stage of the research process: interaction with users [8]. Their Informfully platform supports the distribution of text, image, audio, and video recommendations through web and mobile interfaces [8]. Researchers can create experimental groups, configure questionnaires, connect external recommendation algorithms through a JSON-based exchange format, and record access times, consumption duration, ratings, feedback, navigation, and other behavioral signals [8].

Experiment configurations can be exported and relaunched with new participants [8]. This makes it possible to repeat not only an algorithmic calculation but also the conditions under which recommendations are presented. The distinction matters because interface design, item placement, timing, and presentation may influence user behavior even when the underlying ranked list remains unchanged [8]. Informfully has already supported four news-recommendation field studies conducted between 2020 and 2023 in Switzerland, Germany, and the Netherlands, as well as two offline benchmarking studies [8].

In the paper’s comparison with existing research tools, Informfully stands out through its combination of web and mobile deployment, support for four media modalities, a modular architecture, questionnaires, localization, responsive design, and an open codebase [8]. It therefore provides infrastructure for the empirical work encouraged by Said and Bellogín: studies that can be documented, repeated, and compared across settings [2].

Kersbergen and colleagues contribute reproducibility at the level of data diagnosis [3]. They implement their KMC-Shapley method in the open-source Illoominate library and evaluate it on public and proprietary interaction datasets with millions of observations [3]. The method exploits the structure of neighborhood-based recommenders to estimate Data Shapley values without repeatedly retraining the model for every individual data point [3]. By combining a scalable method, public implementation, explicit computational assumptions, and concrete debugging tasks, the work exemplifies the relationship between methodological clarity and reusable research artifacts.

Jannach and Chen add an important contextual dimension to this discussion [5]. Reproducible experiments become more informative when the task, objectives, and constraints correspond to a meaningful application [5]. Their call for closer cooperation between academia and industry complements the methodological analysis by Said and Bellogín [2], the multimodal benchmark by Tourani and colleagues [7], the user-study infrastructure by Heitz and colleagues [8], and the data-debugging library by Kersbergen and colleagues [3]. Shared artifacts are particularly valuable when they help researchers study realistic objectives, user interactions, and operational conditions.

Safety begins with the data and continues through the recommendation list

Safety and reliability are addressed at several stages of the recommendation process. De Toni and colleagues provide statistical control over unwanted items in the final recommendation list [1]. Kersbergen and colleagues identify historical interactions that contribute positively or negatively to recommender behavior [3]. Wagne and colleagues discuss grounding, catalogue fidelity, controllability, and safety in generative conversational systems [6]. Tourani and colleagues audit generated item information before it enters retrieval and re-ranking [7].

De Toni and colleagues propose a model-agnostic framework based on conformal risk control [1]. The method calibrates a filtering threshold so that expected exposure to unwanted recommendations remains within a specified risk level [1]. By extending conformal methods from coverage guarantees to more general loss functions, the authors formulate unwanted recommendation as a measurable risk rather than an informal design aspiration [1].

The framework is evaluated in two real-world settings with different forms of negative feedback [1]. KuaiRand represents short-video recommendation and provides explicit “Not Interested” signals [1]. Yambda-500m represents music streaming and contains dislike-related behavioral information [1]. The use of both datasets recognizes that undesirable recommendations are not expressed through a single universal action. A direct report, brief dwell time, skipped item, or disliked song may all indicate dissatisfaction, but they occur in different interfaces and carry different behavioral meanings.

Filtering risky items can produce shorter recommendation sets, protecting the user while reducing the available choice [1]. De Toni and colleagues address this issue by selecting previously consumed items that satisfy a safety heuristic, merging them with the filtered candidate pool, and re-ranking the result [1]. The procedure can retain the intended recommendation-set size while respecting the calibrated risk constraint [1]. Across the tested recommendation models and datasets, the approach enabled a controllable reduction in unwanted content with limited computational overhead [1].

Kersbergen and colleagues approach reliability from the training-data side [3]. Their motivating cases show how unexpected interaction patterns can produce undesirable recommendations even when no single observation appears obviously incorrect [3]. A user may shop for several people in one session, a bot may generate unnatural browsing sequences, or a rare co-occurrence may connect products that should not be recommended together.

The scale of the industrial setting illustrates the difficulty of detecting such cases manually. Bol offers approximately 51 million products from more than 45,500 external partners to 26 million active users [3]. The paper discusses an incident in which sensitive adult products became associated with children’s toys because the products appeared together in historical browsing sessions [3]. A conventional response would be to add a filtering rule after the problem has been observed. KMC-Shapley instead helps identify the interactions that contributed to the behavior, enabling a more systematic debugging process.

The authors evaluate their method on session-based and next-basket recommendation tasks using public and proprietary datasets with millions of interactions [3]. The resulting Data Shapley values identify observations with a notable effect on recommendation performance [3]. The values can support dangerous-product detection, low-quality data identification, dataset pruning, and sustainability-oriented recommendation [3]. The authors also explore whether importance values computed for neighborhood-based recommenders can serve as proxies for data importance in neural models [3].

The work by De Toni and colleagues and Kersbergen and colleagues provides complementary safeguards. De Toni and colleagues control risk in the list that reaches the user [1]. Kersbergen and colleagues help determine why the model learned a problematic association [3]. A practical workflow could use output-level monitoring to limit exposure, data-importance analysis to trace the source of an issue, and subsequent recalibration to verify that the revised system remains within the intended risk level.

Safety also intersects with the generative systems examined by Wagne and colleagues [6] and Tourani and colleagues [7]. Wagne and colleagues identify grounding, catalogue fidelity, controllability, bias, and safe response generation as core requirements for conversational recommendation [6]. Tourani and colleagues provide a related safeguard by auditing generated descriptions before they are used in retrieval [7]. Across these contributions, safety becomes a sequence of inspectable steps: examine the training data, validate generated evidence, control recommendation risk, and evaluate the behavior of the complete system.

Generative recommendation needs evidence, interaction, and evaluation

Generative models appear in three complementary roles across the new articles [6,7,9]. Wagne and colleagues survey their use as the foundation of conversational recommender systems [6]. Tourani and colleagues use generative models to enrich item information and re-rank multimodal candidates [7]. Alves and colleagues use generative AI to vary the linguistic framing of recommendation explanations [9].

Wagne and colleagues review generative conversational recommender systems published from 2018 through 2025 [6]. They organize the literature into three analytical layers: architectural and methodological foundations, knowledge and data foundations, and evaluation methodologies [6]. This structure makes it possible to examine not only which model generates a response, but also what information the system uses and how the resulting behavior is assessed.

At the architectural level, the review distinguishes unified, modular, and agentic systems [6]. It also covers strategies for adapting foundation models and for generating recommendations and conversational responses [6]. At the knowledge level, it examines structured catalogues, knowledge graphs, unstructured and multimodal content, short-term preferences, long-term profiles, persona representations, and dialogue histories [6]. At the evaluation level, it considers task effectiveness, conversational quality, efficiency, trust, ethical concerns, evaluator types, evaluation methods, and the stakeholders represented in the assessment [6].

The review describes a transition from classical conversational systems based on intent detection, attribute slots, manually designed dialogue states, and separate ranking modules toward mixed-initiative systems that maintain context, retrieve heterogeneous evidence, invoke tools, and generate recommendations within a shared interaction loop [6]. This enables more flexible preference elicitation and more natural conversation. It also increases the number of components whose behavior must be validated.

Wagne and colleagues emphasize that conversational fluency and recommendation quality are distinct dimensions [6]. A system may produce a coherent answer while mentioning unavailable items, overlooking catalogue constraints, weakening collaborative signals, or generating an unsupported explanation [6]. Their evaluation framework therefore places grounding, catalogue fidelity, controllability, safety, efficiency, trust, and stakeholder effects alongside recommendation accuracy and dialogue quality [6].

Tourani and colleagues operationalize several of these requirements in a multimodal movie-recommendation setting [7]. Their approach retrieves candidates using enriched textual and visual item representations rather than relying only on a language model’s internal knowledge [7]. Configurable fusion methods—including concatenation, principal-component analysis, and canonical-correlation analysis—make the contribution of different modalities experimentally visible [7]. An optional LLM stage then re-ranks retrieved candidates instead of producing an unconstrained list of movie names [7].

This design connects grounding with reproducibility. Wagne and colleagues argue that generative recommenders should be grounded in validated catalogues and evidence [6]. Said and Bellogín emphasize transparent experimental practices and inspectable artifacts [2]. Tourani and colleagues provide a system in which the catalogue, generated descriptions, multimodal features, fusion operations, retrieved candidates, quality-control results, and re-ranking stage can all be examined [7].

Alves and colleagues study a more focused use of generative AI: controlling how an explanation is written while keeping the recommendation process fixed [9]. In a controlled music-recommendation study, 348 Brazilian participants completed a Big Five personality inventory and evaluated recommendations presented with six explanation styles [9]. Five styles reflected linguistic framings associated with the Big Five traits, while the sixth used a feature-based explanation [9].

The language model successfully generated explanations with distinguishable linguistic patterns [9]. Explanation framing influenced perceptions including persuasiveness, clarity, transparency, and perceived recommendation accuracy [9]. Matching the wording to a participant’s dominant personality trait, however, did not consistently improve perceptions [9]. Emotionally stable wording performed well across several user profiles, suggesting that contextual and emotional appropriateness may be more effective than direct personality mirroring [9].

The result offers a useful perspective on personalization. Generative models make it practical to produce many explanation variants [9], but the availability of these variants does not determine which one best supports the user. Their effects must be examined empirically.

This observation links the work of Alves and colleagues [9] with the evaluation framework of Wagne and colleagues [6] and the research infrastructure of Heitz and colleagues [8]. Wagne and colleagues call for evaluations that include trust, conversational quality, and behavioral outcomes [6]. Alves and colleagues demonstrate that wording affects users’ perceptions even when the recommended item is unchanged [9]. Heitz and colleagues provide a platform through which researchers can present alternative recommendation or explanation conditions and record how participants respond [8].

Generative recommendation therefore emerges as a combination of evidence management, system architecture, communication design, and empirical evaluation. The model must generate suitable language [6,9], but the system must also retrieve valid items [6,7], preserve constraints [6], expose its evidence [7], support repeatable experiments [2,7,8], and communicate in ways that users find understandable [9]. The language model may have the final word in the interface, but it should not have the only word in the evaluation.

Beyond accuracy: diversity, deployment, and external validity

Several papers examine objectives that cannot be summarized by a single offline accuracy score. Zhan and colleagues balance accuracy with diversity in cross-domain sequential recommendation [4]. Jannach and Chen discuss the business, technical, and societal objectives that shape deployed recommender systems [5]. De Toni and colleagues add unwanted-content risk to the optimization problem [1]. Heitz and colleagues support studies of actual user behavior [8], while Alves and colleagues demonstrate that presentation affects user perceptions independently of the underlying recommendation algorithm [9].

Zhan and colleagues introduce category-aware cross-domain sequential recommendation [4]. Their HGTL model constructs a heterogeneous graph containing users, items, and categories from different domains [4]. Connections are expanded using semantic similarity and category co-occurrence, while user-, item-, and category-oriented meta-paths guide information aggregation [4]. This reduces dependence on overlapping users as the sole bridge between domains and allows item and category relations to contribute to knowledge transfer.

The model combines item-level sequential preferences with category-level preferences [4]. An item-category attention mechanism accounts for the fact that one item may belong to several categories [4]. A personalized gating mechanism determines which source-domain information should be transferred [4]. This is useful because indiscriminate transfer may introduce redundant or irrelevant signals even when the source domain contains valuable information.

The diversity-enhanced HGTL-D variant adds personalized maximal marginal relevance re-ranking [4]. It estimates a user’s inclination toward diversity from the category entropy of the interaction history and adjusts the relevance–diversity balance accordingly [4]. Users with broad historical interests can receive more exploration, while users with concentrated preferences are not assigned the same diversity level.

Across three real-world datasets, HGTL improved accuracy over the compared baselines and supported extension to multi-domain scenarios [4]. HGTL-D further increased diversity while retaining competitive accuracy [4]. In the transfer from CDs to movies, the method improved ILAD@10 by 1.07% over the strongest diversity-oriented baseline and increased long-tail coverage by 4.20% [4]. In another setting, HGTL-D improved the underlying HGTL model by 4.94% in nDCG@10 and 3.79% in HR@10 [4]. These results show that accuracy and diversity need not form a fixed trade-off when transfer and re-ranking are personalized.

Jannach and Chen place such multi-objective design within the wider landscape of deployed recommender systems [5]. They survey applications in e-commerce, media, tourism, news, education, finance, health, social platforms, human resources, and other domains [5]. The intended purpose of a recommendation varies across these settings, as do the provider’s business model and the interests of users, item providers, platforms, and society.

Industrial recommender systems may be evaluated through revenue, retention, satisfaction, discovery, supplier outcomes, latency, scalability, catalogue dynamics, maintenance effort, and long-term value [5]. A method that improves an offline ranking metric may also alter training cost, inference time, system complexity, or the ability to respond to changes in user preferences and item availability [5]. Jannach and Chen consequently advocate a system-oriented perspective in which algorithms are evaluated as components of an operational service.

Their call for closer academia–industry collaboration is reflected in several papers in the collection. Kersbergen and colleagues combine public experiments with proprietary e-commerce data and production-motivated debugging cases [3]. De Toni and colleagues evaluate risk control in video and music settings with different forms of negative feedback [1]. Tourani and colleagues release a resource for an emerging multimodal task [7]. Heitz and colleagues support end-to-end field experiments [8]. Alves and colleagues evaluate generated explanations with participants rather than assessing them only through linguistic properties [9].

Said and Bellogín provide a methodological foundation for this agenda [2]. Their survey documents stronger norms around transparency and artifact availability while encouraging greater variation in datasets, algorithms, tasks, and evaluation practices [2]. The other papers demonstrate several routes toward that variety: conformal guarantees [1], data-centric debugging [3], cross-domain graph learning [4], industrially grounded evaluation [5], conversational recommendation [6], multimodal retrieval [7], reusable user-study infrastructure [8], and controlled explanation experiments [9].

The papers also show how different objectives can reinforce one another. Open infrastructure makes user studies easier to repeat [8] and supports the reproducibility goals discussed by Said and Bellogín [2]. Auditable multimodal resources support grounded generative recommendation [7] and address several of the evaluation concerns identified by Wagne and colleagues [6]. Data-importance methods help diagnose production behavior [3] while providing the reusable artifacts encouraged by reproducibility research [2]. Personalized diversity can improve catalogue coverage while maintaining ranking performance [4]. Statistical risk control can reduce unwanted recommendations without simply returning shorter lists [1]. Explanation studies reveal how communication contributes to the user experience [9].

Taken together, the nine articles provide an increasingly complete account of the path from data to deployment. Historical interactions should be inspectable [3]. Generated evidence should be auditable [7]. Recommendation risks should be controllable [1]. Conversational systems should remain grounded in real catalogues [6]. Explanations require evaluation with users [9]. Diversity should respond to individual preferences [4]. Experiments should be reproducible [2,7,8], and their objectives should correspond to the environments in which recommender systems are used [5].

Oh, and do you want your photo to be featured in the “Just Accepted” collage next time? Then upload a photo to your ACM profile!