RecList is an open source library providing behavioral, "black-box" testing for recommender systems. Inspired by the pioneering work of Ribeiro et al. 2020 in NLP, we introduce a general plug-and-play procedure to scale up behavioral testing, with an easy-to-extend interface for custom use cases.
While quantitative metrics over held-out data points are important, a lot more tests are needed for recommenders to properly function in the wild and not erode our confidence in them: for example, a model may boast an accuracy improvement over the entire dataset, but actually be significantly worse than another on rare items or new users; or again, a model that correctly recommends HDMI cables as add-on for shoppers buying a TV, may also wrongly recommend TVs to shoppers just buying a cable.
RecList goal is to operationalize these important intuitions into a practical package for testing research and production models in a more nuanced way, without requiring unnecessary custom code and ad hoc procedures.
If you are not familiar with the library, we suggest first taking our small tour to get acquainted with the main abstractions through ready-made models and tests.
| Name | Link |
|---|---|
| Tutorial 101 - Introduction to Reclist | 
|
| Tutorial - RecList at EvalRS2023 (KDD) |
|
| Tutorial - FashionCLIP Evaluation with RecList |
|
- The original paper (arxiv), initial release post and beta announcement.
- EvalRS22@CIKM and EvalRS23@KDD , for music recommendations with RecList.
- A colab notebook, for a quick interactive tour.
- Our website for past talks and presentations.
- RecList is free software released under the MIT license, and it has been adopted by popular open-source data challenges.
- After a major API re-factoring, RecList is now in beta.
This doc is structured as follows:
You can take a quick tour online using our colab notebook. If you want to use RecList locally, clone the repository, create and activate a virtual env, and install the required packages from pip (you can also install from root of course).
git clone https://github.com/jacopotagliabue/reclist
cd reclist
python3 -m venv venv
source venv/bin/activate
pip install reclist
cd examples
python dummy.pyThe sample script will run a suite of tests on a dummy dataset and model, showcasing a typical workflow with the library. Note the commented arguments in the script, which you can use to customize the behavior of the library once you familiarize yourself with the basic patterns (e.g. using S3 to store the plots, leveraging a third-party tool to track experiments).
Once your development setup is working as expected, you can run
python evalrs_2023.pyto explore tests on a real-world dataset (make sure the files are available in the examples folder before you run the script). Finally, once you've run successfully the sample scripts, take the guided tour below to learn more about the abstractions and the full capabilities of RecList.
An instance of RecList represents a suite of tests for recommender systems.
As evalrs_2023.py shows, we leave users quite a wide range of options: we provide out of the box standard metrics in case your dataset is DataFrame-shaped (or you can / wish turn it into such a shape), but don't force you any pattern if you just want to use RecList for the scaffolding it provides.
For example, the following code only assumes you have a dataset with golden labels, predictions, and metadata (e.g. item features) in the shape of a DataFrame:
cdf = DFSessionRecList(
dataset=df_events,
model_name="myDataFrameRandomModel",
predictions=df_predictions,
y_test=df_dataset,
logger=LOGGER.LOCAL,
metadata_store= METADATA_STORE.LOCAL,
similarity_model=my_sim_model,
)
cdf(verbose=True)Our library pre-packages standard recSys metrics and important behavioral tests, but it is built with extensibility in mind: you can re-use tests in new suites, or you can write new domain-specific suites and tests. Any suite must inherit from the main interface, and then declare its tests as functions decorated with @rec_test.
In the example, an instance is created with one slice-based test: the decorator and return type are used to automatically generate a chart.
class MyRecList(RecList):
@rec_test(test_type="AccuracyByCountry", display_type=CHART_TYPE.BARS)
def accuracy_by_country(self):
"""
Compute the accuracy by country
NOTE: the accuracy here is just a random number.
"""
from random import randint
return {"US": randint(0, 100), "CA": randint(0, 100), "FR": randint(0, 100) }Inheritance is powerful, as we can build new suites by re-using existing ones. Here, we inherit the tests from an existing "parent" list and just add one more to create a new suite:
class ChildRecList(MyParentRecList):
@rec_test(test_type='custom_test', display_type=CHART_TYPE.SCALAR)
def my_test(self):
"""
Custom test, returning my lucky number as an example
"""
from random import randint
return { "luck_number": randint(0, 100) }Any model can be tested, as no assumption is made on the model's structure, but only the availability of predictions and ground truth. Once again, while our example leverages a DataFrame-shaped dataset for these entities, you are free to build your own RecList instance with any shape you prefer, provided you implement the metrics accordingly (see dummy.py for an example with different input types).
Once you run a suite of tests, results are dumped automatically and versioned in a folder (local or on S3), structured as follows (name of the suite, name of the model, run timestamp):
.reclist/
myList/
myModel/
1637357392/
1637357404/
If you use RecList as part of your standard testings - either for research or production purposes - you can use the JSON report for machine-to-machine communication with downstream systems (e.g. you may want to automatically fail the pipeline if tests are not passed).
RecList provides a dataset and model agnostic framework to scale up behavioral tests. We provide some suggested abstractions based on DataFrames to make existing tests and metrics fully re-usable, but we don't force any pattern on the user. As out-of-the box functionality, the package provides:
- tests and metrics to be used on your own datasets and models;
- automated storage of results, with versioning, both in a local folder or on S3;
- flexible, Python interface to declare tests-as-functions, and annotate them with display_type for automated charts;
- pre-built connectors with popular experiment trackers (e.g. Neptune, Comet), and an extensible interface to add your own (see below);
- reference implementations based on popular data challenges that used RecList: for an example of the "less wrong" latent space metric you can check the song2vec implementation here.
RecList supports streaming the results of your tests directly to your cloud platform of choice, both as metrics and charts.
If you have the Python client installed, you can use the Neptune logger by simply specifying it at init time, and either passing NEPTUNE_KEY and NEPTUNE_PROJECT_NAME as kwargs, or setting them as environment variables.
cdf = DFSessionRecList(
dataset=df_events,
model_name="myDataFrameRandomModel",
predictions=df_predictions,
y_test=df_dataset,
logger=LOGGER.NEPTUNE,
metadata_store= METADATA_STORE.LOCAL,
similarity_model=my_sim_model
)
cdf(verbose=True)If you have the Python client installed, you can use the Comet logger by simply specifying it at init time, and either passing COMET_KEY, COMET_PROJECT_NAME, COMET_WORKSPACE as kwargs, or setting them as environment variables.
cdf = DFSessionRecList(
dataset=df_events,
model_name="myDataFrameRandomModel",
predictions=df_predictions,
y_test=df_dataset,
logger=LOGGER.COMET,
metadata_store= METADATA_STORE.LOCAL,
similarity_model=my_sim_model
)
cdf(verbose=True)If you wish to add a new platform, you can do so by simply implementing a new class inheriting from RecLogger.
The original authors are:
- Patrick John Chia - LinkedIn, GitHub
- Jacopo Tagliabue - LinkedIn, GitHub
- Federico Bianchi - LinkedIn, GitHub
- Chloe He - LinkedIn, GitHub
- Brian Ko - LinkedIn, GitHub
RecList is a community project made possible by the generous support of awesome folks. Between June and December 2022, the development of our beta has been supported by Comet, Neptune , Gantry. Our beta has been developed with the help of:
If you have questions or feedback, please reach out to: jacopo dot tagliabue at nyu dot edu.
All the code is released under an open MIT license. If you found RecList useful, please cite our WWW paper:
@inproceedings{10.1145/3487553.3524215,
author = {Chia, Patrick John and Tagliabue, Jacopo and Bianchi, Federico and He, Chloe and Ko, Brian},
title = {Beyond NDCG: Behavioral Testing of Recommender Systems with RecList},
year = {2022},
isbn = {9781450391306},
publisher = {Association for Computing Machinery},
address = {New York, NY, USA},
url = {https://doi.org/10.1145/3487553.3524215},
doi = {10.1145/3487553.3524215},
pages = {99–104},
numpages = {6},
keywords = {recommender systems, open source, behavioral testing},
location = {Virtual Event, Lyon, France},
series = {WWW '22 Companion}
}This package was created with Cookiecutter and the audreyr/cookiecutter-pypackage project template.