Merge pull request #127 from metabrainz/merge-content-resolver

Merge content resolver

.github/workflows/ci.yml

@@ -17,7 +17,7 @@ jobs:
 
     strategy:
       matrix:
-        python-version: ["3.8", "3.9", "3.10", "3.11", "3.12"]
+        python-version: ["3.9", "3.10", "3.11", "3.12"]
 
     steps:
 

.gitignore

@@ -7,3 +7,4 @@ docs/_build
 build
 dist
 troi.egg-info
+*.db

README.md

@@ -1,13 +1,28 @@
 # Introduction
 
-This project is part of ListenBrainz' open source music recommendation efforts with an API-first
-philiosophy. API-first means that user do no need to download a lot of data before they
-can start working with Troi -- all the needed data should ideally live in online APIs, making
-it very easy for someone to get started hacking on music recommendations.
+The Troi Playlisting Engine combines all of ListenBrainz' playlist efforts:
+
+1. Playlist generation: Music recommendations and algorithmic playlist generation using a
+pipeline architecture that allows easy construction of custom pipelines that output playlists.
+You can see this part in action on ListenBrainz's Created for You pages, where we show of Weekly jams
+and Weekly Discovery playlists. The playlist generation tools use an API-first approach were users
+don't need to download massive amounts of data, but instead fetch the data via APIs as needed.
+
+2. Local content database: Using these tools a user can scan their music collection on disk or
+via a Subsonic API (e.g. Navidrome, Funkwhale, Gonic), download metadata for it and then resolve global
+playlists (playlist with only MBIDs) to files available in a local collection. We also have
+support for duplicate file detection, top tags in your collection and other insights.
+
+3. Playlist exchange: We're in the process of building this toolkit out to support saving/loading playlists
+in a number of format to hopefully break playlists free from the music silos (Spotify, Apple, etc)
+
+The project is named after [Deanna Troi](https://en.wikipedia.org/wiki/Deanna_Troi).
 
 ## Features
 
-The troi engine features the following concepts:
+### Playlist generation
+
+Troi can be used to generate playlists:
 
 1. A pipelined architecture for creating playlists from any number of sources.
 2. Data sources that fetch data from APIs and feed the data into the troi pipelines.
@@ -19,7 +34,7 @@ artists, tags, user statistics, user recommendations, LB playlists and MB collec
 Troi is being used to generate weekly recommendations on ListenBrainz (weekly jams, weekly exploration)
 as well as [LB Radio](https://listenbrainz.org/explore/lb-radio/).
 
-## Data-sets
+#### Data-sets
 
 To accomplish this goal, we, the MetaBrainz Foundation, have created and hosted a number of data-sets
 that can be accessed as a part of this project. For instance, the more stable APIs are hosted on our
@@ -36,24 +51,75 @@ We will continue to build and host more datasets as time passes. If an API endpo
 a greater number of people we will elevate these API endpoints to officially supported endpoints
 that we ensure are up to date on online at all times.
 
-The project is named after [Deanna Troi](https://en.wikipedia.org/wiki/Deanna_Troi).
+### Database / Content Resolution
+
+The ListenBrainz Content Resolver resolves global JSPF playlists to
+a local collection of music, using the content resolver.
+
+The features include:
+
+1. ListenBrainz Radio Local: allows you to generate radio-style playlists that
+that are created using only the files in the local collection, or if that is not
+possible, a global playlist with MBIDS will be resolved to a local file collection
+as best as possible.
+
+2. Periodic-jams: ListenBrainz periodic-jams, but fully resolved against your own
+local collection. This is optimized for local collections and gives better results than
+the global troi patch by the same name.
+
+3. Resolve global playlists (usually JSPF files with MusicBrainz IDs) to a local collection
+of music. Resolution happens via: MusicBrainz IDs, metadata matching or fuzzy metadata matching.
+
+4. Metadata fetching: Several of the features here require metadata to be downloaded
+from ListenBrainz in order to power the LB Radio Local.
+
+5. Scan local file collections. MP3, Ogg Vorbis, Ogg Opus, WMA, M4A and FLAC file are supported.
+
+6. Scan a remote subsonic API collection. We've tested Navidrome, Funkwhale and Gonic.
+
+7. Print a report of duplicate files in the collection
+
+8. Print a list of top tags for the collection
+
+9. Print a list of tracks that failed to resolve and print the list of albums that they
+belong to. This gives the user feedback about tracks that could be added to the collection
+to improve the local matching.
 
 ## Documentation
 
 Full documentation for Troi is available at [troi.readthedocs.org](https://troi.readthedocs.org).
 
-## Installation for end users
+### Installation for end users
 
 Troi is available for download via [PyPi](https://pypi.org/project/troi/).
 
-## Installation for Development
+```
+virtualenv -p python3 .ve
+pip3 install troi[nmslib]
+troi --help
+```
+
+Troi also depends on [nmslib-metabrainz](https://github.com/metabrainz/nmslib-metabrainz)
+to enable fuzzy matching of tracks against a local collection. nmslib-metabrainz is not
+required to run troi, it's only required for fuzzy matching, so if you're having a hard
+time installing nsmlib, omit it like this:
+
+```
+virtualenv -p python3 .ve
+pip3 install troi
+troi --help
+```
+
+### Installation for Development
+
+Note: If you have trouble installing nmslib, it is optional. Remove nsmlib from the install command below:
 
 **Linux and Mac**
 
 ```
 virtualenv -p python3 .ve
 source .ve/bin/activate
-pip3 install .[tests]
+pip3 install -e .[nmslib,tests]
 troi --help
 ```
 
@@ -62,38 +128,13 @@ troi --help
 ```
 virtualenv -p python .ve
 .ve\Scripts\activate.bat
-pip install .[tests]
+pip install -e .[nmslib,tests]
 troi --help
 ```
 
-## Basic commands
-
-List available patches:
-
-    troi list
-
-Generate a playlist using a patch:
-
-    troi playlist --print [patch-name]
-
-If the patch requires arguments, running it with no arguments will print a usage statement, e.g.
-
-    $ troi playlist lb-radio
-        Usage: lb-radio [OPTIONS] MODE PROMPT
-
-          Generate a playlist from one or more Artist MBIDs
-
-          MODE which mode to generate playlists in. must be one of easy, mediumedium, hard
-          PROMPT is the LB radio prompt. See troi/parse_prompt.py for details.
-
-        Options:
-          --help  Show this message and exit.
-
-To generate an LB Radio playlist on easy mode with an artist and a tag, use the following:
-
-    troi playlist lb-radio easy "artist:(pretty lights) tag:(chillwave)"
+## User-guide
 
-The use the --upload and --token options to upload the playlist to ListenBrainz.
+For details on how to run Troi, please see [see our user guide](https://troi.readthedocs.io/en/latest/user-guide.html).
 
 ## Running tests
 

TODO

@@ -0,0 +1,2 @@
+TODOS
+ - Make troi silent by default, enabled current level of messages only for CLI use. Coming in next PR.

config.py.sample

@@ -0,0 +1,16 @@
+# Where to find the database file
+DATABASE_FILE = ""
+
+# To connect to a subsonic API
+SUBSONIC_HOST = ""  # include http:// or https://
+SUBSONIC_USER = ""
+SUBSONIC_PASSWORD = ""
+SUBSONIC_PORT = 4533
+
+# List of music directories to scan by default
+# If paths are passed to scan command, this list is ignored.
+# Invalid directories are skipped.
+MUSIC_DIRECTORIES = [
+    'My/Music/Directory 1',
+    'My/Music/Directory 2',
+]

docs/index.rst

@@ -1,18 +1,22 @@
-Troi Recommendation Toolkit
-===========================
+Troi Playlisting Engine
+=======================
 
-This project aims to create an open source music recommendation toolkit with an API-first
-philiosophy. API-first means that user do no need to download a lot of data before they
-can start working with Troi -- all the needed data should ideally live in online APIs, making
-it very easy for someone to get started hacking on music recommendations.
+The Troi Playlisting Engine combines all of ListenBrainz' playlist efforts:
 
-Troi implements a toolkit that allows developers to create data pipelines comprised of Elements
-that generate, filter or manipulate data in a data pipeline. Troi assumes that MusicBrainz IDs
-are available for all the data passing through the system. This allows us to create a powerful
-toolkit underpinned by rich datasets, but it does have the disadvantage that recommendations
-can only be created for music present in MusicBrainz.
+1. Playlist generation: Music recommendations and algorithmic playlist generation using a
+pipeline architecture that allows easy construction of custom pipelines that output playlists.
+You can see this part in action on ListenBrainz's Created for You pages, where we show of Weekly jams
+and Weekly Discovery playlists. The playlist generation tools use an API-first approach were users
+don't need to download massive amounts of data, but instead fetch the data via APIs as needed.
+
+2. Local content database: Using these tools a user can scan their music collection on disk or
+via a Subsonic API (e.g. Navidrome, Funkwhale, Gonic), download metadata for it and then resolve global
+playlists (playlist with only MBIDs) to files available in a local collection. We also have
+support for duplicate file detection, top tags in your collection and other insights.
+
+3. Playlist exchange: We're in the process of building this toolkit out to support saving/loading playlists
+in a number of format to hopefully break playlists free from the music silos (Spotify, Apple, etc)
 
-For a much more detailed introduction to Troi, please see :ref:`technical-introduction`.
 
 User Guide
 ----------
@@ -21,8 +25,8 @@ For end-user guide on how to run Troi and what the various command line argument
 see our :ref:`user-guide`.
 
 
-MetaBrainz APIs
----------------
+MetaBrainz APIs for playlisting and recommendation
+--------------------------------------------------
 
 To accomplish the goal of an API-first toolkit, we, have created and hosted a number of data-sets
 that can be accessed as a part of this project. From Troi you can call any API you'd like, including
@@ -53,6 +57,7 @@ TV series Star Trek: The Next Generation.
 
    installation.rst
    user-guide.rst
+   troi-arguments.rst
    introduction.rst
    patches
    entities

docs/installation.rst

@@ -6,8 +6,12 @@ Installation
 Installation for End Users
 --------------------------
 
-At this point in time we're not quite ready for end users to install Troi. However, for developers, follow the instructions
-below.
+Troi is available for download via `PyPi <https://pypi.org/project/troi/>`_:
+
+.. code-block:: bash
+
+    pip3 install troi
+    troi --help
 
 Installation for Development
 ----------------------------

docs/introduction.rst

@@ -1,7 +1,7 @@
 .. _technical-introduction:
 
-Technical Introduction
-======================
+Playlist generation technical Introduction
+==========================================
 
 Patches
 -------

docs/lb_radio.rst

@@ -1,8 +1,14 @@
+.. _lb-radio:
+
 LB Radio Prompt Reference
 =========================
 
 ListenBrainz Radio is a powerful playlist/radio generation tool that gives users a lot of power to
-automatically make playlists.
+automatically make playlists. The troi toolkit offers two types of LB Radio playlists: Global and local. Global
+playlists generated with the lb-radio patch via the playlist command are made from all the available recorings
+available in MusicBrainz and contain only MusicBrainz MBIDs. Normal music players cannot play these playlists. LB Radio
+Local creates local playlists that are fully resolved against a local playlist and when converted to the m3u format
+are playable with a local playlist.
 
 To generate a playlist, the user will need to enter an "LB Radio prompt", which is a type of search query that specifies
 what music should be added to the playlist. A Radio prompt is composed of one or more terms, taking the form:

docs/troi-arguments.rst

@@ -0,0 +1,22 @@
+.. _troi-arguments:
+
+Command line options
+====================
+
+Playlist generation options
+---------------------------
+
+The full list of command line options for each of the commands is below:
+
+.. click:: troi.cli:cli
+   :prog: troi
+   :nested: full
+   :commands: list, playlist, lb-radio, resolve, weekly-jams, test
+
+Database / content resolution options
+-------------------------------------
+
+.. click:: troi.content_resolver.cli:cli
+   :prog: troi db
+   :nested: full
+   :commands: create, scan, subsonic, clean, metadata, top-tags, duplicates, unresolved