Outliers

[redhog]

Closes #58

- name: remove-worst-10
  type: outliers
  samples: 0.9
  embedding_keys:
    - concept
    - description

This will keep the 90 percent closest to the center (average) embedding of the keys provided. Altermnatively, you could set samples to an integer count of items to keep (or a negative number to throw away). You can also assume a gaussian distribution and set the key std to a number of standard deviations out from the center, instead of setting samples.

Small note about embeddings: If you embed too short values, some embedding models will yield a very "sparse" distribution, where the absolute majority of points lie on the surface of a hyperssphere, meaning that this operation will not work very well!

Optional keys:

  keep: false
  center:
    concept: Horse
    description: A horse is a large steppe roaming and grazing animal. Humans have utilized horses for transport throughout historical times

If keep is true outliers will be returned instead of non-outliers.

If center is provided, it must have the same keys as those listed under embedding_keys, and their values will be used to calculate the "center" embedding, instead of using the average of all embeddings of the input items. This makes it possible to use this operation is a poor-mans-RAG :P

[redhog]

• This is in some way similar to the sample operation. Maybe they should just be one operation with different options?
• Maybe there should be an option to keep the outliers instead of the center (for debugging purposes?)?