Skip to content

This repository proposes multiple ways to encode data with embeddings

License

Notifications You must be signed in to change notification settings

yui-mhcp/encoders

Repository files navigation

πŸ˜‹ Encoder networks

Check the CHANGELOG file to have a global overview of the latest modifications ! πŸ˜‹

Project structure

β”œβ”€β”€ custom_architectures
β”œβ”€β”€ custom_layers
β”œβ”€β”€ custom_train_objects
β”‚   β”œβ”€β”€ generators
β”‚   β”‚   β”œβ”€β”€ file_cache_generator.py  : abstract generator caching processed data
β”‚   β”‚   └── ge2e_generator.py        : generator dedicated to format GE2E input
β”‚   β”œβ”€β”€ losses
β”‚   β”‚   └── ge2e_loss.py    : Keras 3 implementation of the GE2E loss
β”‚   β”œβ”€β”€ metrics
β”‚   β”‚   └── ge2e_metric.py  : custom metric associated to the GE2E loss
β”œβ”€β”€ loggers
β”œβ”€β”€ models
β”‚   β”œβ”€β”€ encoder
β”‚   β”‚   β”œβ”€β”€ audio_encoder.py    : audio encoder class (audio to audio comparison with GE2E loss)
β”‚   β”‚   β”œβ”€β”€ base_encoder.py     : abstract Encoder class (trained with the GE2E loss)
β”‚   β”‚   └── text_encoder.py     : text encoder that uses pretrained embedding models
β”œβ”€β”€ pretrained_models
β”œβ”€β”€ unitests
β”œβ”€β”€ utils
β”œβ”€β”€ speaker_verification.ipynb
└── information_retrieval.ipynb

Check the main project for more information about the unextended modules / structure / main classes.

Important Note : this project is the keras 3 extension of the siamese network project. All features are not available yet. Once the convertion will be completely finished, the siamese networks project will be removed in favor of this one.

Available models

Input types Dataset Architecture Embedding dim Trainer Weights
mel-spectrogram VoxForge, CommonVoice AudioEncoder (CNN 1D + LSTM) 256 me Google Drive

Models must be unzipped in the pretrained_models/ directory !

Installation and usage

Check this installagion guide for the step-by-step instructions !

TO-DO list :

  • Make the TO-DO list
  • Comment the code
  • Optimize KNN in pure keras 3
  • Implement the clustering procedure
  • Implement the similarity matrix evaluation procedure
  • Implement the clustering evaluation procedure
  • Convert the siamese_networks project :
    • Implement the BaseEncoder class
    • Implement the BaseSiamese class
    • Implement the BaseComparator class
    • Implement the SiameseGenerator class
    • Update the README to provide more information about evaluation of encoders
  • Implement text embedding models
  • Implement a vectors database for information retrieval
  • Implement a colbert-vectors database for fine-grained search

Contacts and licence

Contacts :

  • Mail : yui-mhcp@tutanota.com
  • Discord : yui0732

Terms of use

The goal of these projects is to support and advance education and research in Deep Learning technology. To facilitate this, all associated code is made available under the GNU Affero General Public License (AGPL) v3, supplemented by a clause that prohibits commercial use (cf the LICENCE file).

These projects are released as "free software", allowing you to freely use, modify, deploy, and share the software, provided you adhere to the terms of the license. While the software is freely available, it is not public domain and retains copyright protection. The license conditions are designed to ensure that every user can utilize and modify any version of the code for their own educational and research projects.

If you wish to use this project in a proprietary commercial endeavor, you must obtain a separate license. For further details on this process, please contact me directly.

For my protection, it is important to note that all projects are available on an "As Is" basis, without any warranties or conditions of any kind, either explicit or implied. However, do not hesitate to report issues on the repository's project, or make a Pull Request to solve it πŸ˜„

Citation

If you find this project useful in your work, please add this citation to give it more visibility ! πŸ˜‹

@misc{yui-mhcp
    author  = {yui},
    title   = {A Deep Learning projects centralization},
    year    = {2021},
    publisher   = {GitHub},
    howpublished    = {\url{https://github.com/yui-mhcp}}
}

Notes and references

Tutorials :

Github project :

About

This repository proposes multiple ways to encode data with embeddings

Topics

Resources

License

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published