Add tokenize & detokenize to client, fix typos #8
Conversation
src/detokenization.rs
Outdated
| } | ||
|
|
||
| #[derive(Serialize, Debug)] | ||
| struct DetokenizationRequest<'a> { |
There was a problem hiding this comment.
Please rename this to DetokenizationBody.
There was a problem hiding this comment.
OK, I renamed it to BodyDetokenization and ResponseDetokenization in the spirit of the existing BodyCompletion and ResponseCompletion.
| @@ -215,6 +219,28 @@ impl Client { | |||
| .output_of(&task.with_model(model), how) | |||
| .await | |||
| } | |||
|
|
|||
| pub async fn tokenize( | |||
There was a problem hiding this comment.
A minimal example with a comment why you want to call this would be nice. Not required though for me to merge the PR
There was a problem hiding this comment.
ok, I added docstring examples for tokenize()/detokenize().
Thanks a lot for your review! Hope I could address your comments. I will add docstrings for tokenize/detokenize.
That's great. Just saw that the python client supports this (aleph_alpha_client.py#L573C55-L573C79) by fetching |
- implemented client code for the `/tokenize` & `/detokenize` endpoints - added docstring examples
|
Squashed the changes into a single commit. |
Added an implementation for the tasks
tokenize&detokenizeanalogous to the existing other API requestsWhile the Task + Output structs are easily extensible it feels a bit cumbersome for simple requests like tokenize/detokenize. Maybe it would be worth to consider offering simpler functions like
async fn detokenize(token_ids: &Vec<u32>) -> Result<String, Error>.