General preparing for PPO as Newbie

[osadchi]

I'm sorry, but can not find information for regular human :)
I did successfully normal fine-tuning with my dataset, but interesting for same with realtime reward model.
I get noticed even Qwen 3b can provide good score for prompt-response pair.
I'm interested how to use it with Llama-Facotory PPO fine-tuning. How to prepare things to start.
I've found the string where is score 'calculating' (in this discussion #1487), but is not clear how value to score looks like.
I mean, the reward model should give a float value. But Qwen needs additional text of prompt to understand it and produce.

[osadchi]

batch: Dict[str, "torch.Tensor"] = self.prepare_model_inputs(queries, responses)

# Logging
final_prompts = self.tokenizer.batch_decode(batch["input_ids"], skip_special_tokens=False)
for idx, prompt in enumerate(final_prompts):
    print(f"Final prompt sent to reward model (batch index {idx}): {prompt}")

As GPT said :D. I can log the result prompt with this line of code