Slow "Compiling FSM index for all state transitions:"

[alexmadey-oc]

I'm using outlines with vLLM in batch mode. I have many different JSON schemas, some of which may have many rows in a given batch but others may only have a single row. In the output when doing generation there is a log message "Compiling FSM index for all state transitions:". In many cases this will take about 20-40 seconds per new schema. This time adds significantly to my total run time, often taking much longer than the actual generation. I'm wondering if there's any way to speed it up.

[rlouf]

Could you share one of the schemas that take the longest with us?

[alexmadey-oc]

class StatementInvestmentsData(BaseModel):
    call_date: Optional[str]
    cost: Optional[float]
    coupon_max: Optional[float]
    coupon_min: Optional[float]
    currency_code: Optional[str]
    description: Optional[str]
    face_amount: Optional[float]
    footnotes: list[str] = []
    maturity_date_max: Optional[str]
    maturity_date_min: Optional[str]
    percent_net_assets: Optional[float]
    quantity: Optional[float]
    reference_rate: Optional[str]
    spread: Optional[float]
    value: Optional[float]
    yield_max: Optional[float]
    yield_min: Optional[float]

Above takes about 30 seconds. I have about 20 others currently and plan to increase to 50+ in the future.

[alexmadey-oc]

Since it seems like there is some sort of JIT compilation happening it would be cool if this could somehow be cached so it doesn't add minutes to my generation time every time my pipeline runs.