[Feature request]: Enhanced Variable Substitution and Input Processing

[mattjoyce]

What do you need?

Feature Request: Enhanced Variable Substitution and Input Processing in Fabric

Overview

Propose improvements to Fabric's variable handling and input processing to enhance automation capabilities while maintaining backward compatibility through configuration.

Proposed Changes

1. Standardized Variable Notation

Move from direct substitution to double brace notation with simplified CLI usage.
{{role}}
{{max_word_count}}

Why Double Braces?

1. Industry Standard Template Syntax:

   • Used in many popular templating systems
   • Immediately recognizable to developers from various backgrounds
   • Clear visual separation between variables and content
   • Less prone to accidental matches than simpler markers like # or $

2. Advantages over alternatives:

   • #variable: Can conflict with Markdown headers, hashtags
   • $variable: Could be confused with shell variables, environment variables
   • %variable%: Less common in modern systems
   • Single braces: Could conflict with JSON/YAML content

3. Better Pattern Readability:

   # Current syntax (less clear):
   You are a #role evaluator with #points maximum points.
   
   # Proposed syntax (clearly delineated):
   You are a {{role}} evaluator with {{points}} maximum points.
   

4. Future-proofing:

   • Double brace syntax enables powerful plugin architecture integration

   • Nested variable resolution: inner variables resolved before plugin processing

   • Example plugin capabilities:

     a. System Information:

     {{plugin:sys:hostname}}
     {{plugin:sys:user}}
     

     b. Date and Time Operations:

     {{plugin:date:now:RFC3339}}
     {{plugin:date:today:YYYY-MM-DD}}
     {{plugin:date:tomorrow:DD/MM/YYYY}}
     {{plugin:time:utc}}
     

     c. Text Processing (with variable nesting):

     {{plugin:text:trim:{{name}}}}
     {{plugin:text:uppercase:{{name}}}}
     

     d. Data Operations:

     {{plugin:yaml:format:{{config}}}}
     

     e. Database Queries (with variable interpolation):

     {{plugin:sqlite:users.db:SELECT * FROM users WHERE name={{username}}}}
     {{plugin:postgres:SELECT count(*) FROM logs WHERE date > {{date}}}}
     

   These examples demonstrate how the double brace syntax enables:

   • Simple plugin calls
   • Nested variable resolution
   • Complex data operations
   • Future extensibility

5. Enhanced Error Detection:

   • Current #variable syntax makes it impossible to reliably detect missing variables
   • With {{variable}}, Fabric can:
     • Detect and report missing variables before processing
     • Validate all required variables are provided
     • Prevent silent failures where variables aren't replaced
     • Example error:

       Error: Pattern 'expert_review' requires variable 'role' but it was not provided

Implementation details:

• CLI usage simplified: -v=role:expert (no # prefix needed)
• Backward compatibility via .env setting: LEGACY_VARIABLE_SUBSTITUTION=true/false

2. Special Input Variable

• Reserve {{input}} as a special variable for content placement
• If {{input}} exists in pattern: place input at that location
• If not specified: append input at end (maintaining current behavior)
• Enables flexibility for different LLM requirements (e.g., Claude.ai's preference for content placement)

3. Input Front Matter Support

• Allow input streams/files to include YAML front matter for variable definitions at the top of the content.
• Example input:

  --- #Fabric
  role: "Expert"
  points: 30
  ---
  Here is the actual input content that needs evaluation...

• Variables defined in front matter processed before pattern application
• Front matter variables merge with CLI-provided variables (if both present)
• Enables automated workflows to package content and variables together

Benefits

1. More explicit and standard variable syntax ({{varname}})
2. Greater control over input placement in prompts
3. Automation-friendly: scripts can prepare complete input files with their variables
4. Simplified command execution when variables are defined in input
5. Maintains pattern reusability while allowing input-specific variable definitions

Use Cases

1. Standard CLI usage (unchanged):

echo "content" | ./fabric --pattern expert_review -v=role:expert -v=points:30

2. New variable syntax:

echo "content" | ./fabric --pattern expert_review -v=role:expert
# Pattern uses {{role}} instead of #role

3. Input placement control:

# Pattern contains: 
"You are {{role}}
Input: {{input}}
Evaluate the above."

echo "content" | ./fabric --pattern expert_review -v=role:expert

4. Automated input with variables:

cat input.md | ./fabric --pattern expert_review
# Where input.md contains front matter and content, prompt contain the {{tokens}}

Backward Compatibility

• Legacy will likely still work if wrapped with {{Matt}}
• Default input placement behavior maintained when {{input}} not specified (if missing will be appended to pattern at runtime)
• Existing patterns continue to work with LEGACY_VARIABLE_SUBSTITUTION=true

Implementation Notes

• Primary code changes in GetApplyVariables function
• New YAML front matter parsing for input streams
• Minor configuration system modification

Technical Considerations

1. Front Matter Processing:

   • Parse input stream for YAML front matter
   • Extract variables before processing content
   • Handle merge priority between front matter and CLI variables

2. Variable Substitution:

   • Support both legacy and new syntax based on configuration
   • Special handling for reserved {{input}} variable
   • Handling of wrapped braces {{foo{{bar}}}}

3. Error Handling:

   • Invalid YAML in front matter
   • Missing variables notification to stderr
   • Variable syntax conflicts

Contribution Offer

I am willing to take on the development and testing of this enhancement, including:

• Implementation of all proposed features
• Comprehensive test coverage
• Documentation updates
• Backward compatibility testing

Next Steps

1. Community feedback on the proposal
2. Consensus on variable syntax and behavior
3. Potential separation of features
4. Development of implementation plan
5. Code review process
6. Testing and documentation

Please share your thoughts and suggestions on this enhancement, particularly:

• Variable syntax preference
• Front matter format preferences
• Priority rules for CLI vs front matter variables
• Additional use cases to consider

This enhancement aims to make Fabric more powerful for automation while maintaining its simplicity for interactive use.

[eugeis]

@mattjoyce thanks for your proposal, I like it.
I think, we don't need Backward compatibility, it should be enough to update existing patterns and documentation.

Please proceed with the impl. and send a PR.

[mattjoyce]

@eugeis , how complete is restapi?
Does it receive user input and variables yet?

My approach will be to insatiate plugins/template/template.go to handle all substitutions, and then,
plugins/template/sys.go
plugins/template/date.go
plugins/template/text.go

fsdb/patterns.go will just handle the pattern retrival

[eugeis]

1. Only storage based parts (patterns, context, session) are done. The pattern variables are not implemented yet.
   Chatter part is missing.

2. Yes, I like it.

[mattjoyce]

Most of the goals are included in PR #1155
The PR is two commit, and the first can stand alone if needed.
Let me know.

For the first plugins, I have kept them pretty basic, and all work with existing libs, actually native go I think.

[eugeis]

Thank you, I merged it.