x-fidelity is an advanced CLI tool and paired config server designed to perform opinionated framework adherence checks within a codebase. It provides a flexible and extensible way to ensure your projects are using specific standards, tools and best practices. It is based on https://github.com/CacheControl/json-rules-engine a powerful, lightweight rules engine.
=====================================
__ __ ________ ______
| ## | ## | ######## \######
\##\/ ## ______ | ##__ | ##
>## ## | \| ## \ | ##
/ ####\ \######| ###### | ##
| ## \##\ | ## _| ##_
| ## | ## | ## | ## \
\## \## \## \######
-------------------------------------
- Install x-fidelity:
yarn global add x-fidelity export PATH="$PATH:$(yarn global bin)"
- Run cli without parameters to view help
xfidelity
- Execute in current project directory with built-in demo rules for node-fullstack:
xfidelity .
- Intent and Purpose
- Key Features
- Components and entity names to understand
- System Architecture
- Configuring and Extending x-fidelity
- Installation
- Usage
- Hosting Config Servers
- Exemptions
- CI Pipeline Integration
- OpenAI Integration
- X-Fi Best Practices
- Using .xfi-config.json
- Contributing
- License
x-fidelity aims to streamline the process of maintaining code quality and consistency across projects. By providing a flexible, rule-based system, it allows teams to:
- Enforce bespoke coding standards and best practices
- Ensure consistent project archetype structures
- Maintain up-to-date private dependencies
- Catch potential issues early in the development process
- Integrate GenAI-based advanced code analysis (experimental)
The tool is designed to be highly customizable, allowing teams to define their own archetypes, rules, and checks tailored to their specific needs and tech stacks.
x-fidelity is not a replacement for standard linting more generalised code analysis tools. it is intended to help with management of bespoke requirements and as a simple way to experiment with GenAI based code reviews.
- Flexible Archetype System: Define custom project archetypes with specific rules and configurations.
- Customizable Rules: Create and apply rules for various aspects of your codebase.
- Directory Structure Validation: Ensure your project follows a predefined directory structure.
- Dependency Version Checking: Verify that your project uses up-to-date dependencies.
- Content Analysis: Search for specific patterns or strings within your codebase.
- Remote Configuration: Fetch configurations from a remote server for centralized management.
- OpenAI Integration: Leverage AI for advanced code analysis and suggestions.
- Extensible Architecture: Easily add new operators, facts, and rules to suit your needs.
-
Archetype: A predefined configuration template for a specific type of project or technology stack. It defines the rules, operators, facts, and other settings to be applied during analysis.
-
Rule: A set of conditions and corresponding actions that define a specific check or requirement for the codebase. Rules are used to identify warnings or fatal issues in the codebase.
-
Exemption: A time-limited waiver for a given git repo for a given rule until a configured expiry date.
-
Operator: A function that performs a specific comparison or check within a rule. Operators are used to evaluate conditions in rules.
-
Fact: A piece of information about the codebase or its environment that is collected and used during the analysis process. Facts can include file contents, dependency versions, or other relevant data.
-
Config Server: A server that hosts and distributes archetype configurations and rules, allowing for centralized management of x-fidelity settings.
-
Telemetry: Data collected about the usage and performance of x-fidelity, which can be used for improving the tool and understanding its impact.
The following diagram illustrates the overall architecture of the x-fidelity system:
graph TD
subgraph "Client Environments"
CI[CI Environment]
Local[Local Development]
end
subgraph "x-fidelity Core"
Engine[Analysis Engine]
CLI[CLI Interface]
ConfigMgr[Config Manager]
end
subgraph "x-fidelity Infrastructure"
CS[Config Server]
TS[Telemetry Server]
end
subgraph "External Services"
GH[GitHub]
OAI[OpenAI API]
end
subgraph "Data Sources"
Files[Repository Files]
Deps[Dependencies]
end
CI -->|Use| Engine
Local -->|Use| Engine
CI -->|Use| CLI
Local -->|Use| CLI
CLI -->|Initialize| ConfigMgr
Engine -->|Use| ConfigMgr
ConfigMgr -->|Fetch config| CS
Engine -->|Send telemetry| TS
Engine -->|Analyze| Files
Engine -->|Check| Deps
CS -->|Optional: Fetch rules| GH
TS -->|Optional: Store data| GH
Engine -.->|Optional: AI analysis| OAI
classDef optional stroke-dasharray: 5 5
class OAI optional
This diagram shows the main components of x-fidelity and how they interact:
- Client Environments: Where x-fidelity is used (CI systems or local development).
- x-fidelity Core: The main components of the system, including the analysis engine, CLI interface, and configuration manager.
- x-fidelity Infrastructure: Servers for configuration and telemetry.
- External Services: GitHub for repository interaction and optional OpenAI integration.
- Data Sources: The files and dependencies that x-fidelity analyzes.
x-fidelity is designed to be highly extensible, and only has demo rules and archetypes until you define your required configurations. You can add custom rules, operators, facts, and archetypes:
- Custom Archetypes: Define new archetypes as JSON files in your local config directory or on your config server.
- Custom Rules: Add new JSON rule files in the
rules
subdirectory of your local config or on your config server. - Custom Operators: TODO: Implement new operators and add them to your x-fidelity fork or plugin.
- Custom Facts: TODO: Create new fact providers and add them to your x-fidelity fork or plugin.
At minimum you should configure your archetypes and rules using the provided facts and operators. Facts and Operators are more complex and are not yet easily added without forking, and those provided are intended to be flexible enough for general codebase analysis.
Archetypes represent a pattern or template of git repo that you expect to be consistent. They consist of:
- rules: names of rules you have defined to perform either global (repo wide) checks, or checks on each file in a repo.
- facts: data prepared before analysing a repo such as dependency data and file structures.
- operators: these are custom operators, or provided operators that compare facts and return a boolean result. In x-fi boolean true means a rule failure.
- config: core configuration related to codebase analysis:
- minimumDependencyVersions: packages you want to conform to provided semver ranges.
- standardStructure: expected filesystem directories at high level
- blacklistPatterns: array of regex applied to filesystem paths to not include in analysis
- whitelistPatterns: array of regex applied to filesystem paths to include in analysis (after blacklist applied)
Example of a custom archetype JSON file (my-custom-archetype.json
):
{
"rules": ["myCustomRule-global", "standardRule1-iterative", "standardRule2-iterative"],
"operators": ["myCustomOperator", "standardOperator1"],
"facts": ["myCustomFact", "standardFact1"],
"config": {
"minimumDependencyVersions": {
"my-framework": ">2.0.0"
},
"standardStructure": {
"src": {
"components": null,
"utils": null
},
"tests": null
},
"blacklistPatterns": [".*\\/\\..*", ".*\\/(dist|build)(\\/.*|$)"],
"whitelistPatterns": [".*\\.(ts|tsx|js|jsx)$"]
}
}
Example of a custom rule JSON file (myCustomRule-iterative.json
) that warns of files containing 'TODO':
{
"name": "myCustomRule-iterative",
"conditions": {
"all": [
{
"fact": "fileData",
"path": "$.fileContent",
"operator": "fileContains",
"value": "TODO"
}
]
},
"event": {
"type": "warning",
"params": {
"message": "TODO comments should be resolved before committing",
"details": {
"fact": "fileData",
"path": "$.filePath"
}
}
}
}
Note on rule event types:
- Events of type "warning" are treated as such and do not cause the tool to return an error code.
- Events of type "fatality" are strictly enforced and will cause the tool to return an error code 1.
x-fidelity includes several custom operators that can be used in your rules. Here's a brief overview of each. Note that the fileName 'REPO_GLOBAL_CHECK' is not an actual file. Currently it is used to separate global and iterative rules through inclusion and exclusion.
There are a number of basic boolean check operators inherited from the json-rules-engine package that x-fidelity uses: https://github.com/CacheControl/json-rules-engine/blob/master/docs/rules.md#operators
The fileContains
operator checks if a file contains specific patterns. It supports an array of patterns for flexible content matching.
Usage example:
{
"name": "noDatabases-iterative",
"conditions": {
"all": [
{
"fact": "fileData",
"path": "$.fileName",
"operator": "notEqual",
"value": "REPO_GLOBAL_CHECK"
},
{
"fact": "repoFileAnalysis",
"params": {
"checkPattern": [".*oracle.*"],
"resultFact": "fileResultsDB"
},
"operator": "fileContains",
"value": true
}
]
},
"event": {
"type": "warning",
"params": {
"message": "code must not directly call databases",
"details": {
"fact": "fileResults-DB"
}
}
}
}
This operator returns true if any of the patterns in the array are found in the file content.
The outdatedFramework
operator checks if the project is using outdated versions of dependencies. It compares the installed versions of dependencies against the minimum required versions specified in the archetype configuration.
Key features:
- Supports both npm and Yarn package managers
- Handles version ranges and specific versions
- Checks nested dependencies
- Returns
true
if any dependency is outdated, triggering a rule failure event
Usage example in a rule:
{
"name": "outdatedFramework-global",
"conditions": {
"all": [
{
"fact": "fileData",
"path": "$.fileName",
"operator": "equal",
"value": "REPO_GLOBAL_CHECK"
},
{
"fact": "repoDependencyAnalysis",
"params": {
"resultFact": "repoDependencyResults"
},
"operator": "outdatedFramework",
"value": true
}
]
},
"event": {
"type": "fatality",
"params": {
"message": "some core framework dependencies have expired!",
"details": {
"fact": "repoDependencyResults"
}
}
}
}
The nonStandardDirectoryStructure
operator verifies if the project follows the standard directory structure defined in the archetype. It recursively checks the project's directory structure against the specified standard structure.
Key features:
- Performs a deep comparison of the directory structure
- Ignores files and focuses only on directory structure
- Returns
true
if the structure doesn't match, triggering a rule failure - Only runs on the special
REPO_GLOBAL_CHECK
file to ensure a single, comprehensive check
Usage example in a rule:
{
"name": "nonStandardDirectoryStructure-global",
"conditions": {
"all": [
{
"fact": "fileData",
"path": "$.fileName",
"operator": "equal",
"value": "REPO_GLOBAL_CHECK"
},
{
"fact": "fileData",
"path": "$.filePath",
"operator": "nonStandardDirectoryStructure",
"value": {
"fact": "standardStructure"
}
}
]
},
"event": {
"type": "warning",
"params": {
"message": "directory structure does not match the standard.",
"details": {
"fact": "standardStructure"
}
}
}
}
The 'openaiAnalysisHighSeverity' operator will be discussed in the section on the optional OpenAI integration feature.
Install x-fidelity using Node.js 18+ and Yarn:
yarn global add x-fidelity
export PATH="$PATH:$(yarn global bin)"
For persistent access, add the PATH line to your ~/.bashrc
or ~/.zshrc
file.
Run x-fidelity in your project directory:
xfidelity
Use command-line options for more control:
Usage: x-fidelity [options] [directory]
Arguments:
directory code directory to analyze
Options:
-d, --dir <directory> code directory to analyze. if an arg was passed to command it
will be treated as the dir (default: ".")
-a, --archetype <archetype> The archetype to use for analysis (default: "node-fullstack")
-c, --configServer <configServer> The config server URL for fetching remote archetype
configurations and rules
-o, --openaiEnabled <boolean> Enable OpenAI analysis (default: false)
-t, --telemetryCollector <telemetryCollector> The URL telemetry data will be sent to for usage analysis
-m, --mode <mode> Run mode: 'client' or 'server' (default: "client")
-p, --port <port> The port to run the server on (default: "8888")
-l, --localConfigPath <path> Path to local archetype config and rules
-j, --jsonTTL <minutes> Set the server json cache TTL in minutes (default: "10")
-v, --version Output the version number of xfidelity
-h, --help Display help for command
Examples:
# Run client to analyse current dir using config from remote config server
xfidelity --configServer http://localhost:8888
# Analyze a specific directory with java-microservice archetype config from remote config server and enable OpenAI analysis
xfidelity /home/projects/my-codebase -a java-microservice -c https://localhost:8888 -o true
# Run in config server mode with custom port
xfidelity --mode server --port 9999
# Use local config and rules
xfidelity -l /path/to/local/config
# Set custom cache TTL for server mode with local rules
xfidelity --mode server -l /path/to/local/config --jsonTTL 30
x-fidelity supports the following environment variables:
OPENAI_API_KEY
: Your OpenAI API key for AI-powered analysis.OPENAI_MODEL
: The OpenAI model to use (default is 'gpt-4o').XFI_LISTEN_PORT
: The port for the config server to listen on (default is 8888).CERT_PATH
: The path to SSL certificates for HTTPS config server.NODE_TLS_REJECT_UNAUTHORIZED
: Set to '0' to allow self-signed certificates (use with caution).XFI_SHARED_SECRET
: Shared secret for securing telemetry and certain server routes.
Example usage:
export OPENAI_API_KEY=your_api_key_here
export OPENAI_MODEL=gpt-4
export XFI_LISTEN_PORT=9999
export XFI_SHARED_SECRET=your_shared_secret_here
xfidelity -o true
To use local configuration files for archetypes and rules, use the -l
or --localConfigPath
option:
xfidelity -l /path/to/local/config
The local config directory should contain:
- Archetype JSON files (e.g.,
node-fullstack.json
) - A
rules
subdirectory containing rule JSON files
You can override default archetypes or add new ones by placing the corresponding JSON files in the local config directory.
To use a remote configuration server, use the -c
or --configServer
option:
xfidelity -c https://config-server.example.com
The remote server is also the xfidelity cli configured in server mode to serve archetype and rule configurations.
x-fidelity allows for centrally managed, hot-updatable custom rulesets that can be executed within managed CI pipelines and locally, ensuring consistency of applied rules. Here's an overview of the setup required:
- Set up a Node.js host environment (Docker containerization recommended).
- Create a GitHub repository to host your archetypes and rules.
- Clone the GitHub repository to the server filesystem.
- Install the x-fidelity CLI on the server.
- Configure the CLI to:
- Run on startup in server mode (
--mode server
) - Point to your rules directory cloned from GitHub (
--localConfigPath ../rule-repo/config
) - Optionally set the port to listen on (
--port <port>
)
- Run on startup in server mode (
- Create a simple CI pipeline step 'framework fidelity' after git repo clone to workspace:
- Install the x-fidelity CLI
- Run the CLI on the checked-out repo, pointing to the server (
--configServer http://my-server:8888
)
Performance report for config server for 30k requests over 5mins https://app.artillery.io/share/sh_98c74c2e592b6d3fdd326570c3e90caaac318697e2009980ac8d35309685df36
x-fidelity server can run in Docker for easy deployment and configuration. Here is a basic example for local testing. A full repo example is available here:
-
Ensure you have Docker and Docker Compose installed on your system.
-
Create a
docker-compose.yml
file with the following content:services: x-fidelity-server: build: . ports: - 8888:8888 volumes: - ./src:/usr/src/app/src - ../xfi-server/xfi-config:/usr/src/app/config environment: - NODE_ENV=production - XFI_LISTEN_PORT=8888 - CERT_PATH=/usr/src/app/certs
-
Run the following command to start the x-fidelity server:
docker-compose up --build
If you prefer to use the Dockerfile directly:
-
Build the Docker image:
docker build -t x-fidelity .
-
Run the container:
docker run -p 8888:8888 -v /path/to/local/config:/usr/src/app/config x-fidelity
The Dockerfile includes the following features:
- Automatic generation of self-signed SSL certificates for HTTPS support
- Installation of x-fidelity from npm
- Configuration of environment variables for port and certificate path
x-fidelity generates self-signed SSL certificates automatically when running in Docker. This enables HTTPS support out of the box.
Note: When testing the server using the self-signed certificate, you may need to set the following environment variable on x-fidelity clients:
NODE_TLS_REJECT_UNAUTHORIZED=0
Use this with caution and only in testing environments.
When testing using the Docker examples, you can mount your local configuration and source files:
/usr/src/app/src
: Mount your local source files here/usr/src/app/config
: Mount your local x-fidelity configuration files here
Exemptions in x-fidelity provide a way to temporarily waive specific rules for a given repository. This feature is useful when you need to make exceptions to your standard rules due to specific project requirements or during a transition period. Exemptions are only able to be trusted when used in the context of an immutable CI pipeline with config managed by a central governing team.
-
Definition: An exemption is defined for a specific rule and repository URL, with an expiration date and a reason.
-
Storage: Exemptions are stored in JSON files, typically named
<archetype>-exemptions.json
(e.g.,node-fullstack-exemptions.json
). NEW a directory named -exemptions can be created in the config dir and each json file matching*-<archetype>-exemptions.json
will also be included. -
Structure: Each exemption is an object with the following properties:
repoUrl
: The URL of the repository where the exemption applies.rule
: The name of the rule being exempted.expirationDate
: The date when the exemption expires (format: "YYYY-MM-DD").reason
: A brief explanation for the exemption.
-
Application: When x-fidelity runs, it checks if there's an active exemption for each rule violation before reporting it.
-
Expiration: Exemptions automatically expire after the specified date, ensuring that exceptions don't become permanent without review.
-
Creating Exemptions: Add new exemptions to the appropriate exemptions JSON file.
-
Reviewing Exemptions: Regularly review active exemptions to ensure they're still necessary.
-
Removing Exemptions: Remove expired or unnecessary exemptions from the JSON file.
-
Centralized Management: Store exemption files alongside your archetype configurations for easy management and version control.
[
{
"repoUrl": "https://github.com/example/project",
"rule": "outdatedFramework-global",
"expirationDate": "2024-12-31",
"reason": "Upgrading dependencies is scheduled for Q4 2024"
}
]
- Limited Duration: Set short-term expiration dates and renew if necessary, rather than setting far-future dates.
- Clear Documentation: Always provide a clear reason for each exemption.
- Regular Review: Periodically review all exemptions to ensure they're still required.
- Minimal Use: Use exemptions sparingly to maintain the integrity of your code standards.
- Team Communication: Ensure the team is aware of active exemptions and the reasons behind them.
By using exemptions judiciously, you can maintain high code standards while allowing for necessary flexibility in specific situations.
In your CI pipeline (e.g., GitHub Actions, GitLab CI, Jenkins), add a step to run x-fidelity in client mode:
steps:
- name: Check out code
uses: actions/checkout@v2
- name: Install x-fidelity
run: yarn global add x-fidelity
- name: Run x-fidelity
run: xfidelity . --configServer http://x-fidelity-server:8888
This setup allows you to maintain a centralized set of rules and archetypes that can be easily updated and applied across all your projects.
x-fidelity offers advanced AI-powered code analysis through integration with OpenAI's language models. This feature provides in-depth insights and suggestions for improving your codebase.
Important
Carefully consider the costs and data privacy implications before enabling OpenAI integration, especially for large codebases or sensitive projects.
-
Data Collection: x-fidelity gathers relevant information about your codebase, including file contents, structure, and dependencies.
-
AI Analysis: This data is sent to OpenAI's API, where it's analyzed by a powerful language model (default is GPT-4o).
-
Insights Generation: The AI generates detailed insights, suggestions, and potential issues based on best practices, common pitfalls, and the specific context of your project.
-
Results Integration: These insights are integrated into x-fidelity's output, providing you with AI-enhanced analysis alongside the tool's standard checks.
- Code Quality Assessment: Identifies potential bugs, anti-patterns, and areas for improvement.
- Architecture Suggestions: Offers insights on overall code structure and architectural decisions.
- Performance Optimization: Highlights areas where performance could be improved.
- Security Analysis: Flags potential security vulnerabilities.
- Best Practices: Suggests adherence to industry-standard best practices.
- Cost: Using OpenAI's API incurs charges based on the amount of text processed.
- Privacy: Code snippets are sent to OpenAI's servers for analysis. Ensure this complies with your organization's data policies.
- Accuracy: While highly advanced, AI suggestions should be reviewed by human developers for context and applicability.
- Rate Limits: OpenAI's API has rate limits that may affect analysis of very large codebases.
To enable AI-powered code analysis:
- Sign up for an OpenAI API key.
- Set the OPENAI_API_KEY environment variable:
export OPENAI_API_KEY=your_openai_api_key
- Enable OpenAI analysis when running x-fidelity:
xfidelity . -o true
You can also set the OpenAI model using an environment variable (optional):
export OPENAI_MODEL=gpt-4 # Optional, default is gpt-4o
By leveraging OpenAI's advanced language models, x-fidelity provides a unique blend of rule-based checks and AI-powered insights, offering a comprehensive analysis of your codebase that goes beyond traditional static analysis tools.
You can create custom OpenAI rules to leverage AI-powered analysis for specific aspects of your codebase. Here's how to create a new OpenAI rule:
Important
By convention, all OpenAI rules must have a name starting with 'openai'. This naming convention is used to identify and handle OpenAI-specific rules in the system.
- Create a new JSON file in your rules directory (e.g.,
openaiCustomAnalysis-global-rule.json
). - Use the following template structure for your rule:
{
"name": "openaiCustomAnalysis-global",
"conditions": {
"all": [
{
"fact": "fileData",
"path": "$.fileName",
"operator": "equal",
"value": "REPO_GLOBAL_CHECK"
},
{
"fact": "openaiAnalysis",
"params": {
"prompt": "Your custom prompt here",
"resultFact": "openaiCustomAnalysisResult"
},
"operator": "openaiAnalysisHighSeverity",
"value": 8
}
]
},
"event": {
"type": "warning",
"params": {
"message": "Custom message for the warning",
"details": {
"fact": "openaiCustomAnalysisResult"
}
}
}
}
-
Customize the rule:
- Set a unique
name
for your rule, ensuring it starts with 'openai' and ends with '-global'. - Modify the
prompt
in theparams
section to specify what you want the AI to analyze. - Adjust the
value
in theopenaiAnalysisHighSeverity
operator to set the severity threshold (1-10). - Customize the
message
in theevent
params to describe the warning.
- Set a unique
-
Add your new rule to the appropriate archetype configuration file.
This structure allows you to create custom AI-powered rules that can analyze your codebase for specific patterns, best practices, or potential issues. Remember to follow the naming convention to ensure proper handling of OpenAI rules in the system.
- Version Control: Keep your x-fidelity configurations (archetypes and rules) in version control.
- Continuous Integration: Integrate x-fidelity checks into your CI/CD pipeline.
- Regular Updates: Keep your archetypes, rules, and dependencies up to date.
- Documentation: Document custom rules, operators, and archetypes for your team.
- Gradual Implementation: When introducing x-fidelity to an existing project, start with basic checks and gradually increase strictness.
- Team Alignment: Ensure your team understands and agrees on the rules being enforced.
- Performance: Be mindful of the performance impact, especially for large codebases.
- Centralized Management: Use a config server to manage and distribute your archetypes and rules across projects.
The .xfi-config.json
file allows you to configure x-fidelity behavior specific to your repository. This file should be placed in the root of your project directory.
Currently, the .xfi-config.json
file supports the following options:
sensitiveFileFalsePositives
: An array of file paths that should be excluded from sensitive data checks.
Example .xfi-config.json
:
{
"sensitiveFileFalsePositives": [
"path/to/exclude/file1.js",
"path/to/exclude/file2.ts"
]
}
- When x-fidelity runs, it looks for the
.xfi-config.json
file in your project's root directory. - If found, it applies the configurations specified in this file.
- For
sensitiveFileFalsePositives
, the specified files will be excluded from checks that look for sensitive data, such as API keys or passwords.
- Version Control: Include
.xfi-config.json
in your version control system to ensure consistency across your team. - Documentation: Add comments in the listed file explaining why it is a false positive.
- Regular Review: Periodically review your
.xfi-config.json
to ensure the exclusions are still necessary and valid. - Minimal Use: Use exclusions sparingly. It's better to fix issues than to exclude them from checks.
- Feedback: If the rules being applied are resulting in too many false-postives, speak with the team that manages your central rule config.
Remember, while .xfi-config.json
allows you to adjust x-fidelity's behavior in limited ways, it should be used judiciously to maintain the integrity of your code quality checks.
Contributions to x-fidelity are welcome! Please refer to the CONTRIBUTING.md
file for guidelines on how to contribute to this project.
This project is licensed under the MIT License.. See the LICENSE
file for details.