From 28082a1138c1cdd3e79191730021154a6dcbdc2f Mon Sep 17 00:00:00 2001 From: Adam Charnock Date: Wed, 23 Aug 2023 09:43:10 +0000 Subject: [PATCH] Automated docs build [ci skip] --- dev/reference/release-process/index.html | 4 +- dev/search/search_index.json | 2 +- dev/sitemap.xml | 114 +++++++++++------------ dev/sitemap.xml.gz | Bin 734 -> 733 bytes 4 files changed, 60 insertions(+), 60 deletions(-) diff --git a/dev/reference/release-process/index.html b/dev/reference/release-process/index.html index eaa74b9a..0501134f 100644 --- a/dev/reference/release-process/index.html +++ b/dev/reference/release-process/index.html @@ -1166,8 +1166,8 @@

Lightbus release process# Version bump poetry version {patch,minor,major,prepatch,preminor,premajor,prerelease} -export VERSION=$(lightbus version --pyproject) # v1.2.3 -export VERSION_DOCS=$(lightbus version --pyproject --docs) # v1.2 +export VERSION=(lightbus version --pyproject) # v1.2.3 +export VERSION_DOCS=(lightbus version --pyproject --docs) # v1.2 # Commit git add . diff --git a/dev/search/search_index.json b/dev/search/search_index.json index 405dcf70..4c583324 100644 --- a/dev/search/search_index.json +++ b/dev/search/search_index.json @@ -1 +1 @@ -{"config":{"indexing":"full","lang":["en"],"min_search_length":3,"prebuild_index":false,"separator":"[\\s\\-]+"},"docs":[{"location":"","text":"What is Lightbus? \u00b6 Lightbus is a powerful and intuitive messaging client for your backend Python services. Lightbus uses Redis 5 as its underlying transport , although support for other platforms will be added in future. Other languages can also communicate with Lightbus by interacting with Redis . How Lightbus works \u00b6 Lightbus provides you with two tools: A client with which to fire events, and make remote procedure calls (RPCs) from anywhere within your codebase. A stand-alone Lightbus worker process in which you can setup event listeners. This process will also respond to RPCs calls. For example, you could architect an e-commerce system as follows: In this example: Django serves pages using data from the database Django performs remote procedure calls to resize images. The Lightbus worker in the image resizing service performs the image resize and responds. The price monitoring service fires price_monitor.competitor_price_changed events The Lightbus worker in the online shop web service listens for price_monitor.competitor_price_changed events and updates prices in the database accordingly. See the anatomy lesson for further discussion. Designed for ease of use \u00b6 Lightbus is designed to be intuitive and familiar, and common problems are caught with clear and helpful error messages. For example, a na\u00efve authentication API: class AuthApi ( Api ): user_registered = Event ( parameters = ( 'user' , 'email' )) class Meta : name = 'auth' def check_password ( self , user , password ): return ( user == 'admin' and password == 'secret' ) The check_password procedure can be called remotely as follows: import lightbus bus = lightbus . create () is_valid = bus . auth . check_password ( user = 'admin' , password = 'secret' ) # is_valid is True You can also listen for events: # bus.py import lightbus bus = lightbus . create () # Our event handler def send_signup_email ( event_message , user , email ): send_mail ( email , subject = f 'Welcome { user } ' ) # Setup our listeners on startup @bus . client . on_start () def on_start (): bus . auth . user_registered . listen ( send_signup_email , listener_name = \"send_signup_email\" ) Where to start? \u00b6 Starting with the tutorials section will give you a practical introduction to Lightbus. Alternatively, the explanation section will give you a grounding in the high level concepts and theory . Start with whichever section suits you best. You should ultimately look through both sections for a complete understanding. In addition, the how to section gives solutions to common use cases , and the reference section provides detailed technical information regarding specific features. Questions? \u00b6 Get in touch via: Email: adam@adamcharnock.com Phone: +442032896620 (Skype, London/Lisbon timezone) Community chat GitHub: https://github.com/adamcharnock/lightbus/ If you are having a technical problem then the more information you can include the better (problem description, screenshots, and code are all useful).","title":"Home"},{"location":"#what-is-lightbus","text":"Lightbus is a powerful and intuitive messaging client for your backend Python services. Lightbus uses Redis 5 as its underlying transport , although support for other platforms will be added in future. Other languages can also communicate with Lightbus by interacting with Redis .","title":"What is Lightbus?"},{"location":"#how-lightbus-works","text":"Lightbus provides you with two tools: A client with which to fire events, and make remote procedure calls (RPCs) from anywhere within your codebase. A stand-alone Lightbus worker process in which you can setup event listeners. This process will also respond to RPCs calls. For example, you could architect an e-commerce system as follows: In this example: Django serves pages using data from the database Django performs remote procedure calls to resize images. The Lightbus worker in the image resizing service performs the image resize and responds. The price monitoring service fires price_monitor.competitor_price_changed events The Lightbus worker in the online shop web service listens for price_monitor.competitor_price_changed events and updates prices in the database accordingly. See the anatomy lesson for further discussion.","title":"How Lightbus works"},{"location":"#designed-for-ease-of-use","text":"Lightbus is designed to be intuitive and familiar, and common problems are caught with clear and helpful error messages. For example, a na\u00efve authentication API: class AuthApi ( Api ): user_registered = Event ( parameters = ( 'user' , 'email' )) class Meta : name = 'auth' def check_password ( self , user , password ): return ( user == 'admin' and password == 'secret' ) The check_password procedure can be called remotely as follows: import lightbus bus = lightbus . create () is_valid = bus . auth . check_password ( user = 'admin' , password = 'secret' ) # is_valid is True You can also listen for events: # bus.py import lightbus bus = lightbus . create () # Our event handler def send_signup_email ( event_message , user , email ): send_mail ( email , subject = f 'Welcome { user } ' ) # Setup our listeners on startup @bus . client . on_start () def on_start (): bus . auth . user_registered . listen ( send_signup_email , listener_name = \"send_signup_email\" )","title":"Designed for ease of use"},{"location":"#where-to-start","text":"Starting with the tutorials section will give you a practical introduction to Lightbus. Alternatively, the explanation section will give you a grounding in the high level concepts and theory . Start with whichever section suits you best. You should ultimately look through both sections for a complete understanding. In addition, the how to section gives solutions to common use cases , and the reference section provides detailed technical information regarding specific features.","title":"Where to start?"},{"location":"#questions","text":"Get in touch via: Email: adam@adamcharnock.com Phone: +442032896620 (Skype, London/Lisbon timezone) Community chat GitHub: https://github.com/adamcharnock/lightbus/ If you are having a technical problem then the more information you can include the better (problem description, screenshots, and code are all useful).","title":"Questions?"},{"location":"CODE_OF_CONDUCT/","text":"Code of Conduct \u00b6 Our Pledge \u00b6 We as members, contributors, and leaders pledge to make participation in our community a harassment-free experience for everyone, regardless of age, body size, visible or invisible disability, ethnicity, sex characteristics, gender identity and expression, level of experience, education, socio-economic status, nationality, personal appearance, race, religion, or sexual identity and orientation. We pledge to act and interact in ways that contribute to an open, welcoming, diverse, inclusive, and healthy community. Our Standards \u00b6 Examples of behavior that contributes to a positive environment for our community include: Demonstrating empathy and kindness toward other people Being respectful of differing opinions, viewpoints, and experiences Giving and gracefully accepting constructive feedback Accepting responsibility and apologizing to those affected by our mistakes, and learning from the experience Focusing on what is best not just for us as individuals, but for the overall community Examples of unacceptable behavior include: The use of sexualized language or imagery, and sexual attention or advances of any kind Trolling, insulting or derogatory comments, and personal or political attacks Public or private harassment Publishing others' private information, such as a physical or email address, without their explicit permission Other conduct which could reasonably be considered inappropriate in a professional setting Enforcement Responsibilities \u00b6 Community leaders are responsible for clarifying and enforcing our standards of acceptable behavior and will take appropriate and fair corrective action in response to any behavior that they deem inappropriate, threatening, offensive, or harmful. Community leaders have the right and responsibility to remove, edit, or reject comments, commits, code, wiki edits, issues, and other contributions that are not aligned to this Code of Conduct, and will communicate reasons for moderation decisions when appropriate. Scope \u00b6 This Code of Conduct applies within all community spaces, and also applies when an individual is officially representing the community in public spaces. Examples of representing our community include using an official e-mail address, posting via an official social media account, or acting as an appointed representative at an online or offline event. Enforcement \u00b6 Instances of abusive, harassing, or otherwise unacceptable behavior may be reported to the community leaders responsible for enforcement at adam@adamcharnock.com . All complaints will be reviewed and investigated promptly and fairly. All community leaders are obligated to respect the privacy and security of the reporter of any incident. Enforcement Guidelines \u00b6 Community leaders will follow these Community Impact Guidelines in determining the consequences for any action they deem in violation of this Code of Conduct: 1. Correction \u00b6 Community Impact : Use of inappropriate language or other behavior deemed unprofessional or unwelcome in the community. Consequence : A private, written warning from community leaders, providing clarity around the nature of the violation and an explanation of why the behavior was inappropriate. A public apology may be requested. 2. Warning \u00b6 Community Impact : A violation through a single incident or series of actions. Consequence : A warning with consequences for continued behavior. No interaction with the people involved, including unsolicited interaction with those enforcing the Code of Conduct, for a specified period of time. This includes avoiding interactions in community spaces as well as external channels like social media. Violating these terms may lead to a temporary or permanent ban. 3. Temporary Ban \u00b6 Community Impact : A serious violation of community standards, including sustained inappropriate behavior. Consequence : A temporary ban from any sort of interaction or public communication with the community for a specified period of time. No public or private interaction with the people involved, including unsolicited interaction with those enforcing the Code of Conduct, is allowed during this period. Violating these terms may lead to a permanent ban. 4. Permanent Ban \u00b6 Community Impact : Demonstrating a pattern of violation of community standards, including sustained inappropriate behavior, harassment of an individual, or aggression toward or disparagement of classes of individuals. Consequence : A permanent ban from any sort of public interaction within the project community. Attribution \u00b6 This Code of Conduct is adapted from the Contributor Covenant , version 2.0, available at https://www.contributor-covenant.org/version/2/0/code_of_conduct.html . Community Impact Guidelines were inspired by Mozilla's code of conduct enforcement ladder . For answers to common questions about this code of conduct, see the FAQ at https://www.contributor-covenant.org/faq . Translations are available at https://www.contributor-covenant.org/translations . For answers to common questions about this code of conduct, see https://www.contributor-covenant.org/faq","title":"Code of Conduct"},{"location":"CODE_OF_CONDUCT/#code-of-conduct","text":"","title":"Code of Conduct"},{"location":"CODE_OF_CONDUCT/#our-pledge","text":"We as members, contributors, and leaders pledge to make participation in our community a harassment-free experience for everyone, regardless of age, body size, visible or invisible disability, ethnicity, sex characteristics, gender identity and expression, level of experience, education, socio-economic status, nationality, personal appearance, race, religion, or sexual identity and orientation. We pledge to act and interact in ways that contribute to an open, welcoming, diverse, inclusive, and healthy community.","title":"Our Pledge"},{"location":"CODE_OF_CONDUCT/#our-standards","text":"Examples of behavior that contributes to a positive environment for our community include: Demonstrating empathy and kindness toward other people Being respectful of differing opinions, viewpoints, and experiences Giving and gracefully accepting constructive feedback Accepting responsibility and apologizing to those affected by our mistakes, and learning from the experience Focusing on what is best not just for us as individuals, but for the overall community Examples of unacceptable behavior include: The use of sexualized language or imagery, and sexual attention or advances of any kind Trolling, insulting or derogatory comments, and personal or political attacks Public or private harassment Publishing others' private information, such as a physical or email address, without their explicit permission Other conduct which could reasonably be considered inappropriate in a professional setting","title":"Our Standards"},{"location":"CODE_OF_CONDUCT/#enforcement-responsibilities","text":"Community leaders are responsible for clarifying and enforcing our standards of acceptable behavior and will take appropriate and fair corrective action in response to any behavior that they deem inappropriate, threatening, offensive, or harmful. Community leaders have the right and responsibility to remove, edit, or reject comments, commits, code, wiki edits, issues, and other contributions that are not aligned to this Code of Conduct, and will communicate reasons for moderation decisions when appropriate.","title":"Enforcement Responsibilities"},{"location":"CODE_OF_CONDUCT/#scope","text":"This Code of Conduct applies within all community spaces, and also applies when an individual is officially representing the community in public spaces. Examples of representing our community include using an official e-mail address, posting via an official social media account, or acting as an appointed representative at an online or offline event.","title":"Scope"},{"location":"CODE_OF_CONDUCT/#enforcement","text":"Instances of abusive, harassing, or otherwise unacceptable behavior may be reported to the community leaders responsible for enforcement at adam@adamcharnock.com . All complaints will be reviewed and investigated promptly and fairly. All community leaders are obligated to respect the privacy and security of the reporter of any incident.","title":"Enforcement"},{"location":"CODE_OF_CONDUCT/#enforcement-guidelines","text":"Community leaders will follow these Community Impact Guidelines in determining the consequences for any action they deem in violation of this Code of Conduct:","title":"Enforcement Guidelines"},{"location":"CODE_OF_CONDUCT/#1-correction","text":"Community Impact : Use of inappropriate language or other behavior deemed unprofessional or unwelcome in the community. Consequence : A private, written warning from community leaders, providing clarity around the nature of the violation and an explanation of why the behavior was inappropriate. A public apology may be requested.","title":"1. Correction"},{"location":"CODE_OF_CONDUCT/#2-warning","text":"Community Impact : A violation through a single incident or series of actions. Consequence : A warning with consequences for continued behavior. No interaction with the people involved, including unsolicited interaction with those enforcing the Code of Conduct, for a specified period of time. This includes avoiding interactions in community spaces as well as external channels like social media. Violating these terms may lead to a temporary or permanent ban.","title":"2. Warning"},{"location":"CODE_OF_CONDUCT/#3-temporary-ban","text":"Community Impact : A serious violation of community standards, including sustained inappropriate behavior. Consequence : A temporary ban from any sort of interaction or public communication with the community for a specified period of time. No public or private interaction with the people involved, including unsolicited interaction with those enforcing the Code of Conduct, is allowed during this period. Violating these terms may lead to a permanent ban.","title":"3. Temporary Ban"},{"location":"CODE_OF_CONDUCT/#4-permanent-ban","text":"Community Impact : Demonstrating a pattern of violation of community standards, including sustained inappropriate behavior, harassment of an individual, or aggression toward or disparagement of classes of individuals. Consequence : A permanent ban from any sort of public interaction within the project community.","title":"4. Permanent Ban"},{"location":"CODE_OF_CONDUCT/#attribution","text":"This Code of Conduct is adapted from the Contributor Covenant , version 2.0, available at https://www.contributor-covenant.org/version/2/0/code_of_conduct.html . Community Impact Guidelines were inspired by Mozilla's code of conduct enforcement ladder . For answers to common questions about this code of conduct, see the FAQ at https://www.contributor-covenant.org/faq . Translations are available at https://www.contributor-covenant.org/translations . For answers to common questions about this code of conduct, see https://www.contributor-covenant.org/faq","title":"Attribution"},{"location":"explanation/","text":"Explanation overview \u00b6 This section discusses the theoretical and conceptual aspects of Lightbus . This is in contrast to the more practical tutorial and how to sections.","title":"Overview"},{"location":"explanation/#explanation-overview","text":"This section discusses the theoretical and conceptual aspects of Lightbus . This is in contrast to the more practical tutorial and how to sections.","title":"Explanation overview"},{"location":"explanation/anatomy-lesson/","text":"Anatomy lesson \u00b6 Lightbus provides you with two tools: A client with which to fire events , listen for events and make remote procedure calls (RPCs). A stand-alone Lightbus worker process in which you can setup event listeners. This process will also respond to RPCs calls. The client \u00b6 The client allows you to interact with the bus from within your Python codebase. For example: # The bus is created in your bus.py file import lightbus bus = lightbus . create () # Elsewhere in codebase from my_project.bus import bus # Perform a remote procedure call is_valid = bus . auth . check_password ( user = \"admin\" , password = \"secret\" ) # Fire an event bus . auth . user_registered ( user = \"sally\" , email = \"sally@example.com\" ) You can use this client anywhere you need to, such as: Within your Django/Flask views Within scheduled jobs Within Lightbus event & RPC handlers (see below) Important Each service should create its bus client within the service's bus.py file (also known as the servies bus module ). Other code within the service should import the bus client from the bus module as needed. See how to access your bus client . The Lightbus worker process ( lightbus run ) \u00b6 The Lightbus worker is a long running process started using the lightbus run console command. This process serves two purposes: Listens for events and fires any executes any listeners you have created. Respond to incoming remote procedure calls for the service's registered APIs. This process imports your bus module (see the module loading configuration reference) in order to bootstrap itself. Your bus module should therefore: Instantiate the bus client in a module variable named bus Register any API definitions for your service Setup listeners for any events you wish to listen for For example, let's use the auth.create_user() remote procedure call to create a new user every time a customers.new_customer event appears on the bus: # bus.py import lightbus bus = lightbus . create () def create_user_for_customer ( event_message , customer_name , email ): # We can do something locally, or call an # RPC, or both. Here we call an RPC. bus . auth . create_user ( name = customer_name , email = email ) # Setup our listeners on startup @bus . client . on_start () def on_start (): # Create a new user for each new customer bus . customers . new_customer . listen ( create_user_for_customer ) You start this process using the command: lightbus run Lightbus will import the bus module (your bus.py file) and wait for incoming events and remote procedure calls. A service will only need a Lightbus process if it wishes to listen for events or provide any RPCs which can be called.","title":"Anatomy lesson"},{"location":"explanation/anatomy-lesson/#anatomy-lesson","text":"Lightbus provides you with two tools: A client with which to fire events , listen for events and make remote procedure calls (RPCs). A stand-alone Lightbus worker process in which you can setup event listeners. This process will also respond to RPCs calls.","title":"Anatomy lesson"},{"location":"explanation/anatomy-lesson/#the-client","text":"The client allows you to interact with the bus from within your Python codebase. For example: # The bus is created in your bus.py file import lightbus bus = lightbus . create () # Elsewhere in codebase from my_project.bus import bus # Perform a remote procedure call is_valid = bus . auth . check_password ( user = \"admin\" , password = \"secret\" ) # Fire an event bus . auth . user_registered ( user = \"sally\" , email = \"sally@example.com\" ) You can use this client anywhere you need to, such as: Within your Django/Flask views Within scheduled jobs Within Lightbus event & RPC handlers (see below) Important Each service should create its bus client within the service's bus.py file (also known as the servies bus module ). Other code within the service should import the bus client from the bus module as needed. See how to access your bus client .","title":"The client"},{"location":"explanation/anatomy-lesson/#the-lightbus-worker-process-lightbus-run","text":"The Lightbus worker is a long running process started using the lightbus run console command. This process serves two purposes: Listens for events and fires any executes any listeners you have created. Respond to incoming remote procedure calls for the service's registered APIs. This process imports your bus module (see the module loading configuration reference) in order to bootstrap itself. Your bus module should therefore: Instantiate the bus client in a module variable named bus Register any API definitions for your service Setup listeners for any events you wish to listen for For example, let's use the auth.create_user() remote procedure call to create a new user every time a customers.new_customer event appears on the bus: # bus.py import lightbus bus = lightbus . create () def create_user_for_customer ( event_message , customer_name , email ): # We can do something locally, or call an # RPC, or both. Here we call an RPC. bus . auth . create_user ( name = customer_name , email = email ) # Setup our listeners on startup @bus . client . on_start () def on_start (): # Create a new user for each new customer bus . customers . new_customer . listen ( create_user_for_customer ) You start this process using the command: lightbus run Lightbus will import the bus module (your bus.py file) and wait for incoming events and remote procedure calls. A service will only need a Lightbus process if it wishes to listen for events or provide any RPCs which can be called.","title":"The Lightbus worker process (lightbus run)"},{"location":"explanation/apis/","text":"APIs \u00b6 When we refer to an API , we are referring to an Api class definition. All functionality on the bus is defined using APIs. For example, consider an API for support tickets within a company's help desk: class TicketApi ( Api ): ticket_created = Event ( parameters = ( 'id' , 'sender' , 'subject' , 'body' )) class Meta : name = 'help_desk.ticket' def get ( self , id ): return get_ticket_from_db ( pk = id ) This API defines an event, a procedure, and the name used to address the API on the bus. The help desk service could define multiple additional APIs as needed (perhaps for listing help desk staff or retrieving reports). API registration & authoritative/non-authoritative APIs \u00b6 An API can be registered with your service's bus client as follows: import lightbus from my_apis import HelpDeskApi bus = lightbus . create () # Register the API with your service's client bus . client . register_api ( HelpDeskApi ()) Registering an API will: Allow you to fire events on the API using the service's client Cause the lightbus worker for this service (i.e. lightbus run ) to respond to remote procedure calls on the registered API We say that a service which registers an API is authoritative for that API. Services which do not register a given API are non-authoritative for the API. Both authoritative and non-authoritative services can listen for events on any API and call remote procedures on any API. For example, a separate online store service could not fire the help_desk.ticket_created event on the API we defined above. Nor would you reasonably expect the online store to services remote procedure calls for help_desk.ticket_created() . Why? \u00b6 Preventing the online store service from responding to remote procedure calls for the help desk service makes sense. There is no reason the online store should have any awareness of the help desk, so you would not expect it to respond to remote procedure calls regarding tickets. Therefore, the logic for allowing only authoritative services to respond to remote procedure calls is hopefully compelling. The case for limiting event firing to authoritative services is one of architecture, maintainability, and consistency: Allowing any event to be fired by any service within your organisation could quickly lead to spiraling complexity. The authoritative service will always have sufficient information to guarantee basic validity of an emitted message (for example, the event exists, required parameters are present etc). As a result errors can be caught earlier, rather than allowing them to propagate onto the bus and potentially impact distant services. We welcome discussion on this topic, open a GitHub issue if you would like to discuss this further.","title":"APIs"},{"location":"explanation/apis/#apis","text":"When we refer to an API , we are referring to an Api class definition. All functionality on the bus is defined using APIs. For example, consider an API for support tickets within a company's help desk: class TicketApi ( Api ): ticket_created = Event ( parameters = ( 'id' , 'sender' , 'subject' , 'body' )) class Meta : name = 'help_desk.ticket' def get ( self , id ): return get_ticket_from_db ( pk = id ) This API defines an event, a procedure, and the name used to address the API on the bus. The help desk service could define multiple additional APIs as needed (perhaps for listing help desk staff or retrieving reports).","title":"APIs"},{"location":"explanation/apis/#api-registration-authoritativenon-authoritative-apis","text":"An API can be registered with your service's bus client as follows: import lightbus from my_apis import HelpDeskApi bus = lightbus . create () # Register the API with your service's client bus . client . register_api ( HelpDeskApi ()) Registering an API will: Allow you to fire events on the API using the service's client Cause the lightbus worker for this service (i.e. lightbus run ) to respond to remote procedure calls on the registered API We say that a service which registers an API is authoritative for that API. Services which do not register a given API are non-authoritative for the API. Both authoritative and non-authoritative services can listen for events on any API and call remote procedures on any API. For example, a separate online store service could not fire the help_desk.ticket_created event on the API we defined above. Nor would you reasonably expect the online store to services remote procedure calls for help_desk.ticket_created() .","title":"API registration & authoritative/non-authoritative APIs"},{"location":"explanation/apis/#why","text":"Preventing the online store service from responding to remote procedure calls for the help desk service makes sense. There is no reason the online store should have any awareness of the help desk, so you would not expect it to respond to remote procedure calls regarding tickets. Therefore, the logic for allowing only authoritative services to respond to remote procedure calls is hopefully compelling. The case for limiting event firing to authoritative services is one of architecture, maintainability, and consistency: Allowing any event to be fired by any service within your organisation could quickly lead to spiraling complexity. The authoritative service will always have sufficient information to guarantee basic validity of an emitted message (for example, the event exists, required parameters are present etc). As a result errors can be caught earlier, rather than allowing them to propagate onto the bus and potentially impact distant services. We welcome discussion on this topic, open a GitHub issue if you would like to discuss this further.","title":"Why?"},{"location":"explanation/architecture-tips/","text":"Architecture tips \u00b6 These tips draw some core concepts from the Domain Driven Design and CQRS patterns. We only scape the surface of these patterns here, but the ideas covered below are the ones the Lightbus maintainers found most useful when using Lightbus. There is a lot more to be said on each point. The intent here is to provide a digestible starting point. Use common data structures \u00b6 Create structures which represent the data you wish to transmit. Both NamedTuples and dataclasses work well for this purpose. For example: from typing import NamedTuple class Customer ( NamedTuple ): name : str email : str age : int You can then take advantage of Lightbus' schema checking and data casting: import lightbus class CustomerApi ( lightbus . Api ): class Meta : name = \"customer\" def create ( customer : Customer ): # Lightbus will ensure the incoming data conforms to # Customer, and will cast the data to be a Customer object ... You would call the customer.create() API as follows: import lightbus bus = lightbus . create () bus . customer . create ( customer = Customer ( name = \"Joe\" , email = \"joe@gmail.com\" , age = 54 ) ) This provides a standard way of communicating shared concepts across your services. Note You can still send and receive events even if you do not have the data strucuture available. To send you can simply use a dictionary: bus . customer . create ( customer = { \"name\" : \"Joe\" , \"email\" : \"joe@gmail.com\" , \"age\" : 54 } ) Likewise, an RPC or event listener without type hints will simply receive a dictionary. See also use a monorepository . Decide on boundaries \u00b6 Your services will likely use an assortment of entities. It is common for these to map to database tables or [ORM] classes. For example, you may have Customer , Address , Order , and OrderLine entities (and probably many more). You will need to pass these between your services, but how should you structure them in order to do so? Do pass them individually, or combine them into a hierarchy? Do you sometimes omit some fields for brevity, or not? Take the following events for example: customer.signup(customer) \u2013 Does the customer object include the address(es)? order.shipped(order) \u2013 Does the order object contain each line item? order.line_item_deleted(order, line_item) \u2013 Do we need to pass line_item here, or should the line items be included within order ? This is a simple example with only four entities. In a real-world system it is very easy to end up passing around data in forms which become increasingly bloated, inconsistent, and complex. You can mitigate this by grouping your entities together in a way that makes logical sense for your situation . These are your aggregates . Decide on these aggregates in advance, and stick to it. For example, we may group the Customer , Address , Order , and OrderLine entities as follows: A Customer aggregate. Each Customer contains an address field, which is an Address entity. An Order aggregate. Each Order contains a line_items field, which is a list of OrderLine entities. The Customer and Order aggregates also will likely contain other fields not listed above (such as name, email, or quantity). You should only ever pass around the top level aggregates. Additionally: Identify aggregates with UUIDs Do not enforce foreign keys between aggregates. Your application code will need to deal with inconsistencies gracefully (likely in it's user interface). It is likely still a good idea to use sequential integer primary keys in your RDBMS Do not share database-level sequential integer primary keys with other servces Writes are special \u00b6 Writes are inherently different to reads in a distributed system. Reading data is generally straightforward, whereas data modifications need to be propagated to all concerned services. Reading data looks broadly like this: Read data from storage (disk, ORM, memory etc) Perhaps transform the data Display the data, or send it somewhere An initial attempt at writing data may look like this: Write the data locally Broadcast the change (Lightbus event or RPC) Other services receive & apply the change The downside of this approach is that you have two write paths (writing locally and broadcasting the change). If either of these fails then your services will have an inconsistent view of the world. A more CQRS-based approach to writing data looks like this: Broadcast the change (Lightbus event) Services (including the broadcasting service) receive & apply the change The benefit of this approach is that either the change happens or it does not. Additionally all applications share the same write path, which reduces overall complexity. A downside is that changes can take a moment to be applied, so changes made by a user may not be immediately visible. Use a monorepository \u00b6 A monorepository is a repository which stores all your services, rather than having one repository per service. There are pros and cons to this approach and you should perform your own research. The relevant benefits to Lightbus (and distributed architectures in general) are: Atomic commits across multiple services \u2013 useful when a change to one service affects another. Easier sharing of data structures \u2013 this provides a shared language across all of your services. These data structures can be used as type hints on your events and RPCs, which Lightbus will automatically cast data into. Sharing of APIs \u2013 in some cases you may wish for multiple services fire events on the same API. In this case each of these services will need to have access to the API class. (See API registration & authoritative/non-authoritative APIs ) Sharing of global bus configuration \u2013 similar to sharing data structures, your global bus configuration YAML file can likewise be available to all services.","title":"Architecture tips"},{"location":"explanation/architecture-tips/#architecture-tips","text":"These tips draw some core concepts from the Domain Driven Design and CQRS patterns. We only scape the surface of these patterns here, but the ideas covered below are the ones the Lightbus maintainers found most useful when using Lightbus. There is a lot more to be said on each point. The intent here is to provide a digestible starting point.","title":"Architecture tips"},{"location":"explanation/architecture-tips/#use-common-data-structures","text":"Create structures which represent the data you wish to transmit. Both NamedTuples and dataclasses work well for this purpose. For example: from typing import NamedTuple class Customer ( NamedTuple ): name : str email : str age : int You can then take advantage of Lightbus' schema checking and data casting: import lightbus class CustomerApi ( lightbus . Api ): class Meta : name = \"customer\" def create ( customer : Customer ): # Lightbus will ensure the incoming data conforms to # Customer, and will cast the data to be a Customer object ... You would call the customer.create() API as follows: import lightbus bus = lightbus . create () bus . customer . create ( customer = Customer ( name = \"Joe\" , email = \"joe@gmail.com\" , age = 54 ) ) This provides a standard way of communicating shared concepts across your services. Note You can still send and receive events even if you do not have the data strucuture available. To send you can simply use a dictionary: bus . customer . create ( customer = { \"name\" : \"Joe\" , \"email\" : \"joe@gmail.com\" , \"age\" : 54 } ) Likewise, an RPC or event listener without type hints will simply receive a dictionary. See also use a monorepository .","title":"Use common data structures"},{"location":"explanation/architecture-tips/#decide-on-boundaries","text":"Your services will likely use an assortment of entities. It is common for these to map to database tables or [ORM] classes. For example, you may have Customer , Address , Order , and OrderLine entities (and probably many more). You will need to pass these between your services, but how should you structure them in order to do so? Do pass them individually, or combine them into a hierarchy? Do you sometimes omit some fields for brevity, or not? Take the following events for example: customer.signup(customer) \u2013 Does the customer object include the address(es)? order.shipped(order) \u2013 Does the order object contain each line item? order.line_item_deleted(order, line_item) \u2013 Do we need to pass line_item here, or should the line items be included within order ? This is a simple example with only four entities. In a real-world system it is very easy to end up passing around data in forms which become increasingly bloated, inconsistent, and complex. You can mitigate this by grouping your entities together in a way that makes logical sense for your situation . These are your aggregates . Decide on these aggregates in advance, and stick to it. For example, we may group the Customer , Address , Order , and OrderLine entities as follows: A Customer aggregate. Each Customer contains an address field, which is an Address entity. An Order aggregate. Each Order contains a line_items field, which is a list of OrderLine entities. The Customer and Order aggregates also will likely contain other fields not listed above (such as name, email, or quantity). You should only ever pass around the top level aggregates. Additionally: Identify aggregates with UUIDs Do not enforce foreign keys between aggregates. Your application code will need to deal with inconsistencies gracefully (likely in it's user interface). It is likely still a good idea to use sequential integer primary keys in your RDBMS Do not share database-level sequential integer primary keys with other servces","title":"Decide on boundaries"},{"location":"explanation/architecture-tips/#writes-are-special","text":"Writes are inherently different to reads in a distributed system. Reading data is generally straightforward, whereas data modifications need to be propagated to all concerned services. Reading data looks broadly like this: Read data from storage (disk, ORM, memory etc) Perhaps transform the data Display the data, or send it somewhere An initial attempt at writing data may look like this: Write the data locally Broadcast the change (Lightbus event or RPC) Other services receive & apply the change The downside of this approach is that you have two write paths (writing locally and broadcasting the change). If either of these fails then your services will have an inconsistent view of the world. A more CQRS-based approach to writing data looks like this: Broadcast the change (Lightbus event) Services (including the broadcasting service) receive & apply the change The benefit of this approach is that either the change happens or it does not. Additionally all applications share the same write path, which reduces overall complexity. A downside is that changes can take a moment to be applied, so changes made by a user may not be immediately visible.","title":"Writes are special"},{"location":"explanation/architecture-tips/#use-a-monorepository","text":"A monorepository is a repository which stores all your services, rather than having one repository per service. There are pros and cons to this approach and you should perform your own research. The relevant benefits to Lightbus (and distributed architectures in general) are: Atomic commits across multiple services \u2013 useful when a change to one service affects another. Easier sharing of data structures \u2013 this provides a shared language across all of your services. These data structures can be used as type hints on your events and RPCs, which Lightbus will automatically cast data into. Sharing of APIs \u2013 in some cases you may wish for multiple services fire events on the same API. In this case each of these services will need to have access to the API class. (See API registration & authoritative/non-authoritative APIs ) Sharing of global bus configuration \u2013 similar to sharing data structures, your global bus configuration YAML file can likewise be available to all services.","title":"Use a monorepository"},{"location":"explanation/bus/","text":"The bus \u00b6 The bus is the communications channel which links all your services together. Currently this is Redis. You use lightbus.create() in your bus.py file to access this bus: # bus.py import lightbus bus = lightbus . create () This creates a high-level client through which you can perform remote procedure calls and fire events . About buses \u00b6 In computing, a bus is a shared communication medium. A bus allows any software/hardware connected to that medium to communicate, as long as common rules are obeyed. In this sense a bus is very similar to a conversation between a group of people. In electronics the communication medium can be a simple copper cable. In software the communication medium is itself defined by software. Lightbus uses Redis as its communication medium , although support for other mediums may be added in future.","title":"The bus"},{"location":"explanation/bus/#the-bus","text":"The bus is the communications channel which links all your services together. Currently this is Redis. You use lightbus.create() in your bus.py file to access this bus: # bus.py import lightbus bus = lightbus . create () This creates a high-level client through which you can perform remote procedure calls and fire events .","title":"The bus"},{"location":"explanation/bus/#about-buses","text":"In computing, a bus is a shared communication medium. A bus allows any software/hardware connected to that medium to communicate, as long as common rules are obeyed. In this sense a bus is very similar to a conversation between a group of people. In electronics the communication medium can be a simple copper cable. In software the communication medium is itself defined by software. Lightbus uses Redis as its communication medium , although support for other mediums may be added in future.","title":"About buses"},{"location":"explanation/configuration/","text":"Configuration \u00b6 As discussed in the configuration reference , Lightbus has three stages of configuration: Module loading Service-level configuration Global bus configuration See the configuration reference for details on how this works in practice. Here we will discuss the reasoning behind this system. 1. Module loading \u00b6 Lightbus needs to know how to bootstrap itself. It needs to know where to start. This module loading step is how we provide Lightbus with this information, via the LIGHTBUS_MODULE environment variable. The module loading was inspired by Django 's DJANGO_SETTINGS_MODULE environment variable. LIGHTBUS_MODULE has a sensible default of bus in the same way that DJANGO_SETTINGS_MODULE has a sensible default of settings . This default will work in many scenarios, but may also need to be customised depending on one's project structure. 2. Service-level configuration \u00b6 Some configuration must by its nature be specific to a service, and not global to the entire bus. These options are broadly: Configuration which distinguishes this service from any other service ( service_name / process_name ) Configuration related to the specific deployment of this service ( features ) A pointer to the global bus configuration 3. Global bus configuration \u00b6 The global bus configuration provides the bulk of the Lightbus options. This configuration should be consistent across all Lightbus clients. But what is the reasoning here? The reasoning is that the bus is a globally shared resource, therefore everything that uses the bus is going to need a follow a common configuration in order to function. For example, consider the case where one bus client is configured to connect to redis server A for the customers API, and another bus client is configured to connect to redis server B for the same API. What will happen? The result will be that you will have effectively created a network partition. Each bus client will operate in total ignorance of each other. Events could be lost or ignored, and RPCs may never be processed. Some configuration must therefore be common to all clients, and that is what the global configuration provides. Configuration loading over HTTP(S) \u00b6 To this end, Lightbus supports loading configuration over HTTP(S). The intention is not for you to host you Lightbus configuration on the public internet! . Rather, and if you wish, you may find is useful to host your configuration on an internal-only endpoint. Alternatively, you may decide to ensure your build/deploy process distributes a copy of the global configuration file to every service. The discussion on monorepositories in Architecture Tips is relevant here.","title":"Configuration"},{"location":"explanation/configuration/#configuration","text":"As discussed in the configuration reference , Lightbus has three stages of configuration: Module loading Service-level configuration Global bus configuration See the configuration reference for details on how this works in practice. Here we will discuss the reasoning behind this system.","title":"Configuration"},{"location":"explanation/configuration/#1-module-loading","text":"Lightbus needs to know how to bootstrap itself. It needs to know where to start. This module loading step is how we provide Lightbus with this information, via the LIGHTBUS_MODULE environment variable. The module loading was inspired by Django 's DJANGO_SETTINGS_MODULE environment variable. LIGHTBUS_MODULE has a sensible default of bus in the same way that DJANGO_SETTINGS_MODULE has a sensible default of settings . This default will work in many scenarios, but may also need to be customised depending on one's project structure.","title":"1. Module loading"},{"location":"explanation/configuration/#2-service-level-configuration","text":"Some configuration must by its nature be specific to a service, and not global to the entire bus. These options are broadly: Configuration which distinguishes this service from any other service ( service_name / process_name ) Configuration related to the specific deployment of this service ( features ) A pointer to the global bus configuration","title":"2. Service-level configuration"},{"location":"explanation/configuration/#3-global-bus-configuration","text":"The global bus configuration provides the bulk of the Lightbus options. This configuration should be consistent across all Lightbus clients. But what is the reasoning here? The reasoning is that the bus is a globally shared resource, therefore everything that uses the bus is going to need a follow a common configuration in order to function. For example, consider the case where one bus client is configured to connect to redis server A for the customers API, and another bus client is configured to connect to redis server B for the same API. What will happen? The result will be that you will have effectively created a network partition. Each bus client will operate in total ignorance of each other. Events could be lost or ignored, and RPCs may never be processed. Some configuration must therefore be common to all clients, and that is what the global configuration provides.","title":"3. Global bus configuration"},{"location":"explanation/configuration/#configuration-loading-over-https","text":"To this end, Lightbus supports loading configuration over HTTP(S). The intention is not for you to host you Lightbus configuration on the public internet! . Rather, and if you wish, you may find is useful to host your configuration on an internal-only endpoint. Alternatively, you may decide to ensure your build/deploy process distributes a copy of the global configuration file to every service. The discussion on monorepositories in Architecture Tips is relevant here.","title":"Configuration loading over HTTP(S)"},{"location":"explanation/events/","text":"Events \u00b6 Firing an event will place the event onto the bus and return immediately. No information is provided as to whether the event was processed, or indeed of it was received by any other service at all. No return value is provided when firing an event. This is useful when: You wish to allow non- authoritative services to receive information without needing to concern yourself with their implementation You wish the authoritative service to perform a known task in the background The quickstart provides an example of the latter case. At-least-once semantics \u00b6 Delivering a message exactly once is Very Difficult. Delivering a message at-most-once, or at-least-once is much more practical. Lightbus therefore provides at-least-once delivery for events . As a result you can assume your event listeners will always receive an event, but sometimes a listener may be called multiple times for the same event. You can handle this by ensuring your event listeners are idempotent. That is, implement your event listeners in such a way that it doesn't matter how many times they are executed. See how to write idempotent event handlers . Service names & listener names \u00b6 An event will be delivered once to each consumer group . A consumer group is identified by a name in the form: # Consumer group naming format {service_name}-{listener_name} The service name is specified in your service-level configuration . The listener name is setup when you create your event listener (see below). For example, this bus module sets up two listeners. Each listener is given a listener_name , thereby ensuring each listener receives a copy of every competitor_prices.changed event. # Example of setting service and listener names from my_handlers import send_price_alerts , update_db bus = lightbus . create ( service_name = 'price-monitor' , ) # Consumer group name will be: price-monitor-send-price-alerts bus . competitor_prices . changed . listen ( send_price_alerts , listener_name = \"send_price_alerts\" , ) # Consumer group name will be: price-monitor-update-db bus . competitor_prices . changed . listen ( update_db , listener_name = \"update_db\" , ) Process names \u00b6 Your service-level configuration also specifies a process name . This is less critical than the service name & listener name pairing described above, but still serves an important function. The process name is used to track which process within a consumer group is dealing with a given event. This is primarily useful when a service runs multiple Lightbus processes, normally as a result of scaling or reliability requirements. The purpose of a process name is twofold: Each process has a per-determined length of time to handle and acknowledge a given event. If this time is exceeded then failure will be assumed, and the event may be picked up by another process. When a Lightbus process starts up it will check for any outstanding events reserved for its process name. In which case it will process these messages first. This can happen if the process was killed prior to acknowledging messages it was processing. Providing there is no timeout, an event will only be delivered to one process per listener within a service . The default process name is a random 4 character string. If left unchanged, then clause 2 (above) will never be triggered (as a process name will not persist between process restarts). Any outstanding messages will always have to wait for the timeout period to expire, at which point the will be picked up by another process. Considerations \u00b6 Events are more complex, you may need maintain state as events are received. The source-of-truth regarding stored state may no longer be clear. Enforcing consistency can become difficult. Events are more robust. Your service will be able to fire events as long as the bus client can connect. Likewise, you service can listen for events until the cows come home. Incoming events may be delayed by problems in other services, but each service should be isolated from those problems. Concepts such as Domain Driven Design and Event Sourcing can help to tackle some of these problems. Best practices \u00b6 You may find some of these best practices & suggestions useful. Just remember that there can be exceptions to every rule. Note See architecture tips for further details. Event naming \u00b6 Name events using the past tense. Use page_viewed , not page_view . Use order_created , not create_order . This can apply to events which are commands as well: Use email_report_requested , not email_report . Where relevant, consider using domain-based naming rather than technical names. For example, use order_placed , not order_created . Use parcel_delivered , not parcel_updated . Parameter values \u00b6 Parameter values should have consistent meaning over time. Use complete datetimes, not 'tomorrow' or '6 days ago'. Decide on the boundaries between your relations. See Decide on Boundaries with the architecture tips section.","title":"Events"},{"location":"explanation/events/#events","text":"Firing an event will place the event onto the bus and return immediately. No information is provided as to whether the event was processed, or indeed of it was received by any other service at all. No return value is provided when firing an event. This is useful when: You wish to allow non- authoritative services to receive information without needing to concern yourself with their implementation You wish the authoritative service to perform a known task in the background The quickstart provides an example of the latter case.","title":"Events"},{"location":"explanation/events/#at-least-once-semantics","text":"Delivering a message exactly once is Very Difficult. Delivering a message at-most-once, or at-least-once is much more practical. Lightbus therefore provides at-least-once delivery for events . As a result you can assume your event listeners will always receive an event, but sometimes a listener may be called multiple times for the same event. You can handle this by ensuring your event listeners are idempotent. That is, implement your event listeners in such a way that it doesn't matter how many times they are executed. See how to write idempotent event handlers .","title":"At-least-once semantics"},{"location":"explanation/events/#service-names-listener-names","text":"An event will be delivered once to each consumer group . A consumer group is identified by a name in the form: # Consumer group naming format {service_name}-{listener_name} The service name is specified in your service-level configuration . The listener name is setup when you create your event listener (see below). For example, this bus module sets up two listeners. Each listener is given a listener_name , thereby ensuring each listener receives a copy of every competitor_prices.changed event. # Example of setting service and listener names from my_handlers import send_price_alerts , update_db bus = lightbus . create ( service_name = 'price-monitor' , ) # Consumer group name will be: price-monitor-send-price-alerts bus . competitor_prices . changed . listen ( send_price_alerts , listener_name = \"send_price_alerts\" , ) # Consumer group name will be: price-monitor-update-db bus . competitor_prices . changed . listen ( update_db , listener_name = \"update_db\" , )","title":"Service names & listener names"},{"location":"explanation/events/#process-names","text":"Your service-level configuration also specifies a process name . This is less critical than the service name & listener name pairing described above, but still serves an important function. The process name is used to track which process within a consumer group is dealing with a given event. This is primarily useful when a service runs multiple Lightbus processes, normally as a result of scaling or reliability requirements. The purpose of a process name is twofold: Each process has a per-determined length of time to handle and acknowledge a given event. If this time is exceeded then failure will be assumed, and the event may be picked up by another process. When a Lightbus process starts up it will check for any outstanding events reserved for its process name. In which case it will process these messages first. This can happen if the process was killed prior to acknowledging messages it was processing. Providing there is no timeout, an event will only be delivered to one process per listener within a service . The default process name is a random 4 character string. If left unchanged, then clause 2 (above) will never be triggered (as a process name will not persist between process restarts). Any outstanding messages will always have to wait for the timeout period to expire, at which point the will be picked up by another process.","title":"Process names"},{"location":"explanation/events/#considerations","text":"Events are more complex, you may need maintain state as events are received. The source-of-truth regarding stored state may no longer be clear. Enforcing consistency can become difficult. Events are more robust. Your service will be able to fire events as long as the bus client can connect. Likewise, you service can listen for events until the cows come home. Incoming events may be delayed by problems in other services, but each service should be isolated from those problems. Concepts such as Domain Driven Design and Event Sourcing can help to tackle some of these problems.","title":"Considerations"},{"location":"explanation/events/#best-practices","text":"You may find some of these best practices & suggestions useful. Just remember that there can be exceptions to every rule. Note See architecture tips for further details.","title":"Best practices"},{"location":"explanation/events/#event-naming","text":"Name events using the past tense. Use page_viewed , not page_view . Use order_created , not create_order . This can apply to events which are commands as well: Use email_report_requested , not email_report . Where relevant, consider using domain-based naming rather than technical names. For example, use order_placed , not order_created . Use parcel_delivered , not parcel_updated .","title":"Event naming"},{"location":"explanation/events/#parameter-values","text":"Parameter values should have consistent meaning over time. Use complete datetimes, not 'tomorrow' or '6 days ago'. Decide on the boundaries between your relations. See Decide on Boundaries with the architecture tips section.","title":"Parameter values"},{"location":"explanation/internal-architecture/","text":"Internal Architecture \u00b6 Lightbus' internal workings are composed of: The user-facing API . This is provided by the BusClient class, which then makes use of the EventClient and RpcResultClient classes. This is a friendly API that issues helpful errors where appropriate. This also orchestrates system startup and shutdown. An internal message queuing system . This includes four separate internal message queues plus the EventDock & RpcResultDock classes. The message queues provide the internal communication medium between the user-facing API and the Lightbus backend. The EventDock & RpcResultDock classes convert these messages into a simplified API for implementation by the transports. The EventDock contains the EventTransport , and the RpcResultDock contains both the RpcTransport and RpcResultTransport . The Event & RPC transports implement Lightbus functionality for a specific backend (e.g. Redis). The main transports shipped with Lightbus are the RedisEventTransport , RedisRpcTransport , and RedisResultTransport . Diagram \u00b6","title":"Internal Architecture"},{"location":"explanation/internal-architecture/#internal-architecture","text":"Lightbus' internal workings are composed of: The user-facing API . This is provided by the BusClient class, which then makes use of the EventClient and RpcResultClient classes. This is a friendly API that issues helpful errors where appropriate. This also orchestrates system startup and shutdown. An internal message queuing system . This includes four separate internal message queues plus the EventDock & RpcResultDock classes. The message queues provide the internal communication medium between the user-facing API and the Lightbus backend. The EventDock & RpcResultDock classes convert these messages into a simplified API for implementation by the transports. The EventDock contains the EventTransport , and the RpcResultDock contains both the RpcTransport and RpcResultTransport . The Event & RPC transports implement Lightbus functionality for a specific backend (e.g. Redis). The main transports shipped with Lightbus are the RedisEventTransport , RedisRpcTransport , and RedisResultTransport .","title":"Internal Architecture"},{"location":"explanation/internal-architecture/#diagram","text":"","title":"Diagram"},{"location":"explanation/lightbus-vs-celery/","text":"Lightbus vs Celery \u00b6 Lightbus was conceived as a result of using Celery to communicate between multiple Python services. Differences in principle \u00b6 Celery is a task queue: A task queue is tightly coupled. The dispatching code must know what needs to be done A task queue typically doesn't return results Lightbus is a bus: A bus provides loose coupling. The dispatching code says what did happen, not what should happen (events) A bus provides bi-directional communication (remote procedure calls) Differences in practice \u00b6 A number of pain points were identified with Celery that Lightbus aims to address. In particular: Single vs multi- service \u2013 Celery is targeted as being a task queue for a service, rather than a means for multiple services to interact. Conceptual overlap \u2013 The mapping between concepts in Celery and the underlying broker (AMQP at the time) is both unclear and overlapping. Lightbus provides a limited set of well defined concepts to avoid this confusion. Non-sane defaults \u2013 Some Celery settings have non-sane defaults, making setup somewhat perilous at times. Lightbus provides sane defaults for most circumstances, and documentation specifically geared to certain use cases ( metrics , event sourcing ) Tight coupling (as discussed above) \u2013 Celery tasks define the action to take, not what happened. Lightbus uses events, which describe happened, and listening services decide the action to take. General feeling \u2013 Celery feels large and opaque, debugging issues was challenging. Lightbus aims to feel lightweight, with clear logging and debugging tools.","title":"Lightbus vs Celery"},{"location":"explanation/lightbus-vs-celery/#lightbus-vs-celery","text":"Lightbus was conceived as a result of using Celery to communicate between multiple Python services.","title":"Lightbus vs Celery"},{"location":"explanation/lightbus-vs-celery/#differences-in-principle","text":"Celery is a task queue: A task queue is tightly coupled. The dispatching code must know what needs to be done A task queue typically doesn't return results Lightbus is a bus: A bus provides loose coupling. The dispatching code says what did happen, not what should happen (events) A bus provides bi-directional communication (remote procedure calls)","title":"Differences in principle"},{"location":"explanation/lightbus-vs-celery/#differences-in-practice","text":"A number of pain points were identified with Celery that Lightbus aims to address. In particular: Single vs multi- service \u2013 Celery is targeted as being a task queue for a service, rather than a means for multiple services to interact. Conceptual overlap \u2013 The mapping between concepts in Celery and the underlying broker (AMQP at the time) is both unclear and overlapping. Lightbus provides a limited set of well defined concepts to avoid this confusion. Non-sane defaults \u2013 Some Celery settings have non-sane defaults, making setup somewhat perilous at times. Lightbus provides sane defaults for most circumstances, and documentation specifically geared to certain use cases ( metrics , event sourcing ) Tight coupling (as discussed above) \u2013 Celery tasks define the action to take, not what happened. Lightbus uses events, which describe happened, and listening services decide the action to take. General feeling \u2013 Celery feels large and opaque, debugging issues was challenging. Lightbus aims to feel lightweight, with clear logging and debugging tools.","title":"Differences in practice"},{"location":"explanation/marshalling/","text":"Lightbus has four stages of data marshalling: Encode / Decode Serialize / Deserialize Validation Deform / Cast An inbound message will go through this process from top to bottom . An outbound message will go through this process from bottom to top . Inbound flow \u00b6 Messages arriving from the bus go through the following stages in order to prepare the data for use: Decode: Decode the incoming data (JSON decoding by default) Deserialise: Convert decoded data into a Message object Validate: Validate the incoming message against the JSON schema available on the bus. Cast: Best effort casting of parameters/results based on the locally available type hinting. This can be disabled with the cast_values configuration option . Outbound flow \u00b6 This is the reverse of the inbound flow. Messages being sent will go through the following process in order to prepare the data for transmission on bus: Deform: Lightbus handles NamedTuples , dataclasses and other classes by converting them into dictionaries. Other common types such as datetimes, Decimals etc are converted into strings. Internally this is referred to as the deform process and is the inverse of the cast process. Validate: Validate the outgoing message against the JSON schema available on the bus. Serialize: Structures the data in a way suitable for the transport. Encode: Converts the data to a form suitable for transmission. This typically means stringifying it, for which lightbus uses JSON encoding by default. About casting \u00b6 Casting is separate from validation, although both rely on type hints. Whereas validation uses a shared bus-wide schema to check data validity, casting uses any Python type hints available in the local codebase to marshall event and RPC parameters into a format useful to the service's developer.","title":"Marshalling"},{"location":"explanation/marshalling/#inbound-flow","text":"Messages arriving from the bus go through the following stages in order to prepare the data for use: Decode: Decode the incoming data (JSON decoding by default) Deserialise: Convert decoded data into a Message object Validate: Validate the incoming message against the JSON schema available on the bus. Cast: Best effort casting of parameters/results based on the locally available type hinting. This can be disabled with the cast_values configuration option .","title":"Inbound flow"},{"location":"explanation/marshalling/#outbound-flow","text":"This is the reverse of the inbound flow. Messages being sent will go through the following process in order to prepare the data for transmission on bus: Deform: Lightbus handles NamedTuples , dataclasses and other classes by converting them into dictionaries. Other common types such as datetimes, Decimals etc are converted into strings. Internally this is referred to as the deform process and is the inverse of the cast process. Validate: Validate the outgoing message against the JSON schema available on the bus. Serialize: Structures the data in a way suitable for the transport. Encode: Converts the data to a form suitable for transmission. This typically means stringifying it, for which lightbus uses JSON encoding by default.","title":"Outbound flow"},{"location":"explanation/marshalling/#about-casting","text":"Casting is separate from validation, although both rely on type hints. Whereas validation uses a shared bus-wide schema to check data validity, casting uses any Python type hints available in the local codebase to marshall event and RPC parameters into a format useful to the service's developer.","title":"About casting"},{"location":"explanation/performance/","text":"Performance \u00b6 Caveats Lightbus has yet to undergo any profiling or optimisation, therefore it is reasonable to expect performance to improve with time. The performance of Lightbus is primarily governed by the transports used. Lightbus currently only ships with Redis support. Individual process performance \u00b6 Simple benchmarking 1 on a 2018 MacBook Pro indicates the following execution times (plugins disabled, schema and validation enabled, no event/RPC parameters): Firing an event: \u2248 1.7ms (\u00b110%) Performing a remote procedure call: \u2248 6.9ms (\u00b110%) Redis performance \u00b6 The Redis server has the potential to be a central bottleneck in Lightbus' performance. You may start to run into these limits if you are sending tens thousands of events per second. In these cases you can either: Scale via Redis by setting up Redis Cluster Scale via Lightbus by specifying a different Redis instance per-API. See configuration See how to modify Lightbus for details on how to run the benchmarks via pytest \u21a9","title":"Performance"},{"location":"explanation/performance/#performance","text":"Caveats Lightbus has yet to undergo any profiling or optimisation, therefore it is reasonable to expect performance to improve with time. The performance of Lightbus is primarily governed by the transports used. Lightbus currently only ships with Redis support.","title":"Performance"},{"location":"explanation/performance/#individual-process-performance","text":"Simple benchmarking 1 on a 2018 MacBook Pro indicates the following execution times (plugins disabled, schema and validation enabled, no event/RPC parameters): Firing an event: \u2248 1.7ms (\u00b110%) Performing a remote procedure call: \u2248 6.9ms (\u00b110%)","title":"Individual process performance"},{"location":"explanation/performance/#redis-performance","text":"The Redis server has the potential to be a central bottleneck in Lightbus' performance. You may start to run into these limits if you are sending tens thousands of events per second. In these cases you can either: Scale via Redis by setting up Redis Cluster Scale via Lightbus by specifying a different Redis instance per-API. See configuration See how to modify Lightbus for details on how to run the benchmarks via pytest \u21a9","title":"Redis performance"},{"location":"explanation/rpcs/","text":"Remote Procedure Calls (RPCs) \u00b6 A remote procedure call is where you call a procedure available on the bus. The sequence of events is: You call the RPC, bus.auth.check_password() An autoratitive process for the auth API handles the request and sends the response. You receive the result Remote Procedure Calls are useful when: You require information from a service 1 You wish to wait until a remote procedure has completed an action You can perform an RPC as follows: support_case = bus . support . case . get ( pk = 123 ) RPCs do not provide a fire and forget mode of operation. Consider using events if you need this feature. At most once semantics \u00b6 Remote procedure calls will be processed at most once. In some situations the call will never be processed, in which case the client will raise a LigutbusTimeout exception. Considerations \u00b6 Whether to use RPCs or events for communication will depend upon your project's particular needs. Some considerations are: RPCs are conceptually simple . You call a procedure and wait for a response. You do not need to store any state locally, you can simply request data on demand (performance considerations aside). RPCs can be fragile . Any errors in the remote service will propagate to the client's service. You should handle these if possible. Their use within a codebase may be non-obvious, leading to poor performance. Lightbus tries to alleviate this somewhat by using the bus.api.method() calling format, making it clear that this is a bus-based operation. This is also achievable with events . However you will need to listen for the events and likely store the data locally. See the events section for further discussion. \u21a9","title":"Remote procedure calls"},{"location":"explanation/rpcs/#remote-procedure-calls-rpcs","text":"A remote procedure call is where you call a procedure available on the bus. The sequence of events is: You call the RPC, bus.auth.check_password() An autoratitive process for the auth API handles the request and sends the response. You receive the result Remote Procedure Calls are useful when: You require information from a service 1 You wish to wait until a remote procedure has completed an action You can perform an RPC as follows: support_case = bus . support . case . get ( pk = 123 ) RPCs do not provide a fire and forget mode of operation. Consider using events if you need this feature.","title":"Remote Procedure Calls (RPCs)"},{"location":"explanation/rpcs/#at-most-once-semantics","text":"Remote procedure calls will be processed at most once. In some situations the call will never be processed, in which case the client will raise a LigutbusTimeout exception.","title":"At most once semantics"},{"location":"explanation/rpcs/#considerations","text":"Whether to use RPCs or events for communication will depend upon your project's particular needs. Some considerations are: RPCs are conceptually simple . You call a procedure and wait for a response. You do not need to store any state locally, you can simply request data on demand (performance considerations aside). RPCs can be fragile . Any errors in the remote service will propagate to the client's service. You should handle these if possible. Their use within a codebase may be non-obvious, leading to poor performance. Lightbus tries to alleviate this somewhat by using the bus.api.method() calling format, making it clear that this is a bus-based operation. This is also achievable with events . However you will need to listen for the events and likely store the data locally. See the events section for further discussion. \u21a9","title":"Considerations"},{"location":"explanation/schema/","text":"Schema \u00b6 Lightbus creates a schema for each of your APIs using the type hints specified on the API. This schema is shared on the bus for consumption by other Lightbus clients. This provides a number of features: The availability of a particular API can be detected by remote clients RPCs, results, and events transmitted on the bus can be validated by both the sender and receiver Tooling can load the schema to provide additional functionality. For example, you can dump your production schema and run your development environment and tests against it. Note that an API's schema will only be available on the bus while there is a worker running to provides it. Once the worker process for an API shuts down the schema on the bus will be cleaned up shortly thereafter. See also \u00b6 See the schema reference section for details on how this works in practice. The schema is created using the JSON schema format, see the schema protocol for details of the transmission format.","title":"Schema"},{"location":"explanation/schema/#schema","text":"Lightbus creates a schema for each of your APIs using the type hints specified on the API. This schema is shared on the bus for consumption by other Lightbus clients. This provides a number of features: The availability of a particular API can be detected by remote clients RPCs, results, and events transmitted on the bus can be validated by both the sender and receiver Tooling can load the schema to provide additional functionality. For example, you can dump your production schema and run your development environment and tests against it. Note that an API's schema will only be available on the bus while there is a worker running to provides it. Once the worker process for an API shuts down the schema on the bus will be cleaned up shortly thereafter.","title":"Schema"},{"location":"explanation/schema/#see-also","text":"See the schema reference section for details on how this works in practice. The schema is created using the JSON schema format, see the schema protocol for details of the transmission format.","title":"See also"},{"location":"explanation/services/","text":"Services \u00b6 A service is one or more processes handling a common task. These processes operate as a tightly-coupled whole. All processes in a service will generally: Share the same API class definitions Moreover, they will normally share the same codebase Create a single instance of the bus client in bus.py using bus = lightbus.create() . For example, your company may have the following: An online store A price monitoring script An image resizing resizing process Each of these would be a service. The store service would have a web process and a Lightbus process. The image resizing service & and price monitoring services would each likely have a Lightbus process only. A simple lightbus deployment could therefore look something like this: In this example the following actions would take place: Django reads from the web service database in order to serve web content The online shop's Lightbus process receives pricing events from the price monitoring service. It updates products in the database using this new pricing data. When the Django app receives an image upload, it performs a RPC to the image resizing service to resize the image 1 . Making the Django process wait for an RPC to respond is probably a bad idea in this case, but it illustrates how it could be done. Using an event (which is fire-and-forget) could be more suitable in reality. \u21a9","title":"Services"},{"location":"explanation/services/#services","text":"A service is one or more processes handling a common task. These processes operate as a tightly-coupled whole. All processes in a service will generally: Share the same API class definitions Moreover, they will normally share the same codebase Create a single instance of the bus client in bus.py using bus = lightbus.create() . For example, your company may have the following: An online store A price monitoring script An image resizing resizing process Each of these would be a service. The store service would have a web process and a Lightbus process. The image resizing service & and price monitoring services would each likely have a Lightbus process only. A simple lightbus deployment could therefore look something like this: In this example the following actions would take place: Django reads from the web service database in order to serve web content The online shop's Lightbus process receives pricing events from the price monitoring service. It updates products in the database using this new pricing data. When the Django app receives an image upload, it performs a RPC to the image resizing service to resize the image 1 . Making the Django process wait for an RPC to respond is probably a bad idea in this case, but it illustrates how it could be done. Using an event (which is fire-and-forget) could be more suitable in reality. \u21a9","title":"Services"},{"location":"explanation/transports/","text":"Transports \u00b6 Transports provide the communications system for Lightbus. There are four types of transport: RPC transports \u2013 sends and consumes RPC calls Result transports \u2013 sends and receives RPC results Event transports \u2013 sends and consumes events Schema transports \u2013 stores and retrieves the bus schema Lightbus ships with a Redis-backed implementation of each of these transports. For configuration details see the transport configuration reference . Lightbus can be configured to use custom transports either globally, or on a per-API level.","title":"Transports"},{"location":"explanation/transports/#transports","text":"Transports provide the communications system for Lightbus. There are four types of transport: RPC transports \u2013 sends and consumes RPC calls Result transports \u2013 sends and receives RPC results Event transports \u2013 sends and consumes events Schema transports \u2013 stores and retrieves the bus schema Lightbus ships with a Redis-backed implementation of each of these transports. For configuration details see the transport configuration reference . Lightbus can be configured to use custom transports either globally, or on a per-API level.","title":"Transports"},{"location":"howto/","text":"Howto overview \u00b6 In this section we address specific problems and common use cases . As with the tutorials we will link to concepts as we go, but the priority here is to provide a clear path to a solution.","title":"Overview"},{"location":"howto/#howto-overview","text":"In this section we address specific problems and common use cases . As with the tutorials we will link to concepts as we go, but the priority here is to provide a clear path to a solution.","title":"Howto overview"},{"location":"howto/access-your-bus-client/","text":"How to access your bus client \u00b6 You create your bus client in your bus module (typically called bus.py ) with the line: # Creating your bus client in your bus.py file import lightbus bus = lightbus . create () However, you will often need to make use of your bus client in other areas of your codebase. For example, you may need to fire an event when a web form is submitted. You can access the bus client in two ways. Method 1: Direct import (recommended) \u00b6 The first approach is to import your bus client directly from your bus module, in the same way you would import anything else in your codebase: # For example from bus import bus # ...or if your service has its own package from my_service.bus import bus You should use this approach in code which is specific to your service (i.e. non-shared/non-library code). This approach is more explicit (good), but hard codes the path to your bus module (bad for shared code). Method 2: get_bus() \u00b6 The second approach uses the lightbus.get_bus() function. This will use the module loading configuration to determine the bus module location. If the bus module has already been imported, then the module's bus attribute will simply be returned. Otherwise the bus module will be imported first. # Anywhere in your codebase import lightbus bus = lightbus . get_bus () This approach is best suited to when you do not know where your bus module will be at runtime. This could be the case when: Writing shared code which is used by multiple services Writing a third party library","title":"Access your bus client"},{"location":"howto/access-your-bus-client/#how-to-access-your-bus-client","text":"You create your bus client in your bus module (typically called bus.py ) with the line: # Creating your bus client in your bus.py file import lightbus bus = lightbus . create () However, you will often need to make use of your bus client in other areas of your codebase. For example, you may need to fire an event when a web form is submitted. You can access the bus client in two ways.","title":"How to access your bus client"},{"location":"howto/access-your-bus-client/#method-1-direct-import-recommended","text":"The first approach is to import your bus client directly from your bus module, in the same way you would import anything else in your codebase: # For example from bus import bus # ...or if your service has its own package from my_service.bus import bus You should use this approach in code which is specific to your service (i.e. non-shared/non-library code). This approach is more explicit (good), but hard codes the path to your bus module (bad for shared code).","title":"Method 1: Direct import (recommended)"},{"location":"howto/access-your-bus-client/#method-2-get_bus","text":"The second approach uses the lightbus.get_bus() function. This will use the module loading configuration to determine the bus module location. If the bus module has already been imported, then the module's bus attribute will simply be returned. Otherwise the bus module will be imported first. # Anywhere in your codebase import lightbus bus = lightbus . get_bus () This approach is best suited to when you do not know where your bus module will be at runtime. This could be the case when: Writing shared code which is used by multiple services Writing a third party library","title":"Method 2: get_bus()"},{"location":"howto/event-sourcing/","text":"How to use Lightbus for event sourcing \u00b6 We won't go into the details of event sourcing here, but we can roughly describe our messaging needs as follows: We are optimising for reliability and completeness, performance is secondary Sent events must be valid Received events must be processed regardless of their validity Event history is very important Note Your needs may not precisely match this scenario, so be prepared to tweak the following configuration to your needs. Global configuration \u00b6 For the above event sourced scenario, a sample Lightbus configuration may look like something like this: # Lightbus config for event sourcing bus : schema : transport : redis : url : \"redis://redis_host:6379/0\" apis : # Here we specify the default for your entire bus, but you could # also specify the config for a specific API by using the API's name # instead of 'default'. default : validate : # Sent (outgoing) events must be valid outgoing : true # Received (incoming) events must be processed # regardless of their validity incoming : false event_transport : redis : url : 'redis://redis_host:6379/0' # Load few events into order to prioritise consistency batch_size : 1 # Do not truncate the event stream. We keep all events # as these events are our source of truth max_stream_length : null # Per-API streams, as we wish to prioritise # ordering (this is the default) stream_use : \"per_api\" Run a single Lightbus worker \u00b6 Running only a single Lightbus worker process (with a specific process name) will ensure messages are processed in order. lightbus run --service-name=example_service --process-name=worker Set process names \u00b6 If your event volume requires multiple workers then ensure you set a deterministic per-process name for each. This will allow restarted workers to immediately pickup any previously claimed messages without needing to wait for a timeout. For example, if you have three Lightbus workers you can start each as follows: lightbus run --service-name=example_service --process-name=worker1 lightbus run --service-name=example_service --process-name=worker2 lightbus run --service-name=example_service --process-name=worker3 Ordering will not be maintained when running multiple workers.","title":"Use Lightbus for event sourcing"},{"location":"howto/event-sourcing/#how-to-use-lightbus-for-event-sourcing","text":"We won't go into the details of event sourcing here, but we can roughly describe our messaging needs as follows: We are optimising for reliability and completeness, performance is secondary Sent events must be valid Received events must be processed regardless of their validity Event history is very important Note Your needs may not precisely match this scenario, so be prepared to tweak the following configuration to your needs.","title":"How to use Lightbus for event sourcing"},{"location":"howto/event-sourcing/#global-configuration","text":"For the above event sourced scenario, a sample Lightbus configuration may look like something like this: # Lightbus config for event sourcing bus : schema : transport : redis : url : \"redis://redis_host:6379/0\" apis : # Here we specify the default for your entire bus, but you could # also specify the config for a specific API by using the API's name # instead of 'default'. default : validate : # Sent (outgoing) events must be valid outgoing : true # Received (incoming) events must be processed # regardless of their validity incoming : false event_transport : redis : url : 'redis://redis_host:6379/0' # Load few events into order to prioritise consistency batch_size : 1 # Do not truncate the event stream. We keep all events # as these events are our source of truth max_stream_length : null # Per-API streams, as we wish to prioritise # ordering (this is the default) stream_use : \"per_api\"","title":"Global configuration"},{"location":"howto/event-sourcing/#run-a-single-lightbus-worker","text":"Running only a single Lightbus worker process (with a specific process name) will ensure messages are processed in order. lightbus run --service-name=example_service --process-name=worker","title":"Run a single Lightbus worker"},{"location":"howto/event-sourcing/#set-process-names","text":"If your event volume requires multiple workers then ensure you set a deterministic per-process name for each. This will allow restarted workers to immediately pickup any previously claimed messages without needing to wait for a timeout. For example, if you have three Lightbus workers you can start each as follows: lightbus run --service-name=example_service --process-name=worker1 lightbus run --service-name=example_service --process-name=worker2 lightbus run --service-name=example_service --process-name=worker3 Ordering will not be maintained when running multiple workers.","title":"Set process names"},{"location":"howto/metrics/","text":"How to use Lightbus for metrics \u00b6 When we talk about metrics we may mean all or any of the following: Current information is most important Previous events will become irrelevant as soon as new data is received Lost events are therefore tolerable, as long as we keep up with new events Events may be high volume, so optimisations may be needed Note Your needs may not precisely match this scenario, so be prepared to tweak the following configuration to your needs. For the above metrics-based scenario, a sample Lightbus configuration may look like something like this: # Lightbus config for metrics bus : schema : transport : redis : url : \"redis://redis_host:6379/0\" apis : # Here we specify the default for your entire bus, but you could # also specify the config for a specific API by using the API's name # instead of 'default'. default : # Disable validation to enhance performance validate : outgoing : false incoming : false # Assume we will be transmitting simple types, so we can bypass casting for performance cast_values : false event_transport : redis : url : 'redis://redis_host:6379/0' # Load in many events at once for performance improvements batch_size : 100 # No need to keep many historical events around max_stream_length : 10000 # Per-event streams, to allow selective consumption of metrics stream_use : \"per_event\"","title":"Use Lightbus for realtime metrics"},{"location":"howto/metrics/#how-to-use-lightbus-for-metrics","text":"When we talk about metrics we may mean all or any of the following: Current information is most important Previous events will become irrelevant as soon as new data is received Lost events are therefore tolerable, as long as we keep up with new events Events may be high volume, so optimisations may be needed Note Your needs may not precisely match this scenario, so be prepared to tweak the following configuration to your needs. For the above metrics-based scenario, a sample Lightbus configuration may look like something like this: # Lightbus config for metrics bus : schema : transport : redis : url : \"redis://redis_host:6379/0\" apis : # Here we specify the default for your entire bus, but you could # also specify the config for a specific API by using the API's name # instead of 'default'. default : # Disable validation to enhance performance validate : outgoing : false incoming : false # Assume we will be transmitting simple types, so we can bypass casting for performance cast_values : false event_transport : redis : url : 'redis://redis_host:6379/0' # Load in many events at once for performance improvements batch_size : 100 # No need to keep many historical events around max_stream_length : 10000 # Per-event streams, to allow selective consumption of metrics stream_use : \"per_event\"","title":"How to use Lightbus for metrics"},{"location":"howto/migrate-from-celery-to-lightbus/","text":"How to migrate from Celery to Lightbus \u00b6 Migration from Celery to Lightbus can be a very straightforward process depending on how you currently make use of celery. However, before you start out you should take time to familiarise yourself with the principles behind Lightbus, and its differences to Celery: Lightbus anatomy lesson Lightbus vs Celery Queued tasks \u00b6 Celery is often used to schedule background tasks for later processing. For example, a web server may schedule a task to send a welcome email to a new user. This task will go into your Celery queue and be handled by a Celery worker process. The web process does not need any response from the worker, it only needs to know that it will be handled. In Lightbus these are events . We can convert this to Lightbus as follows: # bus.py import lightbus # Create an API for everything user-related class UserApi ( lightbus . Api ): # We specify the event which has occurred (a user signed up) user_signed_up = lightbus . Event ( parameters = ( 'email' , 'username' )) class Meta : # Name our API # (used below to setup our listener) name = \"user\" # Create our bus client (this is a requirement of our bus.py file) bus = lightbus . create () # Register the API (see docs: Explanation -> APIs) bus . client . register_api ( UserApi ()) @bus . client . on_start () def bus_start ( ** kwargs ): from my_app.somewhere import send_welcome_email # Listen for this event being fired. # When the event is fired, send_welcome_email will be called as follows: # send_welcome_email(event, email, username) bus . user . user_signed_up . listen ( send_welcome_email , listener_name = \"send_welcome_email\" ) Then, within our web server code we can send an email as follows: # Your web server code # Import the bus client from your bus.py file from bus import bus # Your registration view @route ( '/register' , methods = ( 'GET' , 'POST' )) def register (): if request . method == 'POST' : # ...validate and create user... # Send registration email via the Lightbus process bus . user . user_signed_up . fire ( username = request . form [ 'username' ], email = request . form [ 'email' ] ) else : # ... This will fire an event on the bus, and our lightbus run process will receive this event and send the welcome email. Fetching Celery task results \u00b6 Sometimes you wish to have a Celery worker process execute a task for you, and then return the result. For example, perhaps you need to get the URL to a resized user avatar image. Image resizing is computationally expensive, so you prefer to do this on your Celery workers. You want the worker to do this task for you, and the web server will wait for it to be done, and the web server needs a response (the image URL) from the worker. In Lightbus these are remote procedure calls (RPCs). We can implement the above example as follows: # bus.py (as above, with one addition) import lightbus class UserApi ( lightbus . Api ): user_signed_up = lightbus . Event ( parameters = ( \"email\" , \"username\" )) class Meta : name = \"user\" # This is our new remote procedure def get_avatar_url ( self , username , width , height ): # ... do some resizing & uploading ... return f \"https://example.com/images/ { width } / { height } / { username } .png\" bus = lightbus . create () bus . client . register_api ( UserApi ()) @bus . client . on_start () def bus_start ( ** kwargs ): from my_app.somewhere import send_welcome_email bus . user . user_signed_up . listen ( send_welcome_email , listener_name = \"send_welcome_email\" ) We can then use this within our web server as follows: # Your web server code # Import the bus client from your bus.py file from bus import bus # Your registration view @route ( '/user-profile/' , methods = ( 'GET' ,)) def user_profile ( username ): # Call our new remote procedure image_url = bus . user . get_avatar_url ( username = username , width = 500 , height = 500 ) return f \"

{ username }

\" Now, as long as our lightbus run process is running, the web server will be able to call the remote procedure and render the user profile HTML. Scheduled tasks \u00b6 Lightbus can replace Celery's periodic tasks through the use of the @bus.client.every() and/or @bus.client.schedule() decorators (see how to schedule recurring tasks ). For example: # bus.py import lightbus bus = lightbus . create () @bus . client . every ( hours = 1 ) def every_hour (): # Will be called every hour refresh_customer_data () The lightbus run command will then execute this function every hour. See how to schedule recurring tasks for more details on the available scheduling options.","title":"Migrate from Celery to Lightbus"},{"location":"howto/migrate-from-celery-to-lightbus/#how-to-migrate-from-celery-to-lightbus","text":"Migration from Celery to Lightbus can be a very straightforward process depending on how you currently make use of celery. However, before you start out you should take time to familiarise yourself with the principles behind Lightbus, and its differences to Celery: Lightbus anatomy lesson Lightbus vs Celery","title":"How to migrate from Celery to Lightbus"},{"location":"howto/migrate-from-celery-to-lightbus/#queued-tasks","text":"Celery is often used to schedule background tasks for later processing. For example, a web server may schedule a task to send a welcome email to a new user. This task will go into your Celery queue and be handled by a Celery worker process. The web process does not need any response from the worker, it only needs to know that it will be handled. In Lightbus these are events . We can convert this to Lightbus as follows: # bus.py import lightbus # Create an API for everything user-related class UserApi ( lightbus . Api ): # We specify the event which has occurred (a user signed up) user_signed_up = lightbus . Event ( parameters = ( 'email' , 'username' )) class Meta : # Name our API # (used below to setup our listener) name = \"user\" # Create our bus client (this is a requirement of our bus.py file) bus = lightbus . create () # Register the API (see docs: Explanation -> APIs) bus . client . register_api ( UserApi ()) @bus . client . on_start () def bus_start ( ** kwargs ): from my_app.somewhere import send_welcome_email # Listen for this event being fired. # When the event is fired, send_welcome_email will be called as follows: # send_welcome_email(event, email, username) bus . user . user_signed_up . listen ( send_welcome_email , listener_name = \"send_welcome_email\" ) Then, within our web server code we can send an email as follows: # Your web server code # Import the bus client from your bus.py file from bus import bus # Your registration view @route ( '/register' , methods = ( 'GET' , 'POST' )) def register (): if request . method == 'POST' : # ...validate and create user... # Send registration email via the Lightbus process bus . user . user_signed_up . fire ( username = request . form [ 'username' ], email = request . form [ 'email' ] ) else : # ... This will fire an event on the bus, and our lightbus run process will receive this event and send the welcome email.","title":"Queued tasks"},{"location":"howto/migrate-from-celery-to-lightbus/#fetching-celery-task-results","text":"Sometimes you wish to have a Celery worker process execute a task for you, and then return the result. For example, perhaps you need to get the URL to a resized user avatar image. Image resizing is computationally expensive, so you prefer to do this on your Celery workers. You want the worker to do this task for you, and the web server will wait for it to be done, and the web server needs a response (the image URL) from the worker. In Lightbus these are remote procedure calls (RPCs). We can implement the above example as follows: # bus.py (as above, with one addition) import lightbus class UserApi ( lightbus . Api ): user_signed_up = lightbus . Event ( parameters = ( \"email\" , \"username\" )) class Meta : name = \"user\" # This is our new remote procedure def get_avatar_url ( self , username , width , height ): # ... do some resizing & uploading ... return f \"https://example.com/images/ { width } / { height } / { username } .png\" bus = lightbus . create () bus . client . register_api ( UserApi ()) @bus . client . on_start () def bus_start ( ** kwargs ): from my_app.somewhere import send_welcome_email bus . user . user_signed_up . listen ( send_welcome_email , listener_name = \"send_welcome_email\" ) We can then use this within our web server as follows: # Your web server code # Import the bus client from your bus.py file from bus import bus # Your registration view @route ( '/user-profile/' , methods = ( 'GET' ,)) def user_profile ( username ): # Call our new remote procedure image_url = bus . user . get_avatar_url ( username = username , width = 500 , height = 500 ) return f \"

{ username }

\" Now, as long as our lightbus run process is running, the web server will be able to call the remote procedure and render the user profile HTML.","title":"Fetching Celery task results"},{"location":"howto/migrate-from-celery-to-lightbus/#scheduled-tasks","text":"Lightbus can replace Celery's periodic tasks through the use of the @bus.client.every() and/or @bus.client.schedule() decorators (see how to schedule recurring tasks ). For example: # bus.py import lightbus bus = lightbus . create () @bus . client . every ( hours = 1 ) def every_hour (): # Will be called every hour refresh_customer_data () The lightbus run command will then execute this function every hour. See how to schedule recurring tasks for more details on the available scheduling options.","title":"Scheduled tasks"},{"location":"howto/modify-lightbus/","text":"How to modify Lightbus \u00b6 Contributions to Lightbus are very welcome. This will talk you though setting up a development installation of Lightbus. Using this installation you will be able to: Modify the Lightbus source code and/or documentation Run the Lightbus test suite View any modified documentation locally Use your development Lightbus install within another project Prerequisites \u00b6 You will need: Redis running locally Getting the code \u00b6 Checkout the Lightbus repository from GitHub: git clone https://github.com/adamcharnock/lightbus.git cd lightbus Environment setup \u00b6 It is a good idea to put asyncio into debug mode. You can do this by setting the following in your shell's environment: PYTHONASYNCIODEBUG=1 The testing framework will also need to know where your redis instance is running. This is set using the REDIS_URL and REDIS_URL_B environment variables: # Default values shown below REDIS_URL=redis://127.0.0.1:6379/10 REDIS_URL_B=redis://127.0.0.1:6379/11 Installation \u00b6 You will need to install Lightbus' standard dependencies, as well as Lightbus' development dependencies. Note that you may need to install poetry if you do not already have it: You can install both of these groups as follows: # Install standard & dev dependencies into a virtual environment poetry install # Enter the virtual environment you have created, # thereby giving you access the the pytest and mkdocs commands (below) poetry shell Running the tests \u00b6 You can run the tests once you have completed the above steps: pytest Note that you can run subsets of the tests as follows: pytest -m unit # Fast with high coverage pytest -m integration pytest -m reliability pytest -m benchmark Viewing the Lightbus documentation locally \u00b6 You can view the documentation of your local Lightbus install as follows: # Serve the docs locally mkdocs serve You can now view the documentation http://127.0.0.1:8000 . The documentation source can be found in docs/ . You can also check for broken links within the docs by running the check_links.sh script: # Check for broken links ./docs/check_links.sh Using within your project \u00b6 You can install your development Lightbus install within your project as follows: # Within your own project # Make sure you remove any existing lightbus version pip uninstall lightbus # Install your local development lightbus pip install --editable /path/to/your/local/lightbus See also \u00b6 Being familiar with the explanation section is highly recommended if modifying the Lightbus source","title":"Modify Lightbus"},{"location":"howto/modify-lightbus/#how-to-modify-lightbus","text":"Contributions to Lightbus are very welcome. This will talk you though setting up a development installation of Lightbus. Using this installation you will be able to: Modify the Lightbus source code and/or documentation Run the Lightbus test suite View any modified documentation locally Use your development Lightbus install within another project","title":"How to modify Lightbus"},{"location":"howto/modify-lightbus/#prerequisites","text":"You will need: Redis running locally","title":"Prerequisites"},{"location":"howto/modify-lightbus/#getting-the-code","text":"Checkout the Lightbus repository from GitHub: git clone https://github.com/adamcharnock/lightbus.git cd lightbus","title":"Getting the code"},{"location":"howto/modify-lightbus/#environment-setup","text":"It is a good idea to put asyncio into debug mode. You can do this by setting the following in your shell's environment: PYTHONASYNCIODEBUG=1 The testing framework will also need to know where your redis instance is running. This is set using the REDIS_URL and REDIS_URL_B environment variables: # Default values shown below REDIS_URL=redis://127.0.0.1:6379/10 REDIS_URL_B=redis://127.0.0.1:6379/11","title":"Environment setup"},{"location":"howto/modify-lightbus/#installation","text":"You will need to install Lightbus' standard dependencies, as well as Lightbus' development dependencies. Note that you may need to install poetry if you do not already have it: You can install both of these groups as follows: # Install standard & dev dependencies into a virtual environment poetry install # Enter the virtual environment you have created, # thereby giving you access the the pytest and mkdocs commands (below) poetry shell","title":"Installation"},{"location":"howto/modify-lightbus/#running-the-tests","text":"You can run the tests once you have completed the above steps: pytest Note that you can run subsets of the tests as follows: pytest -m unit # Fast with high coverage pytest -m integration pytest -m reliability pytest -m benchmark","title":"Running the tests"},{"location":"howto/modify-lightbus/#viewing-the-lightbus-documentation-locally","text":"You can view the documentation of your local Lightbus install as follows: # Serve the docs locally mkdocs serve You can now view the documentation http://127.0.0.1:8000 . The documentation source can be found in docs/ . You can also check for broken links within the docs by running the check_links.sh script: # Check for broken links ./docs/check_links.sh","title":"Viewing the Lightbus documentation locally"},{"location":"howto/modify-lightbus/#using-within-your-project","text":"You can install your development Lightbus install within your project as follows: # Within your own project # Make sure you remove any existing lightbus version pip uninstall lightbus # Install your local development lightbus pip install --editable /path/to/your/local/lightbus","title":"Using within your project"},{"location":"howto/modify-lightbus/#see-also","text":"Being familiar with the explanation section is highly recommended if modifying the Lightbus source","title":"See also"},{"location":"howto/run-background-tasks/","text":"How to run background tasks \u00b6 Sometimes you may wish to run arbitrary asyncio tasks in the background of the lightbus run process. You can set these up in your bus.py file: # bus.py import asyncio import lightbus bus = lightbus . create () async def my_background_task (): while True : await asyncio . sleep ( 1 ) print ( \"Hello!\" ) @bus . client . on_start () def on_startup ( ** kwargs ): bus . client . add_background_task ( my_background_task ()) Important points to note are: The background task will be automatically cancelled when the bus is closed. Any errors in the background task will be bubbled up and cause the Lightbus process to exit. If this is not desired you can implement your own try/except handling within the function being executed. Note If you wish to schedule a recurring task then you should probably use @bus.client.every() or @bus.client.schedule() . See how to schedule recurring tasks .","title":"Run background tasks"},{"location":"howto/run-background-tasks/#how-to-run-background-tasks","text":"Sometimes you may wish to run arbitrary asyncio tasks in the background of the lightbus run process. You can set these up in your bus.py file: # bus.py import asyncio import lightbus bus = lightbus . create () async def my_background_task (): while True : await asyncio . sleep ( 1 ) print ( \"Hello!\" ) @bus . client . on_start () def on_startup ( ** kwargs ): bus . client . add_background_task ( my_background_task ()) Important points to note are: The background task will be automatically cancelled when the bus is closed. Any errors in the background task will be bubbled up and cause the Lightbus process to exit. If this is not desired you can implement your own try/except handling within the function being executed. Note If you wish to schedule a recurring task then you should probably use @bus.client.every() or @bus.client.schedule() . See how to schedule recurring tasks .","title":"How to run background tasks"},{"location":"howto/schedule-recurring-tasks/","text":"How to schedule recurring tasks \u00b6 Recurring tasks can be scheduled in two ways: The @bus.client.every() decorator \u2013 Will execute a function or coroutine at a given interval The @bus.client.schedule() decorator \u2013 Similar to every() , but takes complex schedules as provided by the schedule library. Simple recurring tasks using @bus.client.every() \u00b6 Lightbus natively supports simple recurring tasks using the @bus.client.every() decorator: # bus.py import lightbus bus = lightbus . create () @bus . client . every ( seconds = 1 ) def do_it (): print ( \"Hello!\" ) The interval can be specified using the seconds , minutes , hours , and days keys. Pass also_run_immediately=True to execute the function/coroutine immediately, as well as at the given interval. Complex schedules using @bus.client.schedule() \u00b6 Lightbus also supports using schedules specified using the schedule library. This allows for schedules such as 'every Monday at 1am', rather than simple intervals. For example: import lightbus import schedule bus = lightbus . create () # Run the task every 1-3 seconds, varying randomly @bus . client . schedule ( schedule . every ( 1 ) . to ( 3 ) . seconds ) def do_it (): print ( \"Hello using schedule library\" ) Long running tasks \u00b6 If your tasks are long running you may prefer to handle these in a separate Lightbus process. This will avoid blocking the processing of incoming events and RPCs. For example, the default RPC timeout is 5 seconds. Any task which runs for longer than this has the possibility of causing incoming RPCs to timeout. You can move your task processing to a separate process as follows: # Process 1: Handles scheduled tasks only lightbus run --only tasks # Process 2: Handles everything else (events and rpcs) lightbus run --skip tasks","title":"Schedule recurring tasks"},{"location":"howto/schedule-recurring-tasks/#how-to-schedule-recurring-tasks","text":"Recurring tasks can be scheduled in two ways: The @bus.client.every() decorator \u2013 Will execute a function or coroutine at a given interval The @bus.client.schedule() decorator \u2013 Similar to every() , but takes complex schedules as provided by the schedule library.","title":"How to schedule recurring tasks"},{"location":"howto/schedule-recurring-tasks/#simple-recurring-tasks-using-busclientevery","text":"Lightbus natively supports simple recurring tasks using the @bus.client.every() decorator: # bus.py import lightbus bus = lightbus . create () @bus . client . every ( seconds = 1 ) def do_it (): print ( \"Hello!\" ) The interval can be specified using the seconds , minutes , hours , and days keys. Pass also_run_immediately=True to execute the function/coroutine immediately, as well as at the given interval.","title":"Simple recurring tasks using @bus.client.every()"},{"location":"howto/schedule-recurring-tasks/#complex-schedules-using-busclientschedule","text":"Lightbus also supports using schedules specified using the schedule library. This allows for schedules such as 'every Monday at 1am', rather than simple intervals. For example: import lightbus import schedule bus = lightbus . create () # Run the task every 1-3 seconds, varying randomly @bus . client . schedule ( schedule . every ( 1 ) . to ( 3 ) . seconds ) def do_it (): print ( \"Hello using schedule library\" )","title":"Complex schedules using @bus.client.schedule()"},{"location":"howto/schedule-recurring-tasks/#long-running-tasks","text":"If your tasks are long running you may prefer to handle these in a separate Lightbus process. This will avoid blocking the processing of incoming events and RPCs. For example, the default RPC timeout is 5 seconds. Any task which runs for longer than this has the possibility of causing incoming RPCs to timeout. You can move your task processing to a separate process as follows: # Process 1: Handles scheduled tasks only lightbus run --only tasks # Process 2: Handles everything else (events and rpcs) lightbus run --skip tasks","title":"Long running tasks"},{"location":"howto/write-idempotent-event-handlers/","text":"How to write idempotent event handlers \u00b6 An idempotent function is a function which, when called multiple times with the same parameters, will not result in any change after the first call. As Wikipedia puts it : Idempotence is the property of certain operations in mathematics and computer science whereby they can be applied multiple times without changing the result beyond the initial application Why is idempotency useful? \u00b6 Consider that Lightbus events will be delivered at least once . This is in comparison to RPCs which are called at most once . This is not Lightbus trying to be awkward, rather it is because it is theoretically impossible to ensure exactly-once delivery. Lightbus therefore guarantees you that events will always arrive, but with the trade-off that sometimes the event may arrive multiple times. In the wost case, your event handlers will be executed multiple times. In some cases this will not cause a problem. For example, an event handler which performs a simple update to a record with a specific ID. Executing this update a second time will leave the record in the same state it was in after the first execution. However, an event handler which creates a new record will face a problem. If the event handler is executed twice then two records will be created, even though the event was identical in both cases. Ensuring idempotency for database operations \u00b6 A simple and reliable way to ensure idempotency is through the use of UUIDs and upserts . A UUID is a standard way of producing a globally unique ID. An upsert is a combination of an 'update' and an 'insert' which says: *'Insert this record, but if you find a duplicate then update it instead'. Take the following event handler as an example: # BAD, not idempotent def handle_page_view ( event , url ): db . execute ( \"INSERT INTO views (url)\" ) This is not an idempotent event handler because multiple invocations for the same event will result in multiple records being created. An idempotent version of the above can be rewritten as: # GOOD, idempotent def handle_page_view ( event , uuid , url ): try : # Try to perform an insert first (this should normally work fine) db . execute ( \"INSERT INTO views ( %s , %s )\" , [ uuid , url ]) except IntegrityError : # UUID already exists, so do an update db . execute ( \"UPDATE views SET url = %s WHERE view_uuid = %s \" , [ url , uuid ]) This handler can be executed any number of times for the same parameters without ever creating duplicate records. It is idempotent. Note that the views table must include a UUID column which is UNIQUE , otherwise an error would not be raised. Additionally, the code which fires the event must now provide a value for the UUID parameter. Many databases and frameworks provide direct support for performing upsert operations which will make the above simpler (and therefore less error prone). Upserts in Django \u00b6 Django provides the update_or_create() method which serves exactly this purpose: # Idempotent def handle_page_view ( event , uuid , url ): Views . objects . update_or_create ( uuid = uuid , defaults = { 'url' : url }) Upserts in SQL \u00b6 Many relational databases can also perform upsert in a single SQL query: INSERT INTO views ( uuid , url ) VALUES ( 'uuid-here' , 'http://lightbus.org' ) ON CONFLICT ( uuid ) DO UPDATE SET url = 'http://lightbus.org' ; Ensuring idempotency in other cases \u00b6 There are additional areas where idempotency can become relevant: Event handlers which call external APIs Event handlers which send email Handling these cases is beyond the scope of this documentation. Rest APIs which support idempotency keys will likely help here 1 . You may also decide that multiple invocations are acceptable in some situations as long as they are sufficiently rare (for example, sending emails). Blog post from Stripe: Designing robust and predictable APIs with idempotency \u21a9","title":"Write idempotent event handlers"},{"location":"howto/write-idempotent-event-handlers/#how-to-write-idempotent-event-handlers","text":"An idempotent function is a function which, when called multiple times with the same parameters, will not result in any change after the first call. As Wikipedia puts it : Idempotence is the property of certain operations in mathematics and computer science whereby they can be applied multiple times without changing the result beyond the initial application","title":"How to write idempotent event handlers"},{"location":"howto/write-idempotent-event-handlers/#why-is-idempotency-useful","text":"Consider that Lightbus events will be delivered at least once . This is in comparison to RPCs which are called at most once . This is not Lightbus trying to be awkward, rather it is because it is theoretically impossible to ensure exactly-once delivery. Lightbus therefore guarantees you that events will always arrive, but with the trade-off that sometimes the event may arrive multiple times. In the wost case, your event handlers will be executed multiple times. In some cases this will not cause a problem. For example, an event handler which performs a simple update to a record with a specific ID. Executing this update a second time will leave the record in the same state it was in after the first execution. However, an event handler which creates a new record will face a problem. If the event handler is executed twice then two records will be created, even though the event was identical in both cases.","title":"Why is idempotency useful?"},{"location":"howto/write-idempotent-event-handlers/#ensuring-idempotency-for-database-operations","text":"A simple and reliable way to ensure idempotency is through the use of UUIDs and upserts . A UUID is a standard way of producing a globally unique ID. An upsert is a combination of an 'update' and an 'insert' which says: *'Insert this record, but if you find a duplicate then update it instead'. Take the following event handler as an example: # BAD, not idempotent def handle_page_view ( event , url ): db . execute ( \"INSERT INTO views (url)\" ) This is not an idempotent event handler because multiple invocations for the same event will result in multiple records being created. An idempotent version of the above can be rewritten as: # GOOD, idempotent def handle_page_view ( event , uuid , url ): try : # Try to perform an insert first (this should normally work fine) db . execute ( \"INSERT INTO views ( %s , %s )\" , [ uuid , url ]) except IntegrityError : # UUID already exists, so do an update db . execute ( \"UPDATE views SET url = %s WHERE view_uuid = %s \" , [ url , uuid ]) This handler can be executed any number of times for the same parameters without ever creating duplicate records. It is idempotent. Note that the views table must include a UUID column which is UNIQUE , otherwise an error would not be raised. Additionally, the code which fires the event must now provide a value for the UUID parameter. Many databases and frameworks provide direct support for performing upsert operations which will make the above simpler (and therefore less error prone).","title":"Ensuring idempotency for database operations"},{"location":"howto/write-idempotent-event-handlers/#upserts-in-django","text":"Django provides the update_or_create() method which serves exactly this purpose: # Idempotent def handle_page_view ( event , uuid , url ): Views . objects . update_or_create ( uuid = uuid , defaults = { 'url' : url })","title":"Upserts in Django"},{"location":"howto/write-idempotent-event-handlers/#upserts-in-sql","text":"Many relational databases can also perform upsert in a single SQL query: INSERT INTO views ( uuid , url ) VALUES ( 'uuid-here' , 'http://lightbus.org' ) ON CONFLICT ( uuid ) DO UPDATE SET url = 'http://lightbus.org' ;","title":"Upserts in SQL"},{"location":"howto/write-idempotent-event-handlers/#ensuring-idempotency-in-other-cases","text":"There are additional areas where idempotency can become relevant: Event handlers which call external APIs Event handlers which send email Handling these cases is beyond the scope of this documentation. Rest APIs which support idempotency keys will likely help here 1 . You may also decide that multiple invocations are acceptable in some situations as long as they are sufficiently rare (for example, sending emails). Blog post from Stripe: Designing robust and predictable APIs with idempotency \u21a9","title":"Ensuring idempotency in other cases"},{"location":"includes/if-you-get-stuck/","text":"If you get stuck... It is really useful to hear from people who have encountered a problem or got stuck. Hearing from you means we can improve our documentation and error messages. If you get stuck drop then please drop an email to adam@adamcharnock.com , visit the Lightbus discord server , or call me (Adam) on +442032896620. The more information you can include the better (problem description, screenshots, and code are all useful)","title":"If you get stuck"},{"location":"includes/note-configuration-auto-complete/","text":"Configuration auto-complete using JSON Schema Many code editors support using a JSON schema to provide auto-complete and validation when editing a JSON file. If you wish, you can write your configuration in JSON (rather than YAML), and load the following JSON schema into your editor: https://lightbus.org/static/default-config-schema.json This will provide you with autocomplete and validation for Lightbus' various configuration options. If you are using custom transports or plugins you should generate your own config schema .","title":"Note configuration auto complete"},{"location":"reference/","text":"Reference overview \u00b6 This section provides detailed information regarding the specific features of Lightbus . A grasp of the tutorial and explantion sections will be useful here.","title":"Overview"},{"location":"reference/#reference-overview","text":"This section provides detailed information regarding the specific features of Lightbus . A grasp of the tutorial and explantion sections will be useful here.","title":"Reference overview"},{"location":"reference/apis/","text":"APIs specify the functionality available on the bus. To do this you define API classes within your bus.py file. You can also define your API elsewhere and import it into your bus.py file. For further discussion of APIs see the concepts section . An example API \u00b6 # An example API. You can define this in your bus.py, # or import into your bus.py file from elsewhere class SupportCaseApi ( Api ): # An event, # available at bus.support.case.case_created case_created = Event ( parameters = ( 'id' , 'sender' , 'subject' , 'body' )) # Options for this API class Meta : # API name on the bus name = 'support.case' # Will be available as a remote procedure call at # bus.support.case.get() def get ( self , id ): return get_case_from_db ( pk = id ) A service can define zero or more APIs, and each API can contain zero or more events and zero or more procedures. The Meta class specifies options regarding the API, with name being the only required option. The name specifies how the API will be accessed on the bus. You could call an RPC on the above API as follows: bus = lightbus . create () # Call the get_case() RPC. case = bus . support . case . get_case ( id = 123 ) You can also fire an event on this API: bus = lightbus . create () # Fire the case_created event bus . support . case . case_created . fire ( id = 123 , sender = 'Joe' , subject = 'I need support please!' , body = '...' , ) Options \u00b6 name (str) \u00b6 Specifies the name of the API. This will determine how the API is addressed on the bus. See naming , below. name is a required option. Naming your APIs \u00b6 As you can from the Meta.name option in the example above, API names can contain periods which allow you to structure your bus in a suitable form for your situation. Some example API naming schemes may look like: # Example API naming schemes for use within Meta.name Format : Example : support.get_case() support.get_activity() Format : . Example : support.case.get() support.activity.get() Format : .. Example : marketing.website.stats.get() ops.monitoring.servers.get_status()","title":"APIs"},{"location":"reference/apis/#an-example-api","text":"# An example API. You can define this in your bus.py, # or import into your bus.py file from elsewhere class SupportCaseApi ( Api ): # An event, # available at bus.support.case.case_created case_created = Event ( parameters = ( 'id' , 'sender' , 'subject' , 'body' )) # Options for this API class Meta : # API name on the bus name = 'support.case' # Will be available as a remote procedure call at # bus.support.case.get() def get ( self , id ): return get_case_from_db ( pk = id ) A service can define zero or more APIs, and each API can contain zero or more events and zero or more procedures. The Meta class specifies options regarding the API, with name being the only required option. The name specifies how the API will be accessed on the bus. You could call an RPC on the above API as follows: bus = lightbus . create () # Call the get_case() RPC. case = bus . support . case . get_case ( id = 123 ) You can also fire an event on this API: bus = lightbus . create () # Fire the case_created event bus . support . case . case_created . fire ( id = 123 , sender = 'Joe' , subject = 'I need support please!' , body = '...' , )","title":"An example API"},{"location":"reference/apis/#options","text":"","title":"Options"},{"location":"reference/apis/#name-str","text":"Specifies the name of the API. This will determine how the API is addressed on the bus. See naming , below. name is a required option.","title":"name (str)"},{"location":"reference/apis/#naming-your-apis","text":"As you can from the Meta.name option in the example above, API names can contain periods which allow you to structure your bus in a suitable form for your situation. Some example API naming schemes may look like: # Example API naming schemes for use within Meta.name Format : Example : support.get_case() support.get_activity() Format : . Example : support.case.get() support.activity.get() Format : .. Example : marketing.website.stats.get() ops.monitoring.servers.get_status()","title":"Naming your APIs"},{"location":"reference/authors/","text":"Authors \u00b6 Current team \u00b6 Adam Charnock , Portugal (Maintainer) Alumni \u00b6 None yet... Thanks \u00b6 Lightbus would not have been possible without the intellectual, emotional, and logistical support of the following individuals and companies: Louis Thibault \u2013 For helping Adam work through many of the knottier problems, and relentlessly cheering him on. Futurepump \u2013 For being an early testing ground and ongoing user of Lightbus. Louis Pilfold \u2013 For being an early sounding board, and persuading Adam to take schemas seriously. Presscast \u2013 For being an early adopter and proving invaluable feedback. Additional thanks to everyone (technical and non-technical) who kindly listened to Adam talk about the reasons and ideas behind Lightbus.","title":"Authors"},{"location":"reference/authors/#authors","text":"","title":"Authors"},{"location":"reference/authors/#current-team","text":"Adam Charnock , Portugal (Maintainer)","title":"Current team"},{"location":"reference/authors/#alumni","text":"None yet...","title":"Alumni"},{"location":"reference/authors/#thanks","text":"Lightbus would not have been possible without the intellectual, emotional, and logistical support of the following individuals and companies: Louis Thibault \u2013 For helping Adam work through many of the knottier problems, and relentlessly cheering him on. Futurepump \u2013 For being an early testing ground and ongoing user of Lightbus. Louis Pilfold \u2013 For being an early sounding board, and persuading Adam to take schemas seriously. Presscast \u2013 For being an early adopter and proving invaluable feedback. Additional thanks to everyone (technical and non-technical) who kindly listened to Adam talk about the reasons and ideas behind Lightbus.","title":"Thanks"},{"location":"reference/code-of-conduct/","text":"Code of Conduct \u00b6 Our Pledge \u00b6 We as members, contributors, and leaders pledge to make participation in our community a harassment-free experience for everyone, regardless of age, body size, visible or invisible disability, ethnicity, sex characteristics, gender identity and expression, level of experience, education, socio-economic status, nationality, personal appearance, race, religion, or sexual identity and orientation. We pledge to act and interact in ways that contribute to an open, welcoming, diverse, inclusive, and healthy community. Our Standards \u00b6 Examples of behavior that contributes to a positive environment for our community include: Demonstrating empathy and kindness toward other people Being respectful of differing opinions, viewpoints, and experiences Giving and gracefully accepting constructive feedback Accepting responsibility and apologizing to those affected by our mistakes, and learning from the experience Focusing on what is best not just for us as individuals, but for the overall community Examples of unacceptable behavior include: The use of sexualized language or imagery, and sexual attention or advances of any kind Trolling, insulting or derogatory comments, and personal or political attacks Public or private harassment Publishing others' private information, such as a physical or email address, without their explicit permission Other conduct which could reasonably be considered inappropriate in a professional setting Enforcement Responsibilities \u00b6 Community leaders are responsible for clarifying and enforcing our standards of acceptable behavior and will take appropriate and fair corrective action in response to any behavior that they deem inappropriate, threatening, offensive, or harmful. Community leaders have the right and responsibility to remove, edit, or reject comments, commits, code, wiki edits, issues, and other contributions that are not aligned to this Code of Conduct, and will communicate reasons for moderation decisions when appropriate. Scope \u00b6 This Code of Conduct applies within all community spaces, and also applies when an individual is officially representing the community in public spaces. Examples of representing our community include using an official e-mail address, posting via an official social media account, or acting as an appointed representative at an online or offline event. Enforcement \u00b6 Instances of abusive, harassing, or otherwise unacceptable behavior may be reported to the community leaders responsible for enforcement at adam@adamcharnock.com . All complaints will be reviewed and investigated promptly and fairly. All community leaders are obligated to respect the privacy and security of the reporter of any incident. Enforcement Guidelines \u00b6 Community leaders will follow these Community Impact Guidelines in determining the consequences for any action they deem in violation of this Code of Conduct: 1. Correction \u00b6 Community Impact : Use of inappropriate language or other behavior deemed unprofessional or unwelcome in the community. Consequence : A private, written warning from community leaders, providing clarity around the nature of the violation and an explanation of why the behavior was inappropriate. A public apology may be requested. 2. Warning \u00b6 Community Impact : A violation through a single incident or series of actions. Consequence : A warning with consequences for continued behavior. No interaction with the people involved, including unsolicited interaction with those enforcing the Code of Conduct, for a specified period of time. This includes avoiding interactions in community spaces as well as external channels like social media. Violating these terms may lead to a temporary or permanent ban. 3. Temporary Ban \u00b6 Community Impact : A serious violation of community standards, including sustained inappropriate behavior. Consequence : A temporary ban from any sort of interaction or public communication with the community for a specified period of time. No public or private interaction with the people involved, including unsolicited interaction with those enforcing the Code of Conduct, is allowed during this period. Violating these terms may lead to a permanent ban. 4. Permanent Ban \u00b6 Community Impact : Demonstrating a pattern of violation of community standards, including sustained inappropriate behavior, harassment of an individual, or aggression toward or disparagement of classes of individuals. Consequence : A permanent ban from any sort of public interaction within the project community. Attribution \u00b6 This Code of Conduct is adapted from the Contributor Covenant , version 2.0, available at https://www.contributor-covenant.org/version/2/0/code_of_conduct.html . Community Impact Guidelines were inspired by Mozilla's code of conduct enforcement ladder . For answers to common questions about this code of conduct, see the FAQ at https://www.contributor-covenant.org/faq . Translations are available at https://www.contributor-covenant.org/translations . For answers to common questions about this code of conduct, see https://www.contributor-covenant.org/faq","title":"Code of Conduct"},{"location":"reference/code-of-conduct/#code-of-conduct","text":"","title":"Code of Conduct"},{"location":"reference/code-of-conduct/#our-pledge","text":"We as members, contributors, and leaders pledge to make participation in our community a harassment-free experience for everyone, regardless of age, body size, visible or invisible disability, ethnicity, sex characteristics, gender identity and expression, level of experience, education, socio-economic status, nationality, personal appearance, race, religion, or sexual identity and orientation. We pledge to act and interact in ways that contribute to an open, welcoming, diverse, inclusive, and healthy community.","title":"Our Pledge"},{"location":"reference/code-of-conduct/#our-standards","text":"Examples of behavior that contributes to a positive environment for our community include: Demonstrating empathy and kindness toward other people Being respectful of differing opinions, viewpoints, and experiences Giving and gracefully accepting constructive feedback Accepting responsibility and apologizing to those affected by our mistakes, and learning from the experience Focusing on what is best not just for us as individuals, but for the overall community Examples of unacceptable behavior include: The use of sexualized language or imagery, and sexual attention or advances of any kind Trolling, insulting or derogatory comments, and personal or political attacks Public or private harassment Publishing others' private information, such as a physical or email address, without their explicit permission Other conduct which could reasonably be considered inappropriate in a professional setting","title":"Our Standards"},{"location":"reference/code-of-conduct/#enforcement-responsibilities","text":"Community leaders are responsible for clarifying and enforcing our standards of acceptable behavior and will take appropriate and fair corrective action in response to any behavior that they deem inappropriate, threatening, offensive, or harmful. Community leaders have the right and responsibility to remove, edit, or reject comments, commits, code, wiki edits, issues, and other contributions that are not aligned to this Code of Conduct, and will communicate reasons for moderation decisions when appropriate.","title":"Enforcement Responsibilities"},{"location":"reference/code-of-conduct/#scope","text":"This Code of Conduct applies within all community spaces, and also applies when an individual is officially representing the community in public spaces. Examples of representing our community include using an official e-mail address, posting via an official social media account, or acting as an appointed representative at an online or offline event.","title":"Scope"},{"location":"reference/code-of-conduct/#enforcement","text":"Instances of abusive, harassing, or otherwise unacceptable behavior may be reported to the community leaders responsible for enforcement at adam@adamcharnock.com . All complaints will be reviewed and investigated promptly and fairly. All community leaders are obligated to respect the privacy and security of the reporter of any incident.","title":"Enforcement"},{"location":"reference/code-of-conduct/#enforcement-guidelines","text":"Community leaders will follow these Community Impact Guidelines in determining the consequences for any action they deem in violation of this Code of Conduct:","title":"Enforcement Guidelines"},{"location":"reference/code-of-conduct/#1-correction","text":"Community Impact : Use of inappropriate language or other behavior deemed unprofessional or unwelcome in the community. Consequence : A private, written warning from community leaders, providing clarity around the nature of the violation and an explanation of why the behavior was inappropriate. A public apology may be requested.","title":"1. Correction"},{"location":"reference/code-of-conduct/#2-warning","text":"Community Impact : A violation through a single incident or series of actions. Consequence : A warning with consequences for continued behavior. No interaction with the people involved, including unsolicited interaction with those enforcing the Code of Conduct, for a specified period of time. This includes avoiding interactions in community spaces as well as external channels like social media. Violating these terms may lead to a temporary or permanent ban.","title":"2. Warning"},{"location":"reference/code-of-conduct/#3-temporary-ban","text":"Community Impact : A serious violation of community standards, including sustained inappropriate behavior. Consequence : A temporary ban from any sort of interaction or public communication with the community for a specified period of time. No public or private interaction with the people involved, including unsolicited interaction with those enforcing the Code of Conduct, is allowed during this period. Violating these terms may lead to a permanent ban.","title":"3. Temporary Ban"},{"location":"reference/code-of-conduct/#4-permanent-ban","text":"Community Impact : Demonstrating a pattern of violation of community standards, including sustained inappropriate behavior, harassment of an individual, or aggression toward or disparagement of classes of individuals. Consequence : A permanent ban from any sort of public interaction within the project community.","title":"4. Permanent Ban"},{"location":"reference/code-of-conduct/#attribution","text":"This Code of Conduct is adapted from the Contributor Covenant , version 2.0, available at https://www.contributor-covenant.org/version/2/0/code_of_conduct.html . Community Impact Guidelines were inspired by Mozilla's code of conduct enforcement ladder . For answers to common questions about this code of conduct, see the FAQ at https://www.contributor-covenant.org/faq . Translations are available at https://www.contributor-covenant.org/translations . For answers to common questions about this code of conduct, see https://www.contributor-covenant.org/faq","title":"Attribution"},{"location":"reference/configuration/","text":"Configuration \u00b6 Lightbus' configuration happens in three stages: Module loading \u2013 Lightbus discovers where your bus.py file can found via the LIGHTBUS_MODULE environment variable. Service-level configuration \u2013 Your bus.py file specifies service-level settings ( service_name and process_name ) Global bus configuration \u2013 Your bus.py provides the location to the global config for your bus. This can be a local file path, or a HTTP(S) URL. See the configuration explanation for further discussion and reasoning around this approach. Configuration auto-complete using JSON Schema Many code editors support using a JSON schema to provide auto-complete and validation when editing a JSON file. If you wish, you can write your configuration in JSON (rather than YAML), and load the following JSON schema into your editor: https://lightbus.org/static/default-config-schema.json This will provide you with autocomplete and validation for Lightbus' various configuration options. If you are using custom transports or plugins you should generate your own config schema . 1. Module loading \u00b6 The first stage in Lightbus' startup is to import your bus.py module. By default Lightbus will attempt to import a module named bus , but you can modify this by specifying the LIGHTBUS_MODULE environment variable. This stage is only required when starting a Lightbus worker process (i.e. lightbus run ). Non-lightbus processes will import the bus module manually in order to access the bus client within (see next stage, below). Note See anatomy lesson for further discusison of the distinction between processes. 2. Service-level configuration \u00b6 The bus module discovered in the module loading stage ( above ) must define a Lightbus client as follows: # Must be in your bus.py file bus = lightbus . create () The above statement serves several purposes: The lightbus run command will use this client to access the bus. You can (and should) import this client elsewhere in the service in order to call RPCs and fire events (see how to access your bus client ). You can configure service-level configuration options for your Lightbus client. Service-level configuration is different to your global configuration because the values will vary between services. The following service-level options are available: # Available service-level configuration options bus = lightbus . create ( # Relevant to event consumption service_name = 'my_service' , # Will be replaced 4 random characters. Default process_name = ' {random4} ' , # Path to the global bus config. # Can be .yaml or .json, and http(s) URLs are supported. config = '/path/to/lightbus.yaml' , # Features to enable. Default is to enable all features. # Can be configured using the --skip and --only arguments features = [ 'rpcs' , 'events' , 'tasks' ], ) The above configuration options can also be set using the following environment variables or command line arguments: Configuration option Environment Variable Command line argument Notes Service name LIGHTBUS_SERVICE_NAME --service-name See service name explanation Process name LIGHTBUS_PROCESS_NAME --process-name See process name explanation Configuration path LIGHTBUS_CONFIG --config Path or URL to global configuration yaml/json file. See global bus configuration . Features LIGHTBUS_FEATURES --only , --skip Features to enable/disable. Comma separated list of rpcs , events , tasks Service & process name placeholders \u00b6 The following placeholders may be used within your service and process name values: Paceholder Example value Notes {hostname} my-host Lower case hostname {pid} 12345 The process' ID {random4} abcd Random 4-character string {random8} abcdefgh Random 8-character string {random16} abcdefghijklmnop Random 16-character string {friendly} delicate-wave-915 Human-friendly random name Event delivery \u00b6 See the events explanation section for a discussion on how service & process names affect event delivery. 3. Global bus configuration \u00b6 The global bus configuration specifies the bus' overall architecture. This configuration takes the form of a YAML or JSON file, and accordingly the filename should end with .yaml or .json . This file is typically shared by all lightbus clients and can be specified as a path on disk, or as a HTTP(S) URL ( see explanation ). A basic default configuration file is as follows: # Root configuration bus : # Bus configuration schema : # Schema configuration transport : # Transport selector config redis : url : \"redis://redis.svc.cluster.local:6379/0\" apis : # API configuration listing default : # Api config event_transport : # Transport selector configuration redis : url : \"redis://redis.svc.cluster.local:6379/0\" rpc_transport : # Transport selector configuration redis : url : \"redis://redis.svc.cluster.local:6379/0\" result_transport : # Transport selector configuration redis : url : \"redis://redis.svc.cluster.local:6379/0\" plugins : # Plugin configuring listing # These plugins ship with Lightbus internal_metrics : # Plugin configuration # Note that plugins are disabled by default, and must be enabled here # in your bus' global configuration enabled : true internal_state : enabled : true ping_enabled : true ping_interval : 60 Each section is detailed below. Root config \u00b6 The available root-level configuration keys are: bus \u2013 Contains the bus config apis \u2013 Contains the API configuration listing plugins - Contains the plugin configuration listing The following keys are also present, but should generally not be specified in your global yaml file . Instead they should be specified for each service, as per the service-level setup : service_name \u2013 Service name process_name \u2013 Process name Bus config \u00b6 The bus config resides under the root config . It contains the following keys: log_level (default: info ) - The log level for the lightbus logger. One of debug , info , warning , error , critical . info is a good level for development purposes, warning will be more suited to production. schema - Contains the schema config Schema config \u00b6 The schema config resides under the bus config . human_readable (default: True ) \u2013 Should the schema JSON be transmitted with human-friendly indentation and spacing? ttl (default: 60 ) \u2013 Integer number of seconds that an API schema should live on the bus. Each schema will be pinged every ttl * 0.8 seconds in order to keep it alive. The bus will also check for new remote schemas every ttl * 0.8 seconds. transport \u2013 Contains the schema transport selector API configuration listing \u00b6 The API configuration listing config resides under the root config . This is a key/value association between APIs and their configurations. The reserved API name of default provides a catch-all configuration for any APIs without specific configurations. Specifically configured APIs do not inherit from the default configuration. For example: ... apis : # Catch-all api config default : # See 'API config' ... default config as above ... # Specific config for the 'marketing.analytics' API. # Use a different Redis instance for the high # volume marketing analytics marketing.analytics : # See 'API config' validate : false cast_values : false event_transport : redis : url : \"redis://redis-marketing.svc.cluster.local:6379/0\" rpc_transport : redis : url : \"redis://redis-marketing.svc.cluster.local:6379/0\" result_transport : redis : url : \"redis://redis-marketing.svc.cluster.local:6379/0\" See API config for further details on the API options available. API config \u00b6 The API config resides under the API configuration listing . APIs are configured using the options below: rpc_timeout (default: 5 ) \u2013 Timeout when calling RPCs on this API (must also be specified on the RPC and result transport) event_listener_setup_timeout (default: 1 ) \u2013 Timeout seconds when setting up event listeners (only applies when using the blocking api) event_fire_timeout (default: 1 ) \u2013 Timeout seconds when firing events on the bus (only applies when using the blocking api) validate \u2013 Contains the api validation config . May also be set to boolean true or false to blanket enable/disable. strict_validation (default: false ) \u2013 Raise an exception if we receive a message from an API for which there is no schema available on the bus. If false a warning will be emitted instead. event_transport \u2013 Contains the transport selector . rpc_transport \u2013 Contains the transport selector . result_transport \u2013 Contains the transport selector . cast_values (default: true ) \u2013 If enabled, incoming values will be best-effort casted based on the annotations of the RPC method signature or event listener. See typing . Transport selector \u00b6 The schema config resides under both the API config and the schema config . Transports are specified as follows: ... parent yaml ... [transport-name]: option: \"value\" option: \"value\" Where [transport-name] can be one of: redis \u2013 The redis-backed transport. debug \u2013 A debug transport which logs what happens but takes no further action. The table below details which transports can be used in which situations Transport RPC Result Event Schema redis \u2714 \u2714 \u2714 \u2714 debug \u2714 \u2714 \u2714 \u2714 Additional transports may be added in future. A single API can use different types for each of its rpc , result , and event needs. The schema transport is global to the bus, and is not configurable on a per-api level. For configuration details see the transport configuration reference . API validation config \u00b6 The schema config resides under the validate key within the API config . Available options are: outgoing (default true ) \u2013 Validate outgoing messages against any available API schema. incoming (default true ) \u2013 Validate incoming messages against any available API schema. A warning will be emitted if validation is enabled and the schema is not present on the bus. You can turn this into an error by enabling strict_validation within the API config . Plugin configuration listing \u00b6 The plugin configuration listing config resides under the root config . This configuration section is a mapping between plugin names and their configuration options. Plugins are made available to Lightbus via the lightbus_plugins entry point (see Lightbus' pyproject.toml for an example). As a result, installing a plugin should be sufficient for it to be made available for configuration. Plugin developers should see the plugins reference for further details. Plugin configuration \u00b6 The Plugin configuration resides under the Plugin configuration listing . This section provides the configuration for an individual plugin. The available configuration options will vary from plugin to plugin, so you should read the plugin's documentation for details. However, all plugins support the enabled property, which defaults to false . You must there set enabled: true if you wish to use the plugin. For example: bus : ... apis : ... plugins : internal_metrics : # Plugin configuration enabled : true","title":"Configuration"},{"location":"reference/configuration/#configuration","text":"Lightbus' configuration happens in three stages: Module loading \u2013 Lightbus discovers where your bus.py file can found via the LIGHTBUS_MODULE environment variable. Service-level configuration \u2013 Your bus.py file specifies service-level settings ( service_name and process_name ) Global bus configuration \u2013 Your bus.py provides the location to the global config for your bus. This can be a local file path, or a HTTP(S) URL. See the configuration explanation for further discussion and reasoning around this approach. Configuration auto-complete using JSON Schema Many code editors support using a JSON schema to provide auto-complete and validation when editing a JSON file. If you wish, you can write your configuration in JSON (rather than YAML), and load the following JSON schema into your editor: https://lightbus.org/static/default-config-schema.json This will provide you with autocomplete and validation for Lightbus' various configuration options. If you are using custom transports or plugins you should generate your own config schema .","title":"Configuration"},{"location":"reference/configuration/#1-module-loading","text":"The first stage in Lightbus' startup is to import your bus.py module. By default Lightbus will attempt to import a module named bus , but you can modify this by specifying the LIGHTBUS_MODULE environment variable. This stage is only required when starting a Lightbus worker process (i.e. lightbus run ). Non-lightbus processes will import the bus module manually in order to access the bus client within (see next stage, below). Note See anatomy lesson for further discusison of the distinction between processes.","title":"1. Module loading"},{"location":"reference/configuration/#2-service-level-configuration","text":"The bus module discovered in the module loading stage ( above ) must define a Lightbus client as follows: # Must be in your bus.py file bus = lightbus . create () The above statement serves several purposes: The lightbus run command will use this client to access the bus. You can (and should) import this client elsewhere in the service in order to call RPCs and fire events (see how to access your bus client ). You can configure service-level configuration options for your Lightbus client. Service-level configuration is different to your global configuration because the values will vary between services. The following service-level options are available: # Available service-level configuration options bus = lightbus . create ( # Relevant to event consumption service_name = 'my_service' , # Will be replaced 4 random characters. Default process_name = ' {random4} ' , # Path to the global bus config. # Can be .yaml or .json, and http(s) URLs are supported. config = '/path/to/lightbus.yaml' , # Features to enable. Default is to enable all features. # Can be configured using the --skip and --only arguments features = [ 'rpcs' , 'events' , 'tasks' ], ) The above configuration options can also be set using the following environment variables or command line arguments: Configuration option Environment Variable Command line argument Notes Service name LIGHTBUS_SERVICE_NAME --service-name See service name explanation Process name LIGHTBUS_PROCESS_NAME --process-name See process name explanation Configuration path LIGHTBUS_CONFIG --config Path or URL to global configuration yaml/json file. See global bus configuration . Features LIGHTBUS_FEATURES --only , --skip Features to enable/disable. Comma separated list of rpcs , events , tasks","title":"2. Service-level configuration"},{"location":"reference/configuration/#service-process-name-placeholders","text":"The following placeholders may be used within your service and process name values: Paceholder Example value Notes {hostname} my-host Lower case hostname {pid} 12345 The process' ID {random4} abcd Random 4-character string {random8} abcdefgh Random 8-character string {random16} abcdefghijklmnop Random 16-character string {friendly} delicate-wave-915 Human-friendly random name","title":"Service & process name placeholders"},{"location":"reference/configuration/#event-delivery","text":"See the events explanation section for a discussion on how service & process names affect event delivery.","title":"Event delivery"},{"location":"reference/configuration/#3-global-bus-configuration","text":"The global bus configuration specifies the bus' overall architecture. This configuration takes the form of a YAML or JSON file, and accordingly the filename should end with .yaml or .json . This file is typically shared by all lightbus clients and can be specified as a path on disk, or as a HTTP(S) URL ( see explanation ). A basic default configuration file is as follows: # Root configuration bus : # Bus configuration schema : # Schema configuration transport : # Transport selector config redis : url : \"redis://redis.svc.cluster.local:6379/0\" apis : # API configuration listing default : # Api config event_transport : # Transport selector configuration redis : url : \"redis://redis.svc.cluster.local:6379/0\" rpc_transport : # Transport selector configuration redis : url : \"redis://redis.svc.cluster.local:6379/0\" result_transport : # Transport selector configuration redis : url : \"redis://redis.svc.cluster.local:6379/0\" plugins : # Plugin configuring listing # These plugins ship with Lightbus internal_metrics : # Plugin configuration # Note that plugins are disabled by default, and must be enabled here # in your bus' global configuration enabled : true internal_state : enabled : true ping_enabled : true ping_interval : 60 Each section is detailed below.","title":"3. Global bus configuration"},{"location":"reference/configuration/#root-config","text":"The available root-level configuration keys are: bus \u2013 Contains the bus config apis \u2013 Contains the API configuration listing plugins - Contains the plugin configuration listing The following keys are also present, but should generally not be specified in your global yaml file . Instead they should be specified for each service, as per the service-level setup : service_name \u2013 Service name process_name \u2013 Process name","title":"Root config"},{"location":"reference/configuration/#bus-config","text":"The bus config resides under the root config . It contains the following keys: log_level (default: info ) - The log level for the lightbus logger. One of debug , info , warning , error , critical . info is a good level for development purposes, warning will be more suited to production. schema - Contains the schema config","title":"Bus config"},{"location":"reference/configuration/#schema-config","text":"The schema config resides under the bus config . human_readable (default: True ) \u2013 Should the schema JSON be transmitted with human-friendly indentation and spacing? ttl (default: 60 ) \u2013 Integer number of seconds that an API schema should live on the bus. Each schema will be pinged every ttl * 0.8 seconds in order to keep it alive. The bus will also check for new remote schemas every ttl * 0.8 seconds. transport \u2013 Contains the schema transport selector","title":"Schema config"},{"location":"reference/configuration/#api-configuration-listing","text":"The API configuration listing config resides under the root config . This is a key/value association between APIs and their configurations. The reserved API name of default provides a catch-all configuration for any APIs without specific configurations. Specifically configured APIs do not inherit from the default configuration. For example: ... apis : # Catch-all api config default : # See 'API config' ... default config as above ... # Specific config for the 'marketing.analytics' API. # Use a different Redis instance for the high # volume marketing analytics marketing.analytics : # See 'API config' validate : false cast_values : false event_transport : redis : url : \"redis://redis-marketing.svc.cluster.local:6379/0\" rpc_transport : redis : url : \"redis://redis-marketing.svc.cluster.local:6379/0\" result_transport : redis : url : \"redis://redis-marketing.svc.cluster.local:6379/0\" See API config for further details on the API options available.","title":"API configuration listing"},{"location":"reference/configuration/#api-config","text":"The API config resides under the API configuration listing . APIs are configured using the options below: rpc_timeout (default: 5 ) \u2013 Timeout when calling RPCs on this API (must also be specified on the RPC and result transport) event_listener_setup_timeout (default: 1 ) \u2013 Timeout seconds when setting up event listeners (only applies when using the blocking api) event_fire_timeout (default: 1 ) \u2013 Timeout seconds when firing events on the bus (only applies when using the blocking api) validate \u2013 Contains the api validation config . May also be set to boolean true or false to blanket enable/disable. strict_validation (default: false ) \u2013 Raise an exception if we receive a message from an API for which there is no schema available on the bus. If false a warning will be emitted instead. event_transport \u2013 Contains the transport selector . rpc_transport \u2013 Contains the transport selector . result_transport \u2013 Contains the transport selector . cast_values (default: true ) \u2013 If enabled, incoming values will be best-effort casted based on the annotations of the RPC method signature or event listener. See typing .","title":"API config"},{"location":"reference/configuration/#transport-selector","text":"The schema config resides under both the API config and the schema config . Transports are specified as follows: ... parent yaml ... [transport-name]: option: \"value\" option: \"value\" Where [transport-name] can be one of: redis \u2013 The redis-backed transport. debug \u2013 A debug transport which logs what happens but takes no further action. The table below details which transports can be used in which situations Transport RPC Result Event Schema redis \u2714 \u2714 \u2714 \u2714 debug \u2714 \u2714 \u2714 \u2714 Additional transports may be added in future. A single API can use different types for each of its rpc , result , and event needs. The schema transport is global to the bus, and is not configurable on a per-api level. For configuration details see the transport configuration reference .","title":"Transport selector"},{"location":"reference/configuration/#api-validation-config","text":"The schema config resides under the validate key within the API config . Available options are: outgoing (default true ) \u2013 Validate outgoing messages against any available API schema. incoming (default true ) \u2013 Validate incoming messages against any available API schema. A warning will be emitted if validation is enabled and the schema is not present on the bus. You can turn this into an error by enabling strict_validation within the API config .","title":"API validation config"},{"location":"reference/configuration/#plugin-configuration-listing","text":"The plugin configuration listing config resides under the root config . This configuration section is a mapping between plugin names and their configuration options. Plugins are made available to Lightbus via the lightbus_plugins entry point (see Lightbus' pyproject.toml for an example). As a result, installing a plugin should be sufficient for it to be made available for configuration. Plugin developers should see the plugins reference for further details.","title":"Plugin configuration listing"},{"location":"reference/configuration/#plugin-configuration","text":"The Plugin configuration resides under the Plugin configuration listing . This section provides the configuration for an individual plugin. The available configuration options will vary from plugin to plugin, so you should read the plugin's documentation for details. However, all plugins support the enabled property, which defaults to false . You must there set enabled: true if you wish to use the plugin. For example: bus : ... apis : ... plugins : internal_metrics : # Plugin configuration enabled : true","title":"Plugin configuration"},{"location":"reference/events/","text":"Events are defined as properties on your API classes. Events are useful when: You need asynchronous communication between services You wish to loosely-couple your services You need a service to perform a task in the background See event considerations in the explanations section for further discussion. Events provide at-least-once delivery semantics. Given this, your event handlers should be idempotent . Defining events \u00b6 You can define events using the Event class. For example, you could define the following bus.py in your authenication service: # auth_service/bus.py from lightbus import Api , Event class AuthApi ( Api ): user_created = Event ( parameters = ( 'username' , 'email' )) user_updated = Event ( parameters = ( 'username' , 'new_email' )) user_deleted = Event ( parameters = ( 'username' )) class Meta : name = 'auth' Firing events \u00b6 You can fire events as follows: # Anywhere in your code # Import your project's bus instance from bus import bus bus . auth . user_created . fire ( username = 'adam' , password = 'adam@example.com' ) Firing events (asynchronously) \u00b6 You can also fire events asynchronously using asyncio: # Anywhere in your code # Import your project's bus instance from bus import bus await bus . auth . user_created . fire_async ( username = 'adam' , password = 'adam@example.com' ) Listening for events \u00b6 Listening for events is typically a long-running background activity, and is therefore dealt with by the lightbus run command. You can setup event listeners in your services' bus module as follows: # other_service/bus.py import lightbus bus = lightbus . create () user_db = {} def handle_created ( username , email ): user_db [ username ] = email print ( user_db ) def handle_updated ( username , email ): user_db [ username ] = email print ( user_db ) def handle_deleted ( username , email ): user_db . pop ( username ) print ( user_db ) @bus . client . on_start () def on_start (): # Bus client has started up, so register our listeners bus . auth . user_created . listen ( handle_created , listener_name = \"user_created\" ) bus . auth . user_updated . listen ( handle_updated , listener_name = \"user_updated\" , ) bus . auth . user_deleted . listen ( handle_deleted , listener_name = \"user_deleted\" ) Specifying listener_name is required in order to ensure each listeners receives all events it is due. See the events explanation page for further discussion. You cannot have two listeners with the same name for the same API. For example, this will not work (it will raise a DuplicateListenerName exception: bus . competitor_prices . changed . listen ( send_price_alerts , listener_name = \"price_listener\" , ) ### ERROR ## # This will raise a DuplicateListenerName exception because # we have already created a listener named 'price_listener' # on the 'competitor_prices' API (above). bus . competitor_prices . changed . listen ( update_prices , listener_name = \"price_listener\" , # \u21e0 DuplicateListenerName exception ) This restriction only applies to listeners within the same service. Errors in event listeners \u00b6 By default, the Lightbus worker will exit if an event listener encounters an error. The event being handled will remain unacknowledged, and will be re-attempted after acknowledgement_timeout seconds. You can customise this behaviour by passing the on_error parameter to listen() . Possible values are: lightbus.OnError.SHUTDOWN (Default) - Shutdown the Lightbus work upon encountering an error lightbus.OnError.ACKNOWLEDGE_AND_LOG (Default) - Acknowledge erroring messages as being successfully processed, and log the error. For example: # Taken from the above example # Worker shutdown on error (the default) bus . auth . user_created . listen ( handle_created , listener_name = \"user_created\" , on_error = lightbus . OnError . SHUTDOWN , ) # Or, log the error and acknowlegde the message as being processed bus . auth . user_created . listen ( handle_created , listener_name = \"user_created\" , on_error = lightbus . OnError . ACKNOWLEDGE_AND_LOG , ) Listening for events (asynchronous handlers) \u00b6 Event handlers may also be asynchronous. For example: import lightbus import asyncio bus = lightbus . create () # Asynchronous handles can be used too async def handle_created ( username , email ): await asyncio . sleep ( 0.1 ) print ( f \"Username { username } created\" ) @bus . client . on_start () def on_start (): # Bus client has started up, so register our listeners bus . auth . user_created . listen ( handle_created , listener_name = \"user_created\" ) A note on threads Asynchronous handlers will be executed in the same thread as Lightbus, whereas synchronous handlers will be executed in their own thread. Type hints \u00b6 See the typing reference .","title":"Events"},{"location":"reference/events/#defining-events","text":"You can define events using the Event class. For example, you could define the following bus.py in your authenication service: # auth_service/bus.py from lightbus import Api , Event class AuthApi ( Api ): user_created = Event ( parameters = ( 'username' , 'email' )) user_updated = Event ( parameters = ( 'username' , 'new_email' )) user_deleted = Event ( parameters = ( 'username' )) class Meta : name = 'auth'","title":"Defining events"},{"location":"reference/events/#firing-events","text":"You can fire events as follows: # Anywhere in your code # Import your project's bus instance from bus import bus bus . auth . user_created . fire ( username = 'adam' , password = 'adam@example.com' )","title":"Firing events"},{"location":"reference/events/#firing-events-asynchronously","text":"You can also fire events asynchronously using asyncio: # Anywhere in your code # Import your project's bus instance from bus import bus await bus . auth . user_created . fire_async ( username = 'adam' , password = 'adam@example.com' )","title":"Firing events (asynchronously)"},{"location":"reference/events/#listening-for-events","text":"Listening for events is typically a long-running background activity, and is therefore dealt with by the lightbus run command. You can setup event listeners in your services' bus module as follows: # other_service/bus.py import lightbus bus = lightbus . create () user_db = {} def handle_created ( username , email ): user_db [ username ] = email print ( user_db ) def handle_updated ( username , email ): user_db [ username ] = email print ( user_db ) def handle_deleted ( username , email ): user_db . pop ( username ) print ( user_db ) @bus . client . on_start () def on_start (): # Bus client has started up, so register our listeners bus . auth . user_created . listen ( handle_created , listener_name = \"user_created\" ) bus . auth . user_updated . listen ( handle_updated , listener_name = \"user_updated\" , ) bus . auth . user_deleted . listen ( handle_deleted , listener_name = \"user_deleted\" ) Specifying listener_name is required in order to ensure each listeners receives all events it is due. See the events explanation page for further discussion. You cannot have two listeners with the same name for the same API. For example, this will not work (it will raise a DuplicateListenerName exception: bus . competitor_prices . changed . listen ( send_price_alerts , listener_name = \"price_listener\" , ) ### ERROR ## # This will raise a DuplicateListenerName exception because # we have already created a listener named 'price_listener' # on the 'competitor_prices' API (above). bus . competitor_prices . changed . listen ( update_prices , listener_name = \"price_listener\" , # \u21e0 DuplicateListenerName exception ) This restriction only applies to listeners within the same service.","title":"Listening for events"},{"location":"reference/events/#errors-in-event-listeners","text":"By default, the Lightbus worker will exit if an event listener encounters an error. The event being handled will remain unacknowledged, and will be re-attempted after acknowledgement_timeout seconds. You can customise this behaviour by passing the on_error parameter to listen() . Possible values are: lightbus.OnError.SHUTDOWN (Default) - Shutdown the Lightbus work upon encountering an error lightbus.OnError.ACKNOWLEDGE_AND_LOG (Default) - Acknowledge erroring messages as being successfully processed, and log the error. For example: # Taken from the above example # Worker shutdown on error (the default) bus . auth . user_created . listen ( handle_created , listener_name = \"user_created\" , on_error = lightbus . OnError . SHUTDOWN , ) # Or, log the error and acknowlegde the message as being processed bus . auth . user_created . listen ( handle_created , listener_name = \"user_created\" , on_error = lightbus . OnError . ACKNOWLEDGE_AND_LOG , )","title":"Errors in event listeners"},{"location":"reference/events/#listening-for-events-asynchronous-handlers","text":"Event handlers may also be asynchronous. For example: import lightbus import asyncio bus = lightbus . create () # Asynchronous handles can be used too async def handle_created ( username , email ): await asyncio . sleep ( 0.1 ) print ( f \"Username { username } created\" ) @bus . client . on_start () def on_start (): # Bus client has started up, so register our listeners bus . auth . user_created . listen ( handle_created , listener_name = \"user_created\" ) A note on threads Asynchronous handlers will be executed in the same thread as Lightbus, whereas synchronous handlers will be executed in their own thread.","title":"Listening for events (asynchronous handlers)"},{"location":"reference/events/#type-hints","text":"See the typing reference .","title":"Type hints"},{"location":"reference/plugins-development/","text":"Plugins provide hooks into Lightbus' inner workings. For example, the bundled StatePlugin hooks into the before_worker_start and after_worker_stopped hooks. The plugin uses these hooks to bus events indicating the state of the worker. A internal.state.worker_started event indicates a worker has started, and a internal.state.worker_stopped event indicates a worker has stopped. Consuming these events will provide a picture of the current state of workers on the bus. Example plugin \u00b6 Let's create a simple plugin which will print a configurable message to the console every time a message is sent. This requires three steps: Define the plugin Make Lightbus aware of the plugin Configure the plugin 1. Define the plugin \u00b6 Create the following in a python module (we'll assume it is at my_project.greeting_plugin ): from lightbus.plugins import LightbusPlugin class GreetingPlugin ( LightbusPlugin ): \"\"\"Print a greeting after every event it sent\"\"\" priority = 200 def __init__ ( self , greeting : str ): self . greeting = greeting @classmethod def from_config ( cls , config : \"Config\" , greeting : str = \"Hello world!\" ): # The arguments to this method define the configuration options which # can be set in the bus' yaml configuration (see below) return cls ( greeting = greeting ) async def after_event_sent ( self , * , event_message , client ): # Print a simple greeting after an event is sent print ( self . greeting ) 2. Make Lightbus aware of the plugin \u00b6 Lightbus is made aware of the plugin via an entrypoint in your project's pyproject.toml file: # Your pyproject.toml file ... [tool.poetry.plugins.lightbus_plugins] # `greeting_plugin` defines the plugin name in the config # `my_project.greeting_plugin` is your plugin's python module # `GreetingPlugin` is your plugins class name greeting_plugin = \"my_project.greeting_plugin:GreetingPlugin\" Once you've made this change (or for subsequent modifications) you will need to run: pip install . This will setup the entry_points you have specified 3. Configure the plugin \u00b6 You can configure your plugin in your bus' configuration YAML file (see the configuration reference ). For example: bus : ... apis : ... plugins : greeting_plugin : # The 'enabled' configuration is available for all plugins, # if set to 'false' the plugin will not be loaded. # Default is false enabled : true # Lightbus is aware of our `greeting` option as it reads it # from our `from_config()` method above. # If we omit this it will have the default value of \"Hello world!\" greeting : \"Hello world, I sent an event!\" Plugin hooks/methods \u00b6 The following hooks are available. Each of these should be implemented as an asynchronous method on your plugin class. For full reference see the LightbusPlugin class before_parse_args receive_args before_worker_start after_worker_stopped before_rpc_call after_rpc_call before_rpc_execution after_rpc_execution before_event_sent after_event_sent before_event_execution after_event_execution exception","title":"Plugin development"},{"location":"reference/plugins-development/#example-plugin","text":"Let's create a simple plugin which will print a configurable message to the console every time a message is sent. This requires three steps: Define the plugin Make Lightbus aware of the plugin Configure the plugin","title":"Example plugin"},{"location":"reference/plugins-development/#1-define-the-plugin","text":"Create the following in a python module (we'll assume it is at my_project.greeting_plugin ): from lightbus.plugins import LightbusPlugin class GreetingPlugin ( LightbusPlugin ): \"\"\"Print a greeting after every event it sent\"\"\" priority = 200 def __init__ ( self , greeting : str ): self . greeting = greeting @classmethod def from_config ( cls , config : \"Config\" , greeting : str = \"Hello world!\" ): # The arguments to this method define the configuration options which # can be set in the bus' yaml configuration (see below) return cls ( greeting = greeting ) async def after_event_sent ( self , * , event_message , client ): # Print a simple greeting after an event is sent print ( self . greeting )","title":"1. Define the plugin"},{"location":"reference/plugins-development/#2-make-lightbus-aware-of-the-plugin","text":"Lightbus is made aware of the plugin via an entrypoint in your project's pyproject.toml file: # Your pyproject.toml file ... [tool.poetry.plugins.lightbus_plugins] # `greeting_plugin` defines the plugin name in the config # `my_project.greeting_plugin` is your plugin's python module # `GreetingPlugin` is your plugins class name greeting_plugin = \"my_project.greeting_plugin:GreetingPlugin\" Once you've made this change (or for subsequent modifications) you will need to run: pip install . This will setup the entry_points you have specified","title":"2. Make Lightbus aware of the plugin"},{"location":"reference/plugins-development/#3-configure-the-plugin","text":"You can configure your plugin in your bus' configuration YAML file (see the configuration reference ). For example: bus : ... apis : ... plugins : greeting_plugin : # The 'enabled' configuration is available for all plugins, # if set to 'false' the plugin will not be loaded. # Default is false enabled : true # Lightbus is aware of our `greeting` option as it reads it # from our `from_config()` method above. # If we omit this it will have the default value of \"Hello world!\" greeting : \"Hello world, I sent an event!\"","title":"3. Configure the plugin"},{"location":"reference/plugins-development/#plugin-hooksmethods","text":"The following hooks are available. Each of these should be implemented as an asynchronous method on your plugin class. For full reference see the LightbusPlugin class before_parse_args receive_args before_worker_start after_worker_stopped before_rpc_call after_rpc_call before_rpc_execution after_rpc_execution before_event_sent after_event_sent before_event_execution after_event_execution exception","title":"Plugin hooks/methods"},{"location":"reference/plugins/","text":"Lightbus ships with two plugins, both of which are disabled by default. State Plugin \u00b6 Every Lightbus worker process which has the state plugin enabled will report it's state to the bus. This state information is available as the following events on the internal.state API. This plugin should add minimal load to the bus and may be useful in developing tooling around the bus. Events \u00b6 worker_started \u00b6 Parameters: process_name , metrics_enabled , api_names , listening_for , timestamp , ping_interval Fired when the worker starts up. worker_ping \u00b6 Parameters: process_name , metrics_enabled , api_names , listening_for , timestamp , ping_interval Fires every 60 seconds after worker startup. This indicates that the worker is still alive and has not died unexpectedly. This interval is configurable (see below). worker_stopped \u00b6 Parameters: process_name , timestamp Fires when a worker shuts down cleanly. Configuration \u00b6 The following configuration options are available: enabled (bool) \u00b6 Default: False Should the plugin be enabled? ping_enabled (bool) \u00b6 Default: True Should ping messages be sent? ping_enabled (int, seconds) \u00b6 Default: 60 How often (in seconds) should a ping event be sent. A lower interval means more frequent messages, but reduces the time it takes any listeners to discover dead workers. Example configuration \u00b6 bus : ... apis : ... plugins : internal_state : enabled : true ping_enabled : true ping_interval : 60 Metrics Plugin \u00b6 The metrics plugin sends metric events for every event & RPC processes. It therefore has a much bigger impact on performance than the state plugin, but also provides much more detailed information. Events \u00b6 The following events will be fired on the internal.metrics API: Event Parameters rpc_call_sent process_name , id , api_name , procedure_name , kwargs , timestamp rpc_call_received process_name , id , api_name , procedure_name , timestamp rpc_response_sent process_name , id , api_name , procedure_name , result , timestamp rpc_response_received process_name , id , api_name , procedure_name , timestamp event_fired process_name , event_id , api_name , event_name , kwargs , timestamp event_received process_name , event_id , api_name , event_name , kwargs , timestamp event_processed process_name , event_id , api_name , event_name , kwargs , timestamp Configuration \u00b6 The metrics plugin only includes the enabled configuration option. Configuration should therefore be: bus : ... apis : ... plugins : internal_metrics : enabled : true","title":"Plugins"},{"location":"reference/plugins/#state-plugin","text":"Every Lightbus worker process which has the state plugin enabled will report it's state to the bus. This state information is available as the following events on the internal.state API. This plugin should add minimal load to the bus and may be useful in developing tooling around the bus.","title":"State Plugin"},{"location":"reference/plugins/#events","text":"","title":"Events"},{"location":"reference/plugins/#worker_started","text":"Parameters: process_name , metrics_enabled , api_names , listening_for , timestamp , ping_interval Fired when the worker starts up.","title":"worker_started"},{"location":"reference/plugins/#worker_ping","text":"Parameters: process_name , metrics_enabled , api_names , listening_for , timestamp , ping_interval Fires every 60 seconds after worker startup. This indicates that the worker is still alive and has not died unexpectedly. This interval is configurable (see below).","title":"worker_ping"},{"location":"reference/plugins/#worker_stopped","text":"Parameters: process_name , timestamp Fires when a worker shuts down cleanly.","title":"worker_stopped"},{"location":"reference/plugins/#configuration","text":"The following configuration options are available:","title":"Configuration"},{"location":"reference/plugins/#enabled-bool","text":"Default: False Should the plugin be enabled?","title":"enabled (bool)"},{"location":"reference/plugins/#ping_enabled-bool","text":"Default: True Should ping messages be sent?","title":"ping_enabled (bool)"},{"location":"reference/plugins/#ping_enabled-int-seconds","text":"Default: 60 How often (in seconds) should a ping event be sent. A lower interval means more frequent messages, but reduces the time it takes any listeners to discover dead workers.","title":"ping_enabled (int, seconds)"},{"location":"reference/plugins/#example-configuration","text":"bus : ... apis : ... plugins : internal_state : enabled : true ping_enabled : true ping_interval : 60","title":"Example configuration"},{"location":"reference/plugins/#metrics-plugin","text":"The metrics plugin sends metric events for every event & RPC processes. It therefore has a much bigger impact on performance than the state plugin, but also provides much more detailed information.","title":"Metrics Plugin"},{"location":"reference/plugins/#events_1","text":"The following events will be fired on the internal.metrics API: Event Parameters rpc_call_sent process_name , id , api_name , procedure_name , kwargs , timestamp rpc_call_received process_name , id , api_name , procedure_name , timestamp rpc_response_sent process_name , id , api_name , procedure_name , result , timestamp rpc_response_received process_name , id , api_name , procedure_name , timestamp event_fired process_name , event_id , api_name , event_name , kwargs , timestamp event_received process_name , event_id , api_name , event_name , kwargs , timestamp event_processed process_name , event_id , api_name , event_name , kwargs , timestamp","title":"Events"},{"location":"reference/plugins/#configuration_1","text":"The metrics plugin only includes the enabled configuration option. Configuration should therefore be: bus : ... apis : ... plugins : internal_metrics : enabled : true","title":"Configuration"},{"location":"reference/release-process/","text":"Lightbus release process \u00b6 Lightbus releases are performed as follows: # Ensure poetry.lock is up to date poetry lock # Version bump poetry version { patch,minor,major,prepatch,preminor,premajor,prerelease } export VERSION = $( lightbus version --pyproject ) # v1.2.3 export VERSION_DOCS = $( lightbus version --pyproject --docs ) # v1.2 # Commit git add . git commit -m \"Releasing version $VERSION \" # Make docs git checkout gh-pages git pull origin gh-pages git checkout master mike deploy v $VERSION_DOCS --message = \"Build docs for release of $VERSION [ci skip]\" mike delete latest mike alias v $VERSION_DOCS latest # Tagging and branching git tag \"v $VERSION \" git branch \"v $VERSION \" git push origin \\ refs/tags/ \"v $VERSION \" \\ refs/heads/ \"v $VERSION \" \\ master \\ gh-pages # Wait for CI to pass: https://circleci.com/gh/adamcharnock/lightbus # Build and publish poetry publish --build","title":"Release process"},{"location":"reference/release-process/#lightbus-release-process","text":"Lightbus releases are performed as follows: # Ensure poetry.lock is up to date poetry lock # Version bump poetry version { patch,minor,major,prepatch,preminor,premajor,prerelease } export VERSION = $( lightbus version --pyproject ) # v1.2.3 export VERSION_DOCS = $( lightbus version --pyproject --docs ) # v1.2 # Commit git add . git commit -m \"Releasing version $VERSION \" # Make docs git checkout gh-pages git pull origin gh-pages git checkout master mike deploy v $VERSION_DOCS --message = \"Build docs for release of $VERSION [ci skip]\" mike delete latest mike alias v $VERSION_DOCS latest # Tagging and branching git tag \"v $VERSION \" git branch \"v $VERSION \" git push origin \\ refs/tags/ \"v $VERSION \" \\ refs/heads/ \"v $VERSION \" \\ master \\ gh-pages # Wait for CI to pass: https://circleci.com/gh/adamcharnock/lightbus # Build and publish poetry publish --build","title":"Lightbus release process"},{"location":"reference/rpcs/","text":"Remote procedures calls (RPCs) are defined as methods on your API classes. They are useful when either: You require information from a service You wish to wait until a remote procedure has completed an action RPCs provide at-most-once delivery semantics. If you need at-least-once semantics you should consder using events instead. Definition \u00b6 As covered in previous sections, you define RPCs as follows: # bus.py from lightbus import Api class AuthApi ( Api ): class Meta : name = 'auth' def check_password ( self , username , password ): return username == 'admin' and password == 'secret' def reset_password ( self , username ): reset_users_password_somehow ( username ) def get_user ( self , username ): return get_user ( username ) def promote_to_admin ( username ): user = get_user ( username ) user . admin = True user . save () Calling \u00b6 RPCs are called simply as follows: # Anywhere in your code # Import your project's bus instance from bus import bus # Call the RPC is_valid = bus . auth . check_password ( username = \"adam\" , password = \"secr3t\" ) Calling (asynchronously) \u00b6 You can also perform the call asynchronously using asyncio: # Anywhere in your code # Import your project's bus instance from bus import bus is_valid = await bus . auth . check_password . call_async ( username = \"adam\" , password = \"secr3t\" ) Type hints \u00b6 See the typing reference . Best practices \u00b6 RPC implementation \u00b6 It is best to keep your RPCs simple and easy to understand. In anything but the simplest service it will probably be best to use the API definition as a presentational layer which wraps up the business logic located elsewhere. If you find business logic creeping into your RPC definitions, consider factoring it out and invoking it from the RPC definition. For smaller services this will be less important, but as functionality is shared and used elsewhere within your service you may find it keeps your code more managable. This also leaves your RPCs definitions free do any API-specific legwork such as data marshalling. For example, converting incoming natural keys (e.g. usernames) into the primary keys (i.e. user IDs) which your service's may use internally. Architecture & coupling \u00b6 RPCs often represent a tight coupling between your code and the service you are calling. This may be acceptable to you, but it is worth being aware of potential pitfalls: Failures & timeouts may occurr, which should ideally be handled gracefully Modifications to the remote RPC may require updates to the code which calls the RPC RPCs incur much greater overhead than regular function calls. Utility functions that use RPCs should therefore make it clear that they will be incurring this overhead (either through naming convention or documentation) The more services call to for a given action, the less reliable the action will typically be. (i.e. if any service is unavailable the action will potentially fail) Using events will provide a different set of trade-offs, and ultimately you will need to decide on what is right for your particular scenario. Parameter values \u00b6 When deciding the values your RPC should receive, consider: Can this value become out of date? For example, an entire user object could become out of date, whereas a username or user ID would not. Packing and unpacking large data structures is computationally expensive. Will the meaning of the value change over time? For example, the meaning of 'today' or 'now' will change, but the meaning of a specific date & time will remain the same. Limitations \u00b6 RPCs can only be called with keyword arguments. For example: # Raises an InvalidParameters exception result = bus . auth . check_password ( 'admin' , 'secret' ) # ERROR # Success result = bus . auth . check_password ( username = 'admin' , password = 'secret' )","title":"Remote procedure calls"},{"location":"reference/rpcs/#definition","text":"As covered in previous sections, you define RPCs as follows: # bus.py from lightbus import Api class AuthApi ( Api ): class Meta : name = 'auth' def check_password ( self , username , password ): return username == 'admin' and password == 'secret' def reset_password ( self , username ): reset_users_password_somehow ( username ) def get_user ( self , username ): return get_user ( username ) def promote_to_admin ( username ): user = get_user ( username ) user . admin = True user . save ()","title":"Definition"},{"location":"reference/rpcs/#calling","text":"RPCs are called simply as follows: # Anywhere in your code # Import your project's bus instance from bus import bus # Call the RPC is_valid = bus . auth . check_password ( username = \"adam\" , password = \"secr3t\" )","title":"Calling"},{"location":"reference/rpcs/#calling-asynchronously","text":"You can also perform the call asynchronously using asyncio: # Anywhere in your code # Import your project's bus instance from bus import bus is_valid = await bus . auth . check_password . call_async ( username = \"adam\" , password = \"secr3t\" )","title":"Calling (asynchronously)"},{"location":"reference/rpcs/#type-hints","text":"See the typing reference .","title":"Type hints"},{"location":"reference/rpcs/#best-practices","text":"","title":"Best practices"},{"location":"reference/rpcs/#rpc-implementation","text":"It is best to keep your RPCs simple and easy to understand. In anything but the simplest service it will probably be best to use the API definition as a presentational layer which wraps up the business logic located elsewhere. If you find business logic creeping into your RPC definitions, consider factoring it out and invoking it from the RPC definition. For smaller services this will be less important, but as functionality is shared and used elsewhere within your service you may find it keeps your code more managable. This also leaves your RPCs definitions free do any API-specific legwork such as data marshalling. For example, converting incoming natural keys (e.g. usernames) into the primary keys (i.e. user IDs) which your service's may use internally.","title":"RPC implementation"},{"location":"reference/rpcs/#architecture-coupling","text":"RPCs often represent a tight coupling between your code and the service you are calling. This may be acceptable to you, but it is worth being aware of potential pitfalls: Failures & timeouts may occurr, which should ideally be handled gracefully Modifications to the remote RPC may require updates to the code which calls the RPC RPCs incur much greater overhead than regular function calls. Utility functions that use RPCs should therefore make it clear that they will be incurring this overhead (either through naming convention or documentation) The more services call to for a given action, the less reliable the action will typically be. (i.e. if any service is unavailable the action will potentially fail) Using events will provide a different set of trade-offs, and ultimately you will need to decide on what is right for your particular scenario.","title":"Architecture & coupling"},{"location":"reference/rpcs/#parameter-values","text":"When deciding the values your RPC should receive, consider: Can this value become out of date? For example, an entire user object could become out of date, whereas a username or user ID would not. Packing and unpacking large data structures is computationally expensive. Will the meaning of the value change over time? For example, the meaning of 'today' or 'now' will change, but the meaning of a specific date & time will remain the same.","title":"Parameter values"},{"location":"reference/rpcs/#limitations","text":"RPCs can only be called with keyword arguments. For example: # Raises an InvalidParameters exception result = bus . auth . check_password ( 'admin' , 'secret' ) # ERROR # Success result = bus . auth . check_password ( username = 'admin' , password = 'secret' )","title":"Limitations"},{"location":"reference/schema/","text":"Lightbus processes automatically generate and share schemas for their available APIs. These schemes can be used to validate the following: Remote procedure call parameters Remote procedure call return values Event parameters These schemas are shared using the configured SchemaTransprt (Redis, by default). Each Lightbus process will monitor for any schema changes. Specifying types \u00b6 Lightbus will create a schema by inspecting the parameters and type hints of your APIs' events and procedures. You can use the schema functionality without type hints, but the level of validation provided will be limited to ensuring parameter names match what is expected. The schema protocol reference covers the specifics of the schema data format. Supported data types \u00b6 Lightbus maps Python types to JSON types. While Python-specific values can be sent using Lightbus, these values will arrive in their JSON form. For example, if you send a string then a string will arrive. However, if you send the Decimal value 3.124 , then you will receive the string value 3.124 instead. The following types are reasonably interoperable: Python type sent JSON schema interpretation Type received str string str int , float number int , float bool boolean bool list , tuple array list None null None dict , Mapping , etc object dict Mapping[str, ...] object , with pattern properties set dict Tuple[A, B, C] array with maxItems/minItems and items set. list The following types will be successfully encoded and sent, but will arrive as their encoded equivalent: Python type JSON Schema type Value arrives as bytes , Decimal , complex string str datetime , date str str (ISO 8601) NamedTuple with annotations object with specific typed properties dict object with annotations object with specific typed properties dict Lightbus can also handle the following: Python type JSON Schema type Any {} (any value) Union[...] oneOf{...} (see oneOf ) Enum Sets enum property Automatic validation \u00b6 By default this validation will be validated in both the incoming and outgoing directions. Outgoing refers to the dispatching of events or procedure calls to the bus. Incoming refers to the processing of procedure calls or handling of received events. You can configuring this using the validate configuration option. Validation configuration \u00b6 You can configure the validation behaviour in your bus' config.yaml . validate (bool) = true \u00b6 You can enable/disable validation using a boolean true/false flag: # In config.yaml apis : default : validate : false For finer grained control you can specify individual flags for incoming/outgoing validation: # In config.yaml apis : default : validate : outgoing : true incoming : false strict_validation (bool) = false \u00b6 If strict_validation is true then calling a procedure for which no schema exists will result in an error: # In config.yaml apis : default : strict_validation : true Manual validation \u00b6 Lightbus supports manually loading your bus' schema into your bus client. For example, you may take a dump of your production bus schema and use it to test/develop against either in your local development environment or in any automated testing system. You can load a local schema as follows: from bus import bus # Either load form a single file bus . schema . load_local ( \"/path/to/local/bus-schema.json\" ) # ... or load all schemas in a directory bus . schema . load_local ( \"/path/to/local/schema/\" ) Your bus client will now use the specified schema to validate RPCs and events. Parameters and responses may also be directly validated so you need: Event (parameters): bus.my_api.my_event.validate_parameters(parameters) RPC (parameters): bus.my_api.my_rpc.validate_parameters(parameters) RPC (response): bus.my_api.my_rpc.validate_response(response) You can use these methods to manually validate parameters or response values against the locally loaded schema.","title":"Schema"},{"location":"reference/schema/#specifying-types","text":"Lightbus will create a schema by inspecting the parameters and type hints of your APIs' events and procedures. You can use the schema functionality without type hints, but the level of validation provided will be limited to ensuring parameter names match what is expected. The schema protocol reference covers the specifics of the schema data format.","title":"Specifying types"},{"location":"reference/schema/#supported-data-types","text":"Lightbus maps Python types to JSON types. While Python-specific values can be sent using Lightbus, these values will arrive in their JSON form. For example, if you send a string then a string will arrive. However, if you send the Decimal value 3.124 , then you will receive the string value 3.124 instead. The following types are reasonably interoperable: Python type sent JSON schema interpretation Type received str string str int , float number int , float bool boolean bool list , tuple array list None null None dict , Mapping , etc object dict Mapping[str, ...] object , with pattern properties set dict Tuple[A, B, C] array with maxItems/minItems and items set. list The following types will be successfully encoded and sent, but will arrive as their encoded equivalent: Python type JSON Schema type Value arrives as bytes , Decimal , complex string str datetime , date str str (ISO 8601) NamedTuple with annotations object with specific typed properties dict object with annotations object with specific typed properties dict Lightbus can also handle the following: Python type JSON Schema type Any {} (any value) Union[...] oneOf{...} (see oneOf ) Enum Sets enum property","title":"Supported data types"},{"location":"reference/schema/#automatic-validation","text":"By default this validation will be validated in both the incoming and outgoing directions. Outgoing refers to the dispatching of events or procedure calls to the bus. Incoming refers to the processing of procedure calls or handling of received events. You can configuring this using the validate configuration option.","title":"Automatic validation"},{"location":"reference/schema/#validation-configuration","text":"You can configure the validation behaviour in your bus' config.yaml .","title":"Validation configuration"},{"location":"reference/schema/#validate-bool-true","text":"You can enable/disable validation using a boolean true/false flag: # In config.yaml apis : default : validate : false For finer grained control you can specify individual flags for incoming/outgoing validation: # In config.yaml apis : default : validate : outgoing : true incoming : false","title":"validate (bool) = true"},{"location":"reference/schema/#strict_validation-bool-false","text":"If strict_validation is true then calling a procedure for which no schema exists will result in an error: # In config.yaml apis : default : strict_validation : true","title":"strict_validation (bool) = false"},{"location":"reference/schema/#manual-validation","text":"Lightbus supports manually loading your bus' schema into your bus client. For example, you may take a dump of your production bus schema and use it to test/develop against either in your local development environment or in any automated testing system. You can load a local schema as follows: from bus import bus # Either load form a single file bus . schema . load_local ( \"/path/to/local/bus-schema.json\" ) # ... or load all schemas in a directory bus . schema . load_local ( \"/path/to/local/schema/\" ) Your bus client will now use the specified schema to validate RPCs and events. Parameters and responses may also be directly validated so you need: Event (parameters): bus.my_api.my_event.validate_parameters(parameters) RPC (parameters): bus.my_api.my_rpc.validate_parameters(parameters) RPC (response): bus.my_api.my_rpc.validate_response(response) You can use these methods to manually validate parameters or response values against the locally loaded schema.","title":"Manual validation"},{"location":"reference/testing-and-mocking/","text":"Testing & mocking \u00b6 Lightbus provides utilities to make testing easier. These utilities allow you to: Ensure only specific events & RPCs were fired Access the sent messages Mock RPC responses Mocking with events \u00b6 from lightbus.utilities.testing import BusMocker from bus import bus def test_firing_event (): with BusMocker ( bus ) as bus_mock : # Setup the mocker to expect the auth.user_created was fired. # An error will be raised if the tested code fires any other events bus_mock . mock_event_firing ( \"auth.user_created\" ) # Run the code to be tested bus . auth . user_created . fire ( field = \"x\" ) # Check the event was fired once bus_mock . assert_events_fired ( \"auth.user_created\" , times = 1 ) # Get the fired event message message = bus_mock . get_event_messages ( \"auth.user_created\" )[ 0 ] assert message . kwargs == { \"username\" : \"sarahjane\" } Mocking with RPCs \u00b6 from lightbus.utilities.testing import BusMocker from bus import bus def test_calling_rpc (): with BusMocker ( bus ) as bus_mock : # Setup the mocker to expect the auth.user_created was fired. # An error will be raised if the tested code calls any other RPCs bus_mock . mock_rpc_call ( \"auth.check_password\" , result = True ) # Run the code to be tested bus . auth . check_password ( username = \"sarahjane\" , password = \"secret\" ) # Check the event was fired once bus_mock . assert_rpc_called ( \"auth.check_password\" , times = 1 ) # Get the fired RPC message = bus_mock . get_rpc_messages ( \"auth.check_password\" )[ 0 ] assert message . kwargs == { \"username\" : \"sarahjane\" , \"password\" : \"secret\" } Allowing arbitrary events and RPCs \u00b6 By default the mocker will raise an error if any event or RPC is fired or called which has not be setup using the mock_event_firing / mock_rpc_call methods. You can disable this and therefore allow any events or RPCs to the fired/called by using BusMocker(bus, require_mocking=False) . For example, the following will result in no errors even though we have not setup any mocks: from lightbus.utilities.testing import BusMocker from bus import bus def test_permissive_mocking (): with BusMocker ( bus , require_mocking = False ) as bus_mock : # Neither will cause an error despite the lack of mocking setup. # This is because we have set require_mocking=False bus . auth . user_created . fire ( field = \"x\" ) bus . auth . check_password ( username = \"sarahjane\" , password = \"secret\" ) The @bus_mocker decorator \u00b6 You can also access the bus mocker using the @bus_mocker decorator. We can rewrite our earlier event example (above) as follows: from lightbus.utilities.testing import bus_mocker from bus import bus @bus_mocker ( bus ) def test_firing_event ( bus_mock ): # Setup the mocker to expect the auth.user_created was fired. # An error will be raised if the tested code fires any other events bus_mock . mock_event_firing ( \"auth.user_created\" ) # Run the code to be tested bus . auth . user_created . fire ( field = \"x\" ) # Check the event was fired once bus_mock . assert_events_fired ( \"auth.user_created\" , times = 1 ) # Get the fired event message message = bus_mock . get_event_messages ( \"auth.user_created\" )[ 0 ] assert message . kwargs == { \"username\" : \"sarahjane\" } Testing in Django \u00b6 You can use the mocker in your Django tests just as shown above. For example, we can access the mocker using the context manager: from django.test import TestCase from lightbus.utilities.testing import BusMocker from bus import bus class ExampleTestCase ( TestCase ): def test_something ( self ): with BusMocker ( bus , require_mocking = False ) as bus_mock : ... Or we can use the @bus_mocker decorator: from django.test import TestCase from lightbus.utilities.testing import bus_mocker from bus import bus class ExampleTestCase ( TestCase ): @bus_mocker ( bus ) def test_something ( self , bus_mock ): ...","title":"Testing & Mocking"},{"location":"reference/testing-and-mocking/#testing-mocking","text":"Lightbus provides utilities to make testing easier. These utilities allow you to: Ensure only specific events & RPCs were fired Access the sent messages Mock RPC responses","title":"Testing & mocking"},{"location":"reference/testing-and-mocking/#mocking-with-events","text":"from lightbus.utilities.testing import BusMocker from bus import bus def test_firing_event (): with BusMocker ( bus ) as bus_mock : # Setup the mocker to expect the auth.user_created was fired. # An error will be raised if the tested code fires any other events bus_mock . mock_event_firing ( \"auth.user_created\" ) # Run the code to be tested bus . auth . user_created . fire ( field = \"x\" ) # Check the event was fired once bus_mock . assert_events_fired ( \"auth.user_created\" , times = 1 ) # Get the fired event message message = bus_mock . get_event_messages ( \"auth.user_created\" )[ 0 ] assert message . kwargs == { \"username\" : \"sarahjane\" }","title":"Mocking with events"},{"location":"reference/testing-and-mocking/#mocking-with-rpcs","text":"from lightbus.utilities.testing import BusMocker from bus import bus def test_calling_rpc (): with BusMocker ( bus ) as bus_mock : # Setup the mocker to expect the auth.user_created was fired. # An error will be raised if the tested code calls any other RPCs bus_mock . mock_rpc_call ( \"auth.check_password\" , result = True ) # Run the code to be tested bus . auth . check_password ( username = \"sarahjane\" , password = \"secret\" ) # Check the event was fired once bus_mock . assert_rpc_called ( \"auth.check_password\" , times = 1 ) # Get the fired RPC message = bus_mock . get_rpc_messages ( \"auth.check_password\" )[ 0 ] assert message . kwargs == { \"username\" : \"sarahjane\" , \"password\" : \"secret\" }","title":"Mocking with RPCs"},{"location":"reference/testing-and-mocking/#allowing-arbitrary-events-and-rpcs","text":"By default the mocker will raise an error if any event or RPC is fired or called which has not be setup using the mock_event_firing / mock_rpc_call methods. You can disable this and therefore allow any events or RPCs to the fired/called by using BusMocker(bus, require_mocking=False) . For example, the following will result in no errors even though we have not setup any mocks: from lightbus.utilities.testing import BusMocker from bus import bus def test_permissive_mocking (): with BusMocker ( bus , require_mocking = False ) as bus_mock : # Neither will cause an error despite the lack of mocking setup. # This is because we have set require_mocking=False bus . auth . user_created . fire ( field = \"x\" ) bus . auth . check_password ( username = \"sarahjane\" , password = \"secret\" )","title":"Allowing arbitrary events and RPCs"},{"location":"reference/testing-and-mocking/#the-bus_mocker-decorator","text":"You can also access the bus mocker using the @bus_mocker decorator. We can rewrite our earlier event example (above) as follows: from lightbus.utilities.testing import bus_mocker from bus import bus @bus_mocker ( bus ) def test_firing_event ( bus_mock ): # Setup the mocker to expect the auth.user_created was fired. # An error will be raised if the tested code fires any other events bus_mock . mock_event_firing ( \"auth.user_created\" ) # Run the code to be tested bus . auth . user_created . fire ( field = \"x\" ) # Check the event was fired once bus_mock . assert_events_fired ( \"auth.user_created\" , times = 1 ) # Get the fired event message message = bus_mock . get_event_messages ( \"auth.user_created\" )[ 0 ] assert message . kwargs == { \"username\" : \"sarahjane\" }","title":"The @bus_mocker decorator"},{"location":"reference/testing-and-mocking/#testing-in-django","text":"You can use the mocker in your Django tests just as shown above. For example, we can access the mocker using the context manager: from django.test import TestCase from lightbus.utilities.testing import BusMocker from bus import bus class ExampleTestCase ( TestCase ): def test_something ( self ): with BusMocker ( bus , require_mocking = False ) as bus_mock : ... Or we can use the @bus_mocker decorator: from django.test import TestCase from lightbus.utilities.testing import bus_mocker from bus import bus class ExampleTestCase ( TestCase ): @bus_mocker ( bus ) def test_something ( self , bus_mock ): ...","title":"Testing in Django"},{"location":"reference/transport-configuration/","text":"Transport configuration \u00b6 Lightbus ships with built-in support for Redis. This is provided by the following transports: An event transport \u2013 sends and consumes RPC calls An RPC transport \u2013 sends and receives RPC results A result transport \u2013 sends and consumes events A schema transport \u2013 stores and retrieves the bus schema Configuration auto-complete using JSON Schema Many code editors support using a JSON schema to provide auto-complete and validation when editing a JSON file. If you wish, you can write your configuration in JSON (rather than YAML), and load the following JSON schema into your editor: https://lightbus.org/static/default-config-schema.json This will provide you with autocomplete and validation for Lightbus' various configuration options. If you are using custom transports or plugins you should generate your own config schema . Complete configuration example \u00b6 ... apis : # Catch-all api config default : # ...API Config options here (see configuration reference)... # Per-transport configuration event_transport : redis : url : \"redis://redis_host:6379/0\" batch_size : 10 reclaim_batch_size : 100 serializer : \"lightbus.serializers.ByFieldMessageSerializer\" deserializer : \"lightbus.serializers.ByFieldMessageDeserializer\" acknowledgement_timeout : 60 max_stream_length : 100000 stream_use : \"per_api\" consumption_restart_delay : 5 consumer_ttl : 2592000 rpc_transport : redis : url : \"redis://redis_host:6379/0\" batch_size : 10 serializer : \"lightbus.serializers.BlobMessageSerializer\" deserializer : \"lightbus.serializers.BlobMessageDeserializer\" rpc_timeout : 5 rpc_retry_delay : 1 consumption_restart_delay : 5 result_transport : redis : url : \"redis://redis_host:6379/0\" serializer : \"lightbus.serializers.BlobMessageSerializer\" deserializer : \"lightbus.serializers.BlobMessageDeserializer\" rpc_timeout : 5 rpc_retry_delay : 1 consumption_restart_delay : 5 # Can respecify the above config for specific APIs # if customisation is needed. marketing.analytics : ... # Schema transport configuration is at the root level schema : transport : redis : url : \"redis://redis.svc.cluster.local:6379/0\" Redis Event Transport configuration \u00b6 url \u00b6 Type: str , default: redis://127.0.0.1:6379/0 The connection string for the redis server. Format is: `redis://host:port/db_number` batch_size \u00b6 Type: int , default: 10 The maximum number of messages to be fetched at one time. A higher value will reduce overhead for large volumes of messages. However, should a worker die then the processing of the fetched messages will be delayed by acknowledgement_timeout . In this case those messages will be processed out-of-order. reclaim_batch_size \u00b6 Type: int , default: reclaim_batch_size * 10 The maximum number of messages to be fetched at one time when reclaiming timed out messages . serializer \u00b6 Type: str , default: lightbus.serializers.ByFieldMessageSerializer The serializer to be used in converting the lightbus message into a bus-appropriate format. deserializer \u00b6 Type: str , default: lightbus.serializers.ByFieldMessageDeserializer The deserializer to be used in converting the lightbus message into a bus-appropriate format. acknowledgement_timeout \u00b6 Type: float , default: 60.0 , seconds Any message not processed acknowledgement_timeout seconds will assume to have failed and will therefore be reclaimed up by another Lightbus worker. This is typically caused by a Lightbus worker exiting ungracefully. Long running events You will need to modify this if you have event handlers which take a long time to execute. This value must exceed the length of time it takes any event to be processed reclaim_interval \u00b6 Type: float , default is the value of acknowledgement_timeout , in seconds How often should each Lightbus client attempt to reclaim events? When an error occurs in a Lightbus worker then it may die while still holding a claim over events. Other Lightbus works therefore periodically check for any events which have timed out (see acknowledgement_timeout ) and attempt to reclaim any events found. Reclaimed events are then processed and acknowledged as normal. max_stream_length \u00b6 Type: int , default: 100_000 Streams will be trimmed so they never exceed the given length. Set to null for no limit. stream_use \u00b6 Type: str , default: per_api How should Redis streams be created? There are two options: per_api \u2013 One stream for an entire API. per_event \u2013 One stream for each event on an API Setting this to per_api will ensure event listeners receive all events on an api in order. However, all events for the API will be received even if not needed (Lightbus will discard these unwanted events before passing them to your event handler). This will consume unnecessary resources if an API contains high-volume events which your listener does not care for. Setting this to per_event will ensure that Lightbus only receives the needed events, but messages will only be ordered for an individual event. consumption_restart_delay \u00b6 Type: int , default: 5 , seconds How long to wait before attempting to reconnect after loosing the connection to Redis. consumer_ttl \u00b6 Type: int , default: 2_592_000 , seconds How long to wait before cleaning up inactive consumers. Default is 30 days. Redis RPC Transport configuration \u00b6 url \u00b6 Type: str , default: redis://127.0.0.1:6379/0 The connection string for the redis server. Format is: `redis://host:port/db_number` batch_size \u00b6 Type: int , default: 10 The maximum number of messages to be fetched at one time. A higher value will reduce overhead for large volumes of messages. serializer \u00b6 Type: str , default: lightbus.serializers.BlobMessageSerializer The serializer to be used in converting the lightbus message into a bus-appropriate format. deserializer \u00b6 Type: str , default: lightbus.serializers.BlobMessageDeserializer The deserializer to be used in converting the lightbus message into a bus-appropriate format. rpc_timeout \u00b6 Type: int , default: 5 , seconds How long to wait before we give up waiting for an RPC to be processed. Note on rpc_timeout This configuration option is also repeated in the API config . For now this value needs to be specified in three places: The API config RPC transport config Result transport config (see below) rpc_retry_delay \u00b6 Type: int , default: 1 , seconds How long to wait before reattempting to call the remote RPC. For example, in cases where Redis errors (e.g. connection issues). Execution will be retried once only. consumption_restart_delay \u00b6 Type: int , default: 5 , seconds How long to wait before attempting to reconnect after loosing the connection to Redis. Redis Result Transport configuration \u00b6 url \u00b6 Type: str , default: redis://127.0.0.1:6379/0 The connection string for the redis server. Format is: `redis://host:port/db_number` serializer \u00b6 Type: str , default: lightbus.serializers.BlobMessageSerializer The serializer to be used in converting the lightbus message into a bus-appropriate format. deserializer \u00b6 Type: str , default: lightbus.serializers.BlobMessageDeserializer The deserializer to be used in converting the lightbus message into a bus-appropriate format. rpc_timeout \u00b6 Type: int , default: 5 , seconds How long to wait before we give up waiting for an RPC result to be received. Note on rpc_timeout This configuration option is also repeated in the API config . For now this value needs to be specified in three places: # The API config * RPC transport config (see above) * Result transport config rpc_retry_delay \u00b6 Type: int , default: 1 , seconds How long to wait before reattempting to call the remote RPC. For example, in cases where Redis errors (e.g. connection issues). Execution will be retried once only. consumption_restart_delay \u00b6 Type: int , default: 5 , seconds How long to wait before attempting to reconnect after loosing the connection to Redis. Redis Schema Transport configuration \u00b6 url \u00b6 Type: str , default: redis://127.0.0.1:6379/0 The connection string for the redis server. Format is: `redis://host:port/db_number`","title":"Transport configuration"},{"location":"reference/transport-configuration/#transport-configuration","text":"Lightbus ships with built-in support for Redis. This is provided by the following transports: An event transport \u2013 sends and consumes RPC calls An RPC transport \u2013 sends and receives RPC results A result transport \u2013 sends and consumes events A schema transport \u2013 stores and retrieves the bus schema Configuration auto-complete using JSON Schema Many code editors support using a JSON schema to provide auto-complete and validation when editing a JSON file. If you wish, you can write your configuration in JSON (rather than YAML), and load the following JSON schema into your editor: https://lightbus.org/static/default-config-schema.json This will provide you with autocomplete and validation for Lightbus' various configuration options. If you are using custom transports or plugins you should generate your own config schema .","title":"Transport configuration"},{"location":"reference/transport-configuration/#complete-configuration-example","text":"... apis : # Catch-all api config default : # ...API Config options here (see configuration reference)... # Per-transport configuration event_transport : redis : url : \"redis://redis_host:6379/0\" batch_size : 10 reclaim_batch_size : 100 serializer : \"lightbus.serializers.ByFieldMessageSerializer\" deserializer : \"lightbus.serializers.ByFieldMessageDeserializer\" acknowledgement_timeout : 60 max_stream_length : 100000 stream_use : \"per_api\" consumption_restart_delay : 5 consumer_ttl : 2592000 rpc_transport : redis : url : \"redis://redis_host:6379/0\" batch_size : 10 serializer : \"lightbus.serializers.BlobMessageSerializer\" deserializer : \"lightbus.serializers.BlobMessageDeserializer\" rpc_timeout : 5 rpc_retry_delay : 1 consumption_restart_delay : 5 result_transport : redis : url : \"redis://redis_host:6379/0\" serializer : \"lightbus.serializers.BlobMessageSerializer\" deserializer : \"lightbus.serializers.BlobMessageDeserializer\" rpc_timeout : 5 rpc_retry_delay : 1 consumption_restart_delay : 5 # Can respecify the above config for specific APIs # if customisation is needed. marketing.analytics : ... # Schema transport configuration is at the root level schema : transport : redis : url : \"redis://redis.svc.cluster.local:6379/0\"","title":"Complete configuration example"},{"location":"reference/transport-configuration/#redis-event-transport-configuration","text":"","title":"Redis Event Transport configuration"},{"location":"reference/transport-configuration/#url","text":"Type: str , default: redis://127.0.0.1:6379/0 The connection string for the redis server. Format is: `redis://host:port/db_number`","title":"url"},{"location":"reference/transport-configuration/#batch_size","text":"Type: int , default: 10 The maximum number of messages to be fetched at one time. A higher value will reduce overhead for large volumes of messages. However, should a worker die then the processing of the fetched messages will be delayed by acknowledgement_timeout . In this case those messages will be processed out-of-order.","title":"batch_size"},{"location":"reference/transport-configuration/#reclaim_batch_size","text":"Type: int , default: reclaim_batch_size * 10 The maximum number of messages to be fetched at one time when reclaiming timed out messages .","title":"reclaim_batch_size"},{"location":"reference/transport-configuration/#serializer","text":"Type: str , default: lightbus.serializers.ByFieldMessageSerializer The serializer to be used in converting the lightbus message into a bus-appropriate format.","title":"serializer"},{"location":"reference/transport-configuration/#deserializer","text":"Type: str , default: lightbus.serializers.ByFieldMessageDeserializer The deserializer to be used in converting the lightbus message into a bus-appropriate format.","title":"deserializer"},{"location":"reference/transport-configuration/#acknowledgement_timeout","text":"Type: float , default: 60.0 , seconds Any message not processed acknowledgement_timeout seconds will assume to have failed and will therefore be reclaimed up by another Lightbus worker. This is typically caused by a Lightbus worker exiting ungracefully. Long running events You will need to modify this if you have event handlers which take a long time to execute. This value must exceed the length of time it takes any event to be processed","title":"acknowledgement_timeout"},{"location":"reference/transport-configuration/#reclaim_interval","text":"Type: float , default is the value of acknowledgement_timeout , in seconds How often should each Lightbus client attempt to reclaim events? When an error occurs in a Lightbus worker then it may die while still holding a claim over events. Other Lightbus works therefore periodically check for any events which have timed out (see acknowledgement_timeout ) and attempt to reclaim any events found. Reclaimed events are then processed and acknowledged as normal.","title":"reclaim_interval"},{"location":"reference/transport-configuration/#max_stream_length","text":"Type: int , default: 100_000 Streams will be trimmed so they never exceed the given length. Set to null for no limit.","title":"max_stream_length"},{"location":"reference/transport-configuration/#stream_use","text":"Type: str , default: per_api How should Redis streams be created? There are two options: per_api \u2013 One stream for an entire API. per_event \u2013 One stream for each event on an API Setting this to per_api will ensure event listeners receive all events on an api in order. However, all events for the API will be received even if not needed (Lightbus will discard these unwanted events before passing them to your event handler). This will consume unnecessary resources if an API contains high-volume events which your listener does not care for. Setting this to per_event will ensure that Lightbus only receives the needed events, but messages will only be ordered for an individual event.","title":"stream_use"},{"location":"reference/transport-configuration/#consumption_restart_delay","text":"Type: int , default: 5 , seconds How long to wait before attempting to reconnect after loosing the connection to Redis.","title":"consumption_restart_delay"},{"location":"reference/transport-configuration/#consumer_ttl","text":"Type: int , default: 2_592_000 , seconds How long to wait before cleaning up inactive consumers. Default is 30 days.","title":"consumer_ttl"},{"location":"reference/transport-configuration/#redis-rpc-transport-configuration","text":"","title":"Redis RPC Transport configuration"},{"location":"reference/transport-configuration/#url_1","text":"Type: str , default: redis://127.0.0.1:6379/0 The connection string for the redis server. Format is: `redis://host:port/db_number`","title":"url"},{"location":"reference/transport-configuration/#batch_size_1","text":"Type: int , default: 10 The maximum number of messages to be fetched at one time. A higher value will reduce overhead for large volumes of messages.","title":"batch_size"},{"location":"reference/transport-configuration/#serializer_1","text":"Type: str , default: lightbus.serializers.BlobMessageSerializer The serializer to be used in converting the lightbus message into a bus-appropriate format.","title":"serializer"},{"location":"reference/transport-configuration/#deserializer_1","text":"Type: str , default: lightbus.serializers.BlobMessageDeserializer The deserializer to be used in converting the lightbus message into a bus-appropriate format.","title":"deserializer"},{"location":"reference/transport-configuration/#rpc_timeout","text":"Type: int , default: 5 , seconds How long to wait before we give up waiting for an RPC to be processed. Note on rpc_timeout This configuration option is also repeated in the API config . For now this value needs to be specified in three places: The API config RPC transport config Result transport config (see below)","title":"rpc_timeout"},{"location":"reference/transport-configuration/#rpc_retry_delay","text":"Type: int , default: 1 , seconds How long to wait before reattempting to call the remote RPC. For example, in cases where Redis errors (e.g. connection issues). Execution will be retried once only.","title":"rpc_retry_delay"},{"location":"reference/transport-configuration/#consumption_restart_delay_1","text":"Type: int , default: 5 , seconds How long to wait before attempting to reconnect after loosing the connection to Redis.","title":"consumption_restart_delay"},{"location":"reference/transport-configuration/#redis-result-transport-configuration","text":"","title":"Redis Result Transport configuration"},{"location":"reference/transport-configuration/#url_2","text":"Type: str , default: redis://127.0.0.1:6379/0 The connection string for the redis server. Format is: `redis://host:port/db_number`","title":"url"},{"location":"reference/transport-configuration/#serializer_2","text":"Type: str , default: lightbus.serializers.BlobMessageSerializer The serializer to be used in converting the lightbus message into a bus-appropriate format.","title":"serializer"},{"location":"reference/transport-configuration/#deserializer_2","text":"Type: str , default: lightbus.serializers.BlobMessageDeserializer The deserializer to be used in converting the lightbus message into a bus-appropriate format.","title":"deserializer"},{"location":"reference/transport-configuration/#rpc_timeout_1","text":"Type: int , default: 5 , seconds How long to wait before we give up waiting for an RPC result to be received. Note on rpc_timeout This configuration option is also repeated in the API config . For now this value needs to be specified in three places: # The API config * RPC transport config (see above) * Result transport config","title":"rpc_timeout"},{"location":"reference/transport-configuration/#rpc_retry_delay_1","text":"Type: int , default: 1 , seconds How long to wait before reattempting to call the remote RPC. For example, in cases where Redis errors (e.g. connection issues). Execution will be retried once only.","title":"rpc_retry_delay"},{"location":"reference/transport-configuration/#consumption_restart_delay_2","text":"Type: int , default: 5 , seconds How long to wait before attempting to reconnect after loosing the connection to Redis.","title":"consumption_restart_delay"},{"location":"reference/transport-configuration/#redis-schema-transport-configuration","text":"","title":"Redis Schema Transport configuration"},{"location":"reference/transport-configuration/#url_3","text":"Type: str , default: redis://127.0.0.1:6379/0 The connection string for the redis server. Format is: `redis://host:port/db_number`","title":"url"},{"location":"reference/typing/","text":"Specifying type hints allows Lightbus to validate data types in both incoming and outgoing messages. Type hints are used to create your bus' schema , which is shared across your entire bus. Typing syntax for RPCs \u00b6 You can provide typing information for Remote Procedure Calls using regular Python type hinting: class AuthApi ( lightbus . Api ): class Meta : name = 'auth' def check_password ( self , username : str , password : str ) -> bool : return username == 'admin' and password == 'secret' This will: Ensure the received username parameter is a string Ensure the received password parameter is a string Ensure the returned value is a boolean This behaviour can be configured via the validate configuration option . Typing syntax for events \u00b6 Typing information for events is different to that for RPCs. Firstly, events do not provide return values. Secondly, event parameters are specified differently: # auth_service/bus.py from lightbus import Api , Event , Parameter class AuthApi ( Api ): # WITHOUT types user_created = Event ( parameters = ( 'username' , 'email' , 'is_admin' )) # WITH types user_created = Event ( parameters = ( Parameter ( 'username' , str ), Parameter ( 'new_email' , str ), Parameter ( 'is_admin' , bool , default = False ), )) This will: Ensure the received username parameter is a string Ensure the received new_email parameter is a string Ensure the received is_admin parameter is a boolean. If omitted, False will be used. This behaviour can be configured via the validate configuration option . Data structures \u00b6 In additional to built in types, Lightbus can derive typing information from the following data structures: Named Tuples Dataclasses Any class defining the __to_bus__ and __from_bus__ methods For each of these data structures Lightbus will: Encode values as JSON objects (i.e. dictionaries) Use the structure's type hints in generating the JSON schema... ... and therefore validate incoming/outgoing objects against this schema NamedTuple example \u00b6 # bus.py from lightbus import Api from typing import NamedTuple class User ( NamedTuple ): username : str name : str email : str is_admin : bool = False class AuthApi ( Api ): class Meta : name = 'auth' def get_user ( self , username : str ) -> User : return ... Dataclass example \u00b6 Lightbus supports dataclasses in the same way as it supports named tuples . For example: # bus.py from lightbus import Api from dataclasses import dataclass @dataclass () class User (): username : str name : str email : str is_admin : bool = False class AuthApi ( Api ): class Meta : name = 'auth' def get_user ( self , username : str ) -> User : return ... Custom class example \u00b6 Lightbus can also work with classes of any type provided that: The class defines a __from_bus__(self, value) class method, which returns an instance of the class. If value is a dict, it will be best-effort cased to the class' type annotations before __from_bus__ is invoked. The class defines a __to_bus__(self) method which both annotates its return type, and returns a value suitable for serialising on the bus. from lightbus import Api class User : username : str name : str email : str is_admin : bool = False def do_something ( self ): pass @classmethod def __from_bus__ ( cls , value ): user = cls () user . username = value [ \"username\" ] user . name = value [ \"name\" ] user . email = value [ \"email\" ] user . is_admin = value . get ( \"is_admin\" , False ) return user def __to_bus__ ( self ) -> dict : return dict ( username = self . username , name = self . name , email = self . email , is_admin = self . is_admin , ) class AuthApi ( Api ): class Meta : name = 'auth' def get_user ( self , username : str ) -> User : return ...","title":"Typing"},{"location":"reference/typing/#typing-syntax-for-rpcs","text":"You can provide typing information for Remote Procedure Calls using regular Python type hinting: class AuthApi ( lightbus . Api ): class Meta : name = 'auth' def check_password ( self , username : str , password : str ) -> bool : return username == 'admin' and password == 'secret' This will: Ensure the received username parameter is a string Ensure the received password parameter is a string Ensure the returned value is a boolean This behaviour can be configured via the validate configuration option .","title":"Typing syntax for RPCs"},{"location":"reference/typing/#typing-syntax-for-events","text":"Typing information for events is different to that for RPCs. Firstly, events do not provide return values. Secondly, event parameters are specified differently: # auth_service/bus.py from lightbus import Api , Event , Parameter class AuthApi ( Api ): # WITHOUT types user_created = Event ( parameters = ( 'username' , 'email' , 'is_admin' )) # WITH types user_created = Event ( parameters = ( Parameter ( 'username' , str ), Parameter ( 'new_email' , str ), Parameter ( 'is_admin' , bool , default = False ), )) This will: Ensure the received username parameter is a string Ensure the received new_email parameter is a string Ensure the received is_admin parameter is a boolean. If omitted, False will be used. This behaviour can be configured via the validate configuration option .","title":"Typing syntax for events"},{"location":"reference/typing/#data-structures","text":"In additional to built in types, Lightbus can derive typing information from the following data structures: Named Tuples Dataclasses Any class defining the __to_bus__ and __from_bus__ methods For each of these data structures Lightbus will: Encode values as JSON objects (i.e. dictionaries) Use the structure's type hints in generating the JSON schema... ... and therefore validate incoming/outgoing objects against this schema","title":"Data structures"},{"location":"reference/typing/#namedtuple-example","text":"# bus.py from lightbus import Api from typing import NamedTuple class User ( NamedTuple ): username : str name : str email : str is_admin : bool = False class AuthApi ( Api ): class Meta : name = 'auth' def get_user ( self , username : str ) -> User : return ...","title":"NamedTuple example"},{"location":"reference/typing/#dataclass-example","text":"Lightbus supports dataclasses in the same way as it supports named tuples . For example: # bus.py from lightbus import Api from dataclasses import dataclass @dataclass () class User (): username : str name : str email : str is_admin : bool = False class AuthApi ( Api ): class Meta : name = 'auth' def get_user ( self , username : str ) -> User : return ...","title":"Dataclass example"},{"location":"reference/typing/#custom-class-example","text":"Lightbus can also work with classes of any type provided that: The class defines a __from_bus__(self, value) class method, which returns an instance of the class. If value is a dict, it will be best-effort cased to the class' type annotations before __from_bus__ is invoked. The class defines a __to_bus__(self) method which both annotates its return type, and returns a value suitable for serialising on the bus. from lightbus import Api class User : username : str name : str email : str is_admin : bool = False def do_something ( self ): pass @classmethod def __from_bus__ ( cls , value ): user = cls () user . username = value [ \"username\" ] user . name = value [ \"name\" ] user . email = value [ \"email\" ] user . is_admin = value . get ( \"is_admin\" , False ) return user def __to_bus__ ( self ) -> dict : return dict ( username = self . username , name = self . name , email = self . email , is_admin = self . is_admin , ) class AuthApi ( Api ): class Meta : name = 'auth' def get_user ( self , username : str ) -> User : return ...","title":"Custom class example"},{"location":"reference/command-line-use/dumpconfigschema/","text":"lightbus dumpconfigschema \u00b6 This command will output a JSON schema for the global bus configuration file. The global bus configuration file is typically written as YAML, but it can also be written as JSON. In which case, you can validate the structure against the JSON schema produced by this command. This schema can also be loaded into some editors to provide auto-completion when editing your bus' configuration file. Important Be careful not to confuse this command with dumpschema . The dumpschema command dumps your bus' schema, whereas this dumpconfigschema simply dumps the schema for your bus' configuration. Examples \u00b6 Dump the configuration file schema to standard out: lightbus dumpschema Dump the configuration file schema to a file: lightbus dumpschema --out my_schema.json Schema for 3 APIs saved to my_schema.json Option reference \u00b6 $ lightbus dumpconfigschema --help usage: lightbus dumpconfigschema [-h] [--out FILE] [--bus BUS_MODULE] [--service-name SERVICE_NAME] [--process-name PROCESS_NAME] [--config FILE] [--log-level LOG_LEVEL] optional arguments: -h, --help show this help message and exit Dump config schema command arguments: --out FILE, -o FILE File to write config schema to. If omitted the schema will be written to standard out. (default: None) Common arguments: --bus BUS_MODULE, -b BUS_MODULE The bus module to import. Example 'bus', 'my_project.bus'. Defaults to the value of the LIGHTBUS_MODULE environment variable, or 'bus' (default: None) --service-name SERVICE_NAME, -s SERVICE_NAME Name of service in which this process resides. YOU SHOULD LIKELY SET THIS IN PRODUCTION. Can also be set using the LIGHTBUS_SERVICE_NAME environment. Will default to a random string. (default: None) --process-name PROCESS_NAME, -p PROCESS_NAME A unique name of this process within the service. Can also be set using the LIGHTBUS_PROCESS_NAME environment. Will default to a random string. (default: None) --config FILE Config file to load, JSON or YAML (default: None) --log-level LOG_LEVEL Set the log level. Overrides any value set in config. One of debug, info, warning, critical, exception. (default: None)","title":"dumpconfigschema"},{"location":"reference/command-line-use/dumpconfigschema/#lightbus-dumpconfigschema","text":"This command will output a JSON schema for the global bus configuration file. The global bus configuration file is typically written as YAML, but it can also be written as JSON. In which case, you can validate the structure against the JSON schema produced by this command. This schema can also be loaded into some editors to provide auto-completion when editing your bus' configuration file. Important Be careful not to confuse this command with dumpschema . The dumpschema command dumps your bus' schema, whereas this dumpconfigschema simply dumps the schema for your bus' configuration.","title":"lightbus dumpconfigschema"},{"location":"reference/command-line-use/dumpconfigschema/#examples","text":"Dump the configuration file schema to standard out: lightbus dumpschema Dump the configuration file schema to a file: lightbus dumpschema --out my_schema.json Schema for 3 APIs saved to my_schema.json","title":"Examples"},{"location":"reference/command-line-use/dumpconfigschema/#option-reference","text":"$ lightbus dumpconfigschema --help usage: lightbus dumpconfigschema [-h] [--out FILE] [--bus BUS_MODULE] [--service-name SERVICE_NAME] [--process-name PROCESS_NAME] [--config FILE] [--log-level LOG_LEVEL] optional arguments: -h, --help show this help message and exit Dump config schema command arguments: --out FILE, -o FILE File to write config schema to. If omitted the schema will be written to standard out. (default: None) Common arguments: --bus BUS_MODULE, -b BUS_MODULE The bus module to import. Example 'bus', 'my_project.bus'. Defaults to the value of the LIGHTBUS_MODULE environment variable, or 'bus' (default: None) --service-name SERVICE_NAME, -s SERVICE_NAME Name of service in which this process resides. YOU SHOULD LIKELY SET THIS IN PRODUCTION. Can also be set using the LIGHTBUS_SERVICE_NAME environment. Will default to a random string. (default: None) --process-name PROCESS_NAME, -p PROCESS_NAME A unique name of this process within the service. Can also be set using the LIGHTBUS_PROCESS_NAME environment. Will default to a random string. (default: None) --config FILE Config file to load, JSON or YAML (default: None) --log-level LOG_LEVEL Set the log level. Overrides any value set in config. One of debug, info, warning, critical, exception. (default: None)","title":"Option reference"},{"location":"reference/command-line-use/dumpschema/","text":"lightbus dumpschema \u00b6 The lightbus dumpschema command will dump the bus' JSON schema to either a file or standard out. This schema file can then be manually provided to lightbus run using the --schema option. Why is this useful? \u00b6 The idea behind this command is to aid in testing and local development. You can take a dump of your production bus' schema and use it in your local development or testing environment. This will allow Lightbus to validate your locally emitted events and RPCs against the expectations of your production environment. See manual validation for more information. Examples \u00b6 Dump the schema to standard out: lightbus dumpschema Dump the schema to a file: lightbus dumpschema --out my_schema.json Schema for 3 APIs saved to my_schema.json Options reference \u00b6 $ lightbus dumpschema --help usage: lightbus dumpschema [-h] [--out FILE_OR_DIRECTORY] [--bus BUS_MODULE] [--service-name SERVICE_NAME] [--process-name PROCESS_NAME] [--config FILE] [--log-level LOG_LEVEL] optional arguments: -h, --help show this help message and exit Dump config schema command arguments: --out FILE_OR_DIRECTORY, -o FILE_OR_DIRECTORY File or directory to write schema to. If a directory is specified one schema file will be created for each API. If omitted the schema will be written to standard out. (default: None) Common arguments: --bus BUS_MODULE, -b BUS_MODULE The bus module to import. Example 'bus', 'my_project.bus'. Defaults to the value of the LIGHTBUS_MODULE environment variable, or 'bus' (default: None) --service-name SERVICE_NAME, -s SERVICE_NAME Name of service in which this process resides. YOU SHOULD LIKELY SET THIS IN PRODUCTION. Can also be set using the LIGHTBUS_SERVICE_NAME environment. Will default to a random string. (default: None) --process-name PROCESS_NAME, -p PROCESS_NAME A unique name of this process within the service. Can also be set using the LIGHTBUS_PROCESS_NAME environment. Will default to a random string. (default: None) --config FILE Config file to load, JSON or YAML (default: None) --log-level LOG_LEVEL Set the log level. Overrides any value set in config. One of debug, info, warning, critical, exception. (default: None)","title":"dumpschema"},{"location":"reference/command-line-use/dumpschema/#lightbus-dumpschema","text":"The lightbus dumpschema command will dump the bus' JSON schema to either a file or standard out. This schema file can then be manually provided to lightbus run using the --schema option.","title":"lightbus dumpschema"},{"location":"reference/command-line-use/dumpschema/#why-is-this-useful","text":"The idea behind this command is to aid in testing and local development. You can take a dump of your production bus' schema and use it in your local development or testing environment. This will allow Lightbus to validate your locally emitted events and RPCs against the expectations of your production environment. See manual validation for more information.","title":"Why is this useful?"},{"location":"reference/command-line-use/dumpschema/#examples","text":"Dump the schema to standard out: lightbus dumpschema Dump the schema to a file: lightbus dumpschema --out my_schema.json Schema for 3 APIs saved to my_schema.json","title":"Examples"},{"location":"reference/command-line-use/dumpschema/#options-reference","text":"$ lightbus dumpschema --help usage: lightbus dumpschema [-h] [--out FILE_OR_DIRECTORY] [--bus BUS_MODULE] [--service-name SERVICE_NAME] [--process-name PROCESS_NAME] [--config FILE] [--log-level LOG_LEVEL] optional arguments: -h, --help show this help message and exit Dump config schema command arguments: --out FILE_OR_DIRECTORY, -o FILE_OR_DIRECTORY File or directory to write schema to. If a directory is specified one schema file will be created for each API. If omitted the schema will be written to standard out. (default: None) Common arguments: --bus BUS_MODULE, -b BUS_MODULE The bus module to import. Example 'bus', 'my_project.bus'. Defaults to the value of the LIGHTBUS_MODULE environment variable, or 'bus' (default: None) --service-name SERVICE_NAME, -s SERVICE_NAME Name of service in which this process resides. YOU SHOULD LIKELY SET THIS IN PRODUCTION. Can also be set using the LIGHTBUS_SERVICE_NAME environment. Will default to a random string. (default: None) --process-name PROCESS_NAME, -p PROCESS_NAME A unique name of this process within the service. Can also be set using the LIGHTBUS_PROCESS_NAME environment. Will default to a random string. (default: None) --config FILE Config file to load, JSON or YAML (default: None) --log-level LOG_LEVEL Set the log level. Overrides any value set in config. One of debug, info, warning, critical, exception. (default: None)","title":"Options reference"},{"location":"reference/command-line-use/inspect/","text":"lightbus inspect \u00b6 The lightbus inspect command allows for inspecting activity on the bus. This is currently limited to debugging only events, not RPCs. The querying facilities provided here are not very performant. The lightbus inspect command will pull a large number of messages from the bus and apply filters locally. Important lightbus inspect will only return results for APIs which are currently being served by worker processes. If you have no workers running you will see no results. This is because lightbus inspect relies on the schema being present on the bus. Examples \u00b6 You can query the bus for a specific lightbus event ID : # Will produce JSON output lightbus inspect --id 1bff06d0-fa9b-11e9-b8ca-f218986bf8ce # Will produce human-readable output lightbus inspect --id 1bff06d0-fa9b-11e9-b8ca-f218986bf8ce --format human You can also query the bus using the message ID native to the underlying broker (in this case, this will be the Redis streams message ID): lightbus inspect --native-id 1572389392059-0 You can also query by API and/or event name : # Event/api filtering lightbus inspect --event user_registered lightbus inspect --api my_company.auth Wildcards are also supported in event & api filtering: # Wildcard filtering lightbus inspect --event user_* lightbus inspect --api my_company.* Unexpected results when using wildcards? You may need to quote wilcard strings in order to prevent your shell expanding them. For example: lightbus inspect --api \"my_company.*\" You can see a continually updating stream of all events for a single API: # Continually show events for the given API lightbus inspect --follow --api auth You can query based on the data sent in the event . For example, to query based on the value of the email parameter of a fired event: lightbus inspect --json-path email=joe@example.com You can also query on more complex JSON path expressions : lightbus inspect --json-path address.city=London Cache \u00b6 This command will maintain an cache of fetched messages in ~/.lightbus/ . This will speed up subsequent executions and reduce the load on the bus. Option reference \u00b6 $ lightbus inspect -h usage: lightbus inspect [-h] [--json-path JSON_SEARCH] [--id LIGHTBUS_EVENT_ID] [--native-id NATIVE_EVENT_ID] [--api API_NAME] [--event EVENT_NAME] [--version VERSION_NUMBER] [--format FORMAT] [--cache-only] [--follow] [--validate] [--show-casting] [--internal] [--bus BUS_MODULE] [--service-name SERVICE_NAME] [--process-name PROCESS_NAME] [--config FILE] [--log-level LOG_LEVEL] optional arguments: -h, --help show this help message and exit Inspect command arguments: --json-path JSON_SEARCH, -j JSON_SEARCH Search event body json for the givn value. Eg. address.city=London (default: None) --id LIGHTBUS_EVENT_ID, -i LIGHTBUS_EVENT_ID Find a single event with this Lightbus event ID (default: None) --native-id NATIVE_EVENT_ID, -n NATIVE_EVENT_ID Find a single event with this broker-native ID (default: None) --api API_NAME, -a API_NAME Find events for this API name. Supports the '*' wildcard. (default: None) --event EVENT_NAME, -e EVENT_NAME Find events for this event name. Supports the '*' wildcard. (default: None) --version VERSION_NUMBER, -v VERSION_NUMBER Find events with the specified version number. Can be prefixed by <, >, <=, >= (default: None) --format FORMAT, -F FORMAT Formatting style. One of json, pretty, or human. (default: json) --cache-only, -c Search the local cache only (default: False) --follow, -f Continually listen for new events matching the search criteria. May only be used on a single API (default: False) --validate, -d Validate displayed events against the bus schema. Only available when --format is set to 'human' (experimental) (default: False) --show-casting, -C Show how the message values will be casted for each event listener (experimental) (default: False) --internal, -I Include internal APIs (default: False) Common arguments: --bus BUS_MODULE, -b BUS_MODULE The bus module to import. Example 'bus', 'my_project.bus'. Defaults to the value of the LIGHTBUS_MODULE environment variable, or 'bus' (default: None) --service-name SERVICE_NAME, -s SERVICE_NAME Name of service in which this process resides. YOU SHOULD LIKELY SET THIS IN PRODUCTION. Can also be set using the LIGHTBUS_SERVICE_NAME environment. Will default to a random string. (default: None) --process-name PROCESS_NAME, -p PROCESS_NAME A unique name of this process within the service. Can also be set using the LIGHTBUS_PROCESS_NAME environment. Will default to a random string. (default: None) --config FILE Config file to load, JSON or YAML (default: None) --log-level LOG_LEVEL Set the log level. Overrides any value set in config. One of debug, info, warning, critical, exception. (default: None)","title":"inspect"},{"location":"reference/command-line-use/inspect/#lightbus-inspect","text":"The lightbus inspect command allows for inspecting activity on the bus. This is currently limited to debugging only events, not RPCs. The querying facilities provided here are not very performant. The lightbus inspect command will pull a large number of messages from the bus and apply filters locally. Important lightbus inspect will only return results for APIs which are currently being served by worker processes. If you have no workers running you will see no results. This is because lightbus inspect relies on the schema being present on the bus.","title":"lightbus inspect"},{"location":"reference/command-line-use/inspect/#examples","text":"You can query the bus for a specific lightbus event ID : # Will produce JSON output lightbus inspect --id 1bff06d0-fa9b-11e9-b8ca-f218986bf8ce # Will produce human-readable output lightbus inspect --id 1bff06d0-fa9b-11e9-b8ca-f218986bf8ce --format human You can also query the bus using the message ID native to the underlying broker (in this case, this will be the Redis streams message ID): lightbus inspect --native-id 1572389392059-0 You can also query by API and/or event name : # Event/api filtering lightbus inspect --event user_registered lightbus inspect --api my_company.auth Wildcards are also supported in event & api filtering: # Wildcard filtering lightbus inspect --event user_* lightbus inspect --api my_company.* Unexpected results when using wildcards? You may need to quote wilcard strings in order to prevent your shell expanding them. For example: lightbus inspect --api \"my_company.*\" You can see a continually updating stream of all events for a single API: # Continually show events for the given API lightbus inspect --follow --api auth You can query based on the data sent in the event . For example, to query based on the value of the email parameter of a fired event: lightbus inspect --json-path email=joe@example.com You can also query on more complex JSON path expressions : lightbus inspect --json-path address.city=London","title":"Examples"},{"location":"reference/command-line-use/inspect/#cache","text":"This command will maintain an cache of fetched messages in ~/.lightbus/ . This will speed up subsequent executions and reduce the load on the bus.","title":"Cache"},{"location":"reference/command-line-use/inspect/#option-reference","text":"$ lightbus inspect -h usage: lightbus inspect [-h] [--json-path JSON_SEARCH] [--id LIGHTBUS_EVENT_ID] [--native-id NATIVE_EVENT_ID] [--api API_NAME] [--event EVENT_NAME] [--version VERSION_NUMBER] [--format FORMAT] [--cache-only] [--follow] [--validate] [--show-casting] [--internal] [--bus BUS_MODULE] [--service-name SERVICE_NAME] [--process-name PROCESS_NAME] [--config FILE] [--log-level LOG_LEVEL] optional arguments: -h, --help show this help message and exit Inspect command arguments: --json-path JSON_SEARCH, -j JSON_SEARCH Search event body json for the givn value. Eg. address.city=London (default: None) --id LIGHTBUS_EVENT_ID, -i LIGHTBUS_EVENT_ID Find a single event with this Lightbus event ID (default: None) --native-id NATIVE_EVENT_ID, -n NATIVE_EVENT_ID Find a single event with this broker-native ID (default: None) --api API_NAME, -a API_NAME Find events for this API name. Supports the '*' wildcard. (default: None) --event EVENT_NAME, -e EVENT_NAME Find events for this event name. Supports the '*' wildcard. (default: None) --version VERSION_NUMBER, -v VERSION_NUMBER Find events with the specified version number. Can be prefixed by <, >, <=, >= (default: None) --format FORMAT, -F FORMAT Formatting style. One of json, pretty, or human. (default: json) --cache-only, -c Search the local cache only (default: False) --follow, -f Continually listen for new events matching the search criteria. May only be used on a single API (default: False) --validate, -d Validate displayed events against the bus schema. Only available when --format is set to 'human' (experimental) (default: False) --show-casting, -C Show how the message values will be casted for each event listener (experimental) (default: False) --internal, -I Include internal APIs (default: False) Common arguments: --bus BUS_MODULE, -b BUS_MODULE The bus module to import. Example 'bus', 'my_project.bus'. Defaults to the value of the LIGHTBUS_MODULE environment variable, or 'bus' (default: None) --service-name SERVICE_NAME, -s SERVICE_NAME Name of service in which this process resides. YOU SHOULD LIKELY SET THIS IN PRODUCTION. Can also be set using the LIGHTBUS_SERVICE_NAME environment. Will default to a random string. (default: None) --process-name PROCESS_NAME, -p PROCESS_NAME A unique name of this process within the service. Can also be set using the LIGHTBUS_PROCESS_NAME environment. Will default to a random string. (default: None) --config FILE Config file to load, JSON or YAML (default: None) --log-level LOG_LEVEL Set the log level. Overrides any value set in config. One of debug, info, warning, critical, exception. (default: None)","title":"Option reference"},{"location":"reference/command-line-use/run/","text":"lightbus run \u00b6 The lightbus run command is used to start the Lightbus worker process and will be used in any Lightbus deployment. See the anatomy lesson for further details. Examples \u00b6 In its basic form lightbus run will expect to be able to import a module called bus which will contain your bus client: lightbus run You can also enable/disable specific features . For example, you may wish to run a worker which responds soley to RPCs in order to ensure a timely response: # Handles RPCs lightbus run --only rpcs # Handles everything else (events and tasks) lightbus run --skip rpcs A production use may look something like this (for more information on service & process names see the events explanation ): lightbus run \\ --bus my_company.my_project.bus \\ --service-name video-encoder \\ --process-name lightbus-worker-1 \\ --config /etc/lightbus/global-bus.yaml \\ --log-level debug This could also be re-written as: export LIGHTBUS_MODULE = my_company.my_project.bus export LIGHTBUS_SERVICE_NAME = video-encoder export LIGHTBUS_PROCESS_NAME = lightbus-worker-1 lightbus run \\ --config /etc/lightbus/global-bus.yaml \\ --log-level debug Configuration can also be loaded over HTTP or HTTPS . This may be useful if you wish to pull the global bus config from an (internal-only) endpoint/service: # Can load JSON/YAML over HTTP/HTTPS lightbus run --config http://config.internal/lightbus/global-bus.yaml Option reference \u00b6 $ lightbus run --help usage: lightbus run [-h] [--only ONLY] [--skip SKIP] [--schema FILE_OR_DIRECTORY] [--bus BUS_MODULE] [--service-name SERVICE_NAME] [--process-name PROCESS_NAME] [--config FILE] [--log-level LOG_LEVEL] optional arguments: -h, --help show this help message and exit Run command arguments: --only ONLY, -o ONLY Only provide the specified features. Comma separated list. Possible values: rpcs, events, tasks (default: None) --skip SKIP, -k SKIP Provide all except the specified features. Comma separated list. Possible values: rpcs, events, tasks (default: None) --schema FILE_OR_DIRECTORY, -m FILE_OR_DIRECTORY Manually load the schema from the given file or directory. This will normally be provided by the schema transport, but manual loading may be useful during development or testing. (default: None) Common arguments: --bus BUS_MODULE, -b BUS_MODULE The bus module to import. Example 'bus', 'my_project.bus'. Defaults to the value of the LIGHTBUS_MODULE environment variable, or 'bus' (default: None) --service-name SERVICE_NAME, -s SERVICE_NAME Name of service in which this process resides. YOU SHOULD LIKELY SET THIS IN PRODUCTION. Can also be set using the LIGHTBUS_SERVICE_NAME environment. Will default to a random string. (default: None) --process-name PROCESS_NAME, -p PROCESS_NAME A unique name of this process within the service. Can also be set using the LIGHTBUS_PROCESS_NAME environment. Will default to a random string. (default: None) --config FILE Config file to load, JSON or YAML (default: None) --log-level LOG_LEVEL Set the log level. Overrides any value set in config. One of debug, info, warning, critical, exception. (default: None)","title":"run"},{"location":"reference/command-line-use/run/#lightbus-run","text":"The lightbus run command is used to start the Lightbus worker process and will be used in any Lightbus deployment. See the anatomy lesson for further details.","title":"lightbus run"},{"location":"reference/command-line-use/run/#examples","text":"In its basic form lightbus run will expect to be able to import a module called bus which will contain your bus client: lightbus run You can also enable/disable specific features . For example, you may wish to run a worker which responds soley to RPCs in order to ensure a timely response: # Handles RPCs lightbus run --only rpcs # Handles everything else (events and tasks) lightbus run --skip rpcs A production use may look something like this (for more information on service & process names see the events explanation ): lightbus run \\ --bus my_company.my_project.bus \\ --service-name video-encoder \\ --process-name lightbus-worker-1 \\ --config /etc/lightbus/global-bus.yaml \\ --log-level debug This could also be re-written as: export LIGHTBUS_MODULE = my_company.my_project.bus export LIGHTBUS_SERVICE_NAME = video-encoder export LIGHTBUS_PROCESS_NAME = lightbus-worker-1 lightbus run \\ --config /etc/lightbus/global-bus.yaml \\ --log-level debug Configuration can also be loaded over HTTP or HTTPS . This may be useful if you wish to pull the global bus config from an (internal-only) endpoint/service: # Can load JSON/YAML over HTTP/HTTPS lightbus run --config http://config.internal/lightbus/global-bus.yaml","title":"Examples"},{"location":"reference/command-line-use/run/#option-reference","text":"$ lightbus run --help usage: lightbus run [-h] [--only ONLY] [--skip SKIP] [--schema FILE_OR_DIRECTORY] [--bus BUS_MODULE] [--service-name SERVICE_NAME] [--process-name PROCESS_NAME] [--config FILE] [--log-level LOG_LEVEL] optional arguments: -h, --help show this help message and exit Run command arguments: --only ONLY, -o ONLY Only provide the specified features. Comma separated list. Possible values: rpcs, events, tasks (default: None) --skip SKIP, -k SKIP Provide all except the specified features. Comma separated list. Possible values: rpcs, events, tasks (default: None) --schema FILE_OR_DIRECTORY, -m FILE_OR_DIRECTORY Manually load the schema from the given file or directory. This will normally be provided by the schema transport, but manual loading may be useful during development or testing. (default: None) Common arguments: --bus BUS_MODULE, -b BUS_MODULE The bus module to import. Example 'bus', 'my_project.bus'. Defaults to the value of the LIGHTBUS_MODULE environment variable, or 'bus' (default: None) --service-name SERVICE_NAME, -s SERVICE_NAME Name of service in which this process resides. YOU SHOULD LIKELY SET THIS IN PRODUCTION. Can also be set using the LIGHTBUS_SERVICE_NAME environment. Will default to a random string. (default: None) --process-name PROCESS_NAME, -p PROCESS_NAME A unique name of this process within the service. Can also be set using the LIGHTBUS_PROCESS_NAME environment. Will default to a random string. (default: None) --config FILE Config file to load, JSON or YAML (default: None) --log-level LOG_LEVEL Set the log level. Overrides any value set in config. One of debug, info, warning, critical, exception. (default: None)","title":"Option reference"},{"location":"reference/command-line-use/shell/","text":"lightbus shell \u00b6 The lightbus shell command provides an interactive prompt through which you can interface with the bus. To use this command you must first install bpython : pip install bpython Examples \u00b6 You should see the following when starting up the shell. This is a fully functional Python shell, with your bus loaded in a ready to be used: $ lightbus shell >>> \u2588 Welcome to the Lightbus shell. Use `bus` to access your bus.\\ Upon typing the shell will begin to auto-complete based on the locally available APIs: $ lightbus shell >>> bus.au\u2588 \u250c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2510 \u2502 auth \u2502 \u2514\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2518 You can fire events and call RPCs as follows: # Fire an event >>> bus.auth.user_registered.fire(email=\"joe@example.com\", username=\"joe\") # Call an RPC >>> bus.auth.check_password(username=\"admin\", password=\"secret\") True Option reference \u00b6 $ lightbus shell --help usage: lightbus shell [-h] [--bus BUS_MODULE] [--service-name SERVICE_NAME] [--process-name PROCESS_NAME] [--config FILE] [--log-level LOG_LEVEL] optional arguments: -h, --help show this help message and exit Common arguments: --bus BUS_MODULE, -b BUS_MODULE The bus module to import. Example 'bus', 'my_project.bus'. Defaults to the value of the LIGHTBUS_MODULE environment variable, or 'bus' (default: None) --service-name SERVICE_NAME, -s SERVICE_NAME Name of service in which this process resides. YOU SHOULD LIKELY SET THIS IN PRODUCTION. Can also be set using the LIGHTBUS_SERVICE_NAME environment. Will default to a random string. (default: None) --process-name PROCESS_NAME, -p PROCESS_NAME A unique name of this process within the service. Can also be set using the LIGHTBUS_PROCESS_NAME environment. Will default to a random string. (default: None) --config FILE Config file to load, JSON or YAML (default: None) --log-level LOG_LEVEL Set the log level. Overrides any value set in config. One of debug, info, warning, critical, exception. (default: None)","title":"shell"},{"location":"reference/command-line-use/shell/#lightbus-shell","text":"The lightbus shell command provides an interactive prompt through which you can interface with the bus. To use this command you must first install bpython : pip install bpython","title":"lightbus shell"},{"location":"reference/command-line-use/shell/#examples","text":"You should see the following when starting up the shell. This is a fully functional Python shell, with your bus loaded in a ready to be used: $ lightbus shell >>> \u2588 Welcome to the Lightbus shell. Use `bus` to access your bus.\\ Upon typing the shell will begin to auto-complete based on the locally available APIs: $ lightbus shell >>> bus.au\u2588 \u250c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2510 \u2502 auth \u2502 \u2514\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2518 You can fire events and call RPCs as follows: # Fire an event >>> bus.auth.user_registered.fire(email=\"joe@example.com\", username=\"joe\") # Call an RPC >>> bus.auth.check_password(username=\"admin\", password=\"secret\") True","title":"Examples"},{"location":"reference/command-line-use/shell/#option-reference","text":"$ lightbus shell --help usage: lightbus shell [-h] [--bus BUS_MODULE] [--service-name SERVICE_NAME] [--process-name PROCESS_NAME] [--config FILE] [--log-level LOG_LEVEL] optional arguments: -h, --help show this help message and exit Common arguments: --bus BUS_MODULE, -b BUS_MODULE The bus module to import. Example 'bus', 'my_project.bus'. Defaults to the value of the LIGHTBUS_MODULE environment variable, or 'bus' (default: None) --service-name SERVICE_NAME, -s SERVICE_NAME Name of service in which this process resides. YOU SHOULD LIKELY SET THIS IN PRODUCTION. Can also be set using the LIGHTBUS_SERVICE_NAME environment. Will default to a random string. (default: None) --process-name PROCESS_NAME, -p PROCESS_NAME A unique name of this process within the service. Can also be set using the LIGHTBUS_PROCESS_NAME environment. Will default to a random string. (default: None) --config FILE Config file to load, JSON or YAML (default: None) --log-level LOG_LEVEL Set the log level. Overrides any value set in config. One of debug, info, warning, critical, exception. (default: None)","title":"Option reference"},{"location":"reference/protocols/","text":"Protocols \u00b6 Here we define the specific interactions between Lightbus and its underlying communication medium, Redis. The intention is to provide enough information to allow services written in other languages to interact with Lightbus. Event protocol (Redis) RPC & result protocol (Redis) Schema protocol (Redis)","title":"Overview"},{"location":"reference/protocols/#protocols","text":"Here we define the specific interactions between Lightbus and its underlying communication medium, Redis. The intention is to provide enough information to allow services written in other languages to interact with Lightbus. Event protocol (Redis) RPC & result protocol (Redis) Schema protocol (Redis)","title":"Protocols"},{"location":"reference/protocols/event/","text":"Event protocol (Redis) \u00b6 Here we document the specific interactions between Lightbus and Redis. The concrete implementation of this is provided by the RedisEventTransport class. Before reading you should be familiar with Lightbus' data marshalling . This documentation may be useful when debugging, developing third-party client libraries, or simply for general interest and review. You do not need to be aware of this protocol in order to use Lightbus . Sending events \u00b6 The following command will send an event: XADD {stream_name} [MAXLEN ~ {max_stream_length}] * {field_name_1} {field_value_1} {field_name_2} {field_value_2}... {stream_name} \u00b6 The {stream_name} value is composed in one of the following ways: One stream per event: {api_name}.{event_name} One stream per API: {api_name}.* MAXLEN \u00b6 The MAXLEN ~ {max_stream_length} component is optional, but will be used by Lightbus to limit the stream to the approximate configured length. Fields \u00b6 Field names are strings. Field values are JSON-encoded strings. The following metadata fields must be sent: :id - A unique message ID :api_name - The API name for this event :event_name - The name of this event :version \u2013 The version of this event ( 1 is a sensible default value) Note that metadata fields are prefixed by a colon. User-specified fields should not include this colon. Lightbus does not currently provide specific functionality around the version field, but the field is available to developers via the EventMessage class. Lightbus may implement event functionality around event versions in future (such as event migrations). Consuming events \u00b6 Consuming events involves an initial setup stage in which we check for any events which this process has consumed yet failed to process (for example, due to an error, hardware failure, network problem, etc). We perform this initial check for events as follows: XREAD_GROUP {group_name} {consumer_name} {stream_name} 0 The 0 above indicates we wish to receive un-acknowledged events for this consumer (i.e this Lightbus process). Once we have received and processed any of these events, we can retrieve further events as follows: XREAD_GROUP {group_name} {consumer_name} {stream} > {group_name} \u00b6 The {group_name} value is comprised of the service name and listener name as follows: {service_name}-{listener_name} . {consumer_name} \u00b6 The {consumer_name} is set to the process name of the Lightbus process. {stream_name} \u00b6 The stream name, as described above in sending events . Reclaiming timed-out events \u00b6 Events can be considered timed if another Lightbus process has held onto them for too long. Any client consuming events should check for these from time to time. Timed out events can be claimed as follows: # Get pending messages XPENDING {stream} {group_name} - + {batch_size} # For each message try to claim it XCLAIM {stream} {group_name} {timeout} # If successful, process the event. Otherwise ignore it {batch_size} \u00b6 How many pending messages to fetch in each batch {timeout} \u00b6 Timeout in milliseconds. This is the maximum time a Lightbus process will have to process an event before another processes assumes it has failed and takes over. Encoding & Customisation \u00b6 See also: data marshalling By default field values are serialised using JSON. This can be This encoding is customisable within the Lightbus configuration. You are welcome to use something custom here, but be aware that: A single API must have a single encoding for all events on that API All clients accessing the bus must be configured to use the same custom encoding Data validation \u00b6 Validation of outgoing and incoming events is optional. However, validation of outgoing events is recommended as sending of event messages which fail validation may result in the message being rejected by any consumer. This validation can be performed using the using the schema available through the schema protocol . Data deformation & casting \u00b6 The Lightbus client provides Python-specific functionality to streamline the process of moving between Python data structures and interoperable JSON data structures. The level of functionality required in this regard is not specified here, and is left up to individual library developers.","title":"Event Protocol (Redis)"},{"location":"reference/protocols/event/#event-protocol-redis","text":"Here we document the specific interactions between Lightbus and Redis. The concrete implementation of this is provided by the RedisEventTransport class. Before reading you should be familiar with Lightbus' data marshalling . This documentation may be useful when debugging, developing third-party client libraries, or simply for general interest and review. You do not need to be aware of this protocol in order to use Lightbus .","title":"Event protocol (Redis)"},{"location":"reference/protocols/event/#sending-events","text":"The following command will send an event: XADD {stream_name} [MAXLEN ~ {max_stream_length}] * {field_name_1} {field_value_1} {field_name_2} {field_value_2}...","title":"Sending events"},{"location":"reference/protocols/event/#stream_name","text":"The {stream_name} value is composed in one of the following ways: One stream per event: {api_name}.{event_name} One stream per API: {api_name}.*","title":"{stream_name}"},{"location":"reference/protocols/event/#maxlen","text":"The MAXLEN ~ {max_stream_length} component is optional, but will be used by Lightbus to limit the stream to the approximate configured length.","title":"MAXLEN"},{"location":"reference/protocols/event/#fields","text":"Field names are strings. Field values are JSON-encoded strings. The following metadata fields must be sent: :id - A unique message ID :api_name - The API name for this event :event_name - The name of this event :version \u2013 The version of this event ( 1 is a sensible default value) Note that metadata fields are prefixed by a colon. User-specified fields should not include this colon. Lightbus does not currently provide specific functionality around the version field, but the field is available to developers via the EventMessage class. Lightbus may implement event functionality around event versions in future (such as event migrations).","title":"Fields"},{"location":"reference/protocols/event/#consuming-events","text":"Consuming events involves an initial setup stage in which we check for any events which this process has consumed yet failed to process (for example, due to an error, hardware failure, network problem, etc). We perform this initial check for events as follows: XREAD_GROUP {group_name} {consumer_name} {stream_name} 0 The 0 above indicates we wish to receive un-acknowledged events for this consumer (i.e this Lightbus process). Once we have received and processed any of these events, we can retrieve further events as follows: XREAD_GROUP {group_name} {consumer_name} {stream} >","title":"Consuming events"},{"location":"reference/protocols/event/#group_name","text":"The {group_name} value is comprised of the service name and listener name as follows: {service_name}-{listener_name} .","title":"{group_name}"},{"location":"reference/protocols/event/#consumer_name","text":"The {consumer_name} is set to the process name of the Lightbus process.","title":"{consumer_name}"},{"location":"reference/protocols/event/#stream_name_1","text":"The stream name, as described above in sending events .","title":"{stream_name}"},{"location":"reference/protocols/event/#reclaiming-timed-out-events","text":"Events can be considered timed if another Lightbus process has held onto them for too long. Any client consuming events should check for these from time to time. Timed out events can be claimed as follows: # Get pending messages XPENDING {stream} {group_name} - + {batch_size} # For each message try to claim it XCLAIM {stream} {group_name} {timeout} # If successful, process the event. Otherwise ignore it","title":"Reclaiming timed-out events"},{"location":"reference/protocols/event/#batch_size","text":"How many pending messages to fetch in each batch","title":"{batch_size}"},{"location":"reference/protocols/event/#timeout","text":"Timeout in milliseconds. This is the maximum time a Lightbus process will have to process an event before another processes assumes it has failed and takes over.","title":"{timeout}"},{"location":"reference/protocols/event/#encoding-customisation","text":"See also: data marshalling By default field values are serialised using JSON. This can be This encoding is customisable within the Lightbus configuration. You are welcome to use something custom here, but be aware that: A single API must have a single encoding for all events on that API All clients accessing the bus must be configured to use the same custom encoding","title":"Encoding & Customisation"},{"location":"reference/protocols/event/#data-validation","text":"Validation of outgoing and incoming events is optional. However, validation of outgoing events is recommended as sending of event messages which fail validation may result in the message being rejected by any consumer. This validation can be performed using the using the schema available through the schema protocol .","title":"Data validation"},{"location":"reference/protocols/event/#data-deformation-casting","text":"The Lightbus client provides Python-specific functionality to streamline the process of moving between Python data structures and interoperable JSON data structures. The level of functionality required in this regard is not specified here, and is left up to individual library developers.","title":"Data deformation & casting"},{"location":"reference/protocols/rpc-and-result/","text":"RPC & result protocol (Redis) \u00b6 Here we document the specific interactions between Lightbus and Redis. The concrete implementation of this is provided by the RedisRpcTransport and RedisResultTransport classes. Before reading you should be familiar with Lightbus' data marshalling . This documentation may be useful when debugging, developing third-party client libraries, or simply for general interest and review. You do not need to be aware of this protocol in order to use Lightbus . Sending RPC Calls (client) \u00b6 The following Redis commands will send a remote procedure call: RPUSH \"{api_name}:rpc_queue\" \"{blob_serialized_message}\" SET \"rpc_expiry_key:{rpc_call_message_id}\" EXPIRE \"rpc_expiry_key:{rpc_call_message_id}\" {rpc_timeout_seconds:5} See message serialisation & encoding for the format of {blob_serialized_message} . The return path \u00b6 Each RPC message must specify a return_path in its metadata. This states how the client expects to receive the result of the RPC. Both the client and the server must be able to comprehend and act up the provided return path, otherwise results will fail to be communicated. Assuming you are using the built-in Redis RPC result transport, the return path should be in the following format: redis+key://{api_name}.{rpc_name}:result:{rpc_call_message_id} The server will place the RPC result into a Redis key. The value following redis+key:// will be used as the key name. Receiving RPC Results (client) \u00b6 The following Redis commands will block and await the result of an RPC call: BLPOP \"{api_name}.{rpc_name}:result:{rpc_call_message_id}\" This simply waits from a value to appear within the key specified by the return_path in the sent RPC message. See message serialisation & encoding for the format of the returned RPC result. Consuming incoming RPC calls (worker) \u00b6 RPCs are consumed as follows: # Blocks until a RPC message is received BLPOP \"{api_name}:rpc_queue\" DEL \"rpc_expiry_key:{rpc_call_message_id}\" # If DEL returns 1 (key deleted) then execute the RPC # If DEL returns 0 (key did not exist) then ignore the RPC # Parse blob-serialised RPC message, Execute RPC, get result LPUSH {redis_key_specific_in_return_path} {blob_serialized_result} EXPIRE {redis_key_specific_in_return_path} {result_ttl_seconds} Message serialisation & encoding \u00b6 Above we have often referred to {blob_serialized_message} or {blob_serialized_result} . These are JSON blobs of data in a specific format. Serialisation \u00b6 See also: data marshalling Within the Redis RPC transports all messages are serialised into a single value. This value is referred to as a 'blob'. This serialisation is performed by the BlobMessageSerializer and BlobMessageDeserializer classes. RPC Message \u00b6 This is an outgoing RPC message body. This RPC message represents a request that an RPC be executed and a response be returned. { \"metadata\" : { # Unique base64-encoded UUID \"id\" : \"KrXz5EUXEem2gazeSAARIg==\" , # Fully qualified API name \"api_name\" : \"my_company.auth\" , # The name of the remote procedure call \"procedure_name\" : \"check_password\" , # How and where should the result be sent \"return_path\" : \"redis+key://my_company.auth.check_password:result:KrXz5EUXEem2gazeSAARIg==\" , }, # Key/value arguments potentiually needed to execute the remote procedure call \"kwargs\" : { \"username\" : \"adam\" , \"password\" : \"secret\" , ... } } Result Message \u00b6 This is the response message indicating an RPC has been (successfully or unsuccessfully) executed. { \"metadata\" : { # The newly randomly generated ID of this RPC result message. # (base64-encoded UUID) \"id\" : \"L4iXaEUgEemnBazeSAARIg==\" , # The ID of the received RPC call message \"rpc_message_id\" : \"KrXz5EUXEem2gazeSAARIg==\" , # String representation of any error which occurred \"error\" : \"\" , # Error stack trace. Only used in event of error, may be omitted otherwise \"trace\" : \" {error_stack_trace} \" , }, # The result as returned by the executed RPC \"result\" : ... } Encoding & Customisation \u00b6 See also: data marshalling The message must be encoded as a string once it has been serialised to the above blob structure. The default implementation is to use JSON as this encoding. This encoding is customisable within the Lightbus configuration. You are welcome to use something custom here, but be aware that: A single API must have a single encoding for all RPCs on that API All clients accessing the bus must be configured to use the same custom encoding Data validation \u00b6 Validation of outgoing parameters and incoming results is optional. However, validation of outgoing parameters is recommended as sending of RPC messages which fail validation may result in the message being rejected by the Lightbus worker. This validation can be performed using the using the schema available through the schema protocol . Data deformation & casting \u00b6 The Lightbus client provides Python-specific functionality to streamline the process of moving between Python data structures and interoperable JSON data structures. The level of functionality required in this regard is not specified here, and is left up to individual library developers.","title":"RPC & Result Protocol (Redis)"},{"location":"reference/protocols/rpc-and-result/#rpc-result-protocol-redis","text":"Here we document the specific interactions between Lightbus and Redis. The concrete implementation of this is provided by the RedisRpcTransport and RedisResultTransport classes. Before reading you should be familiar with Lightbus' data marshalling . This documentation may be useful when debugging, developing third-party client libraries, or simply for general interest and review. You do not need to be aware of this protocol in order to use Lightbus .","title":"RPC & result protocol (Redis)"},{"location":"reference/protocols/rpc-and-result/#sending-rpc-calls-client","text":"The following Redis commands will send a remote procedure call: RPUSH \"{api_name}:rpc_queue\" \"{blob_serialized_message}\" SET \"rpc_expiry_key:{rpc_call_message_id}\" EXPIRE \"rpc_expiry_key:{rpc_call_message_id}\" {rpc_timeout_seconds:5} See message serialisation & encoding for the format of {blob_serialized_message} .","title":"Sending RPC Calls (client)"},{"location":"reference/protocols/rpc-and-result/#the-return-path","text":"Each RPC message must specify a return_path in its metadata. This states how the client expects to receive the result of the RPC. Both the client and the server must be able to comprehend and act up the provided return path, otherwise results will fail to be communicated. Assuming you are using the built-in Redis RPC result transport, the return path should be in the following format: redis+key://{api_name}.{rpc_name}:result:{rpc_call_message_id} The server will place the RPC result into a Redis key. The value following redis+key:// will be used as the key name.","title":"The return path"},{"location":"reference/protocols/rpc-and-result/#receiving-rpc-results-client","text":"The following Redis commands will block and await the result of an RPC call: BLPOP \"{api_name}.{rpc_name}:result:{rpc_call_message_id}\" This simply waits from a value to appear within the key specified by the return_path in the sent RPC message. See message serialisation & encoding for the format of the returned RPC result.","title":"Receiving RPC Results (client)"},{"location":"reference/protocols/rpc-and-result/#consuming-incoming-rpc-calls-worker","text":"RPCs are consumed as follows: # Blocks until a RPC message is received BLPOP \"{api_name}:rpc_queue\" DEL \"rpc_expiry_key:{rpc_call_message_id}\" # If DEL returns 1 (key deleted) then execute the RPC # If DEL returns 0 (key did not exist) then ignore the RPC # Parse blob-serialised RPC message, Execute RPC, get result LPUSH {redis_key_specific_in_return_path} {blob_serialized_result} EXPIRE {redis_key_specific_in_return_path} {result_ttl_seconds}","title":"Consuming incoming RPC calls (worker)"},{"location":"reference/protocols/rpc-and-result/#message-serialisation-encoding","text":"Above we have often referred to {blob_serialized_message} or {blob_serialized_result} . These are JSON blobs of data in a specific format.","title":"Message serialisation & encoding"},{"location":"reference/protocols/rpc-and-result/#serialisation","text":"See also: data marshalling Within the Redis RPC transports all messages are serialised into a single value. This value is referred to as a 'blob'. This serialisation is performed by the BlobMessageSerializer and BlobMessageDeserializer classes.","title":"Serialisation"},{"location":"reference/protocols/rpc-and-result/#rpc-message","text":"This is an outgoing RPC message body. This RPC message represents a request that an RPC be executed and a response be returned. { \"metadata\" : { # Unique base64-encoded UUID \"id\" : \"KrXz5EUXEem2gazeSAARIg==\" , # Fully qualified API name \"api_name\" : \"my_company.auth\" , # The name of the remote procedure call \"procedure_name\" : \"check_password\" , # How and where should the result be sent \"return_path\" : \"redis+key://my_company.auth.check_password:result:KrXz5EUXEem2gazeSAARIg==\" , }, # Key/value arguments potentiually needed to execute the remote procedure call \"kwargs\" : { \"username\" : \"adam\" , \"password\" : \"secret\" , ... } }","title":"RPC Message"},{"location":"reference/protocols/rpc-and-result/#result-message","text":"This is the response message indicating an RPC has been (successfully or unsuccessfully) executed. { \"metadata\" : { # The newly randomly generated ID of this RPC result message. # (base64-encoded UUID) \"id\" : \"L4iXaEUgEemnBazeSAARIg==\" , # The ID of the received RPC call message \"rpc_message_id\" : \"KrXz5EUXEem2gazeSAARIg==\" , # String representation of any error which occurred \"error\" : \"\" , # Error stack trace. Only used in event of error, may be omitted otherwise \"trace\" : \" {error_stack_trace} \" , }, # The result as returned by the executed RPC \"result\" : ... }","title":"Result Message"},{"location":"reference/protocols/rpc-and-result/#encoding-customisation","text":"See also: data marshalling The message must be encoded as a string once it has been serialised to the above blob structure. The default implementation is to use JSON as this encoding. This encoding is customisable within the Lightbus configuration. You are welcome to use something custom here, but be aware that: A single API must have a single encoding for all RPCs on that API All clients accessing the bus must be configured to use the same custom encoding","title":"Encoding & Customisation"},{"location":"reference/protocols/rpc-and-result/#data-validation","text":"Validation of outgoing parameters and incoming results is optional. However, validation of outgoing parameters is recommended as sending of RPC messages which fail validation may result in the message being rejected by the Lightbus worker. This validation can be performed using the using the schema available through the schema protocol .","title":"Data validation"},{"location":"reference/protocols/rpc-and-result/#data-deformation-casting","text":"The Lightbus client provides Python-specific functionality to streamline the process of moving between Python data structures and interoperable JSON data structures. The level of functionality required in this regard is not specified here, and is left up to individual library developers.","title":"Data deformation & casting"},{"location":"reference/protocols/schema/","text":"Schema protocol (Redis) \u00b6 Here we document the specific interactions between Lightbus and Redis. The concrete implementation of this is provided by the RedisSchemaTransport class. Before reading you should be familiar with the schema explanation . This documentation may be useful when debugging, developing third-party client libraries, or simply for general interest and review. You do not need to be aware of this protocol in order to use Lightbus . Example API \u00b6 We will use the following API within the examples in the remainder of this page: from lightbus import Api , Event , Parameter class AuthApi ( Api ): user_registered = Event ( parameters = ( Parameter ( 'username' , str ), Parameter ( 'email' , str ), Parameter ( 'is_admin' , bool , default = False ), )) class Meta : name = 'my_company.auth' def check_password ( self , username : str , password : str ) -> bool : return username == 'admin' and password == 'secret' Schema format \u00b6 The Lightbus schema format is a JSON structure containing multiple JSON schemas. Below is the schema for the example my_company.auth API shown above (generalised format follows): // Auto-generated schema for auth API { \"my_company.auth\" : { // Events specify only parameters \"events\" : { \"user_registered\" : { \"parameters\" : { \"$schema\" : \"http://json-schema.org/draft-07/schema#\" , \"title\" : \"Event my_company.auth.user_registered parameters\" , \"type\" : \"object\" , \"username\" : { \"type\" : \"string\" }, \"properties\" : { \"email\" : { \"type\" : \"string\" }, \"is_admin\" : { \"default\" : false , \"type\" : \"boolean\" } }, \"required\" : [ \"username\" , \"email\" ], \"additionalProperties\" : false } } }, // RPCs specify both parameters and response \"rpcs\" : { \"check_password\" : { \"parameters\" : { \"$schema\" : \"http://json-schema.org/draft-07/schema#\" , \"title\" : \"RPC my_company.auth.check_password() parameters\" , \"type\" : \"object\" , \"properties\" : { \"username\" : { \"type\" : \"string\" }, \"password\" : { \"type\" : \"string\" } }, \"required\" : [ \"username\" , \"password\" ], \"additionalProperties\" : false }, \"response\" : { \"$schema\" : \"http://json-schema.org/draft-07/schema#\" , \"title\" : \"RPC my_company.auth.check_password() response\" , \"type\" : \"boolean\" } } } } } The generalised format is as follows: // Generalised Lightbus schema format { \"\" : { \"events\" : { \"\" : { \"parameters\" : { /* json schema */ } } // additional events }, \"rpcs\" : { \"\" : { \"parameters\" : { /* json schema */ }, \"response\" : { /* json schema */ } } // additional procedures } } // additional APIs } Per-schema Redis key \u00b6 Each schema is stored in redis as a string-ified JSON blob using a key formed as follows: schema:{api_name} For example: schema:my_company.auth Storing schemas \u00b6 A schema can be stored on the bus as follows: SET \"schema:{api_name}\" \"{json_schema}\" SADD \"schemas\" \"{api_name}\" EXPIRE \"schema:{api_name}\" \"{json_schema}\" {ttl_seconds} Where: {api_name} is replaced with the fully qualified API name (e.g. my_company.auth ) {json_schema} is replaced with the string-ified JSON structure detailed above {ttl_seconds} is replaced with the maximum time the schema should persist before being expired. Schemas expire to ensure shutdown Lightbus processes no longer advertise their APIs. Lightbus has a default value of 60 seconds. This process must be repeated at least every ttl_seconds in order to keep the schema active on the bus. Loading schemas \u00b6 Available schemas are loaded as follows: schemas = SMEMBERS \"schemas\" for api_name in scheams: GET \"schema:{api_name}\" A schema is only available if the GET succeeds. The data returned by the GET should be JSON decoded, and will contain the structure detailed above.","title":"Schema Protocol (Redis)"},{"location":"reference/protocols/schema/#schema-protocol-redis","text":"Here we document the specific interactions between Lightbus and Redis. The concrete implementation of this is provided by the RedisSchemaTransport class. Before reading you should be familiar with the schema explanation . This documentation may be useful when debugging, developing third-party client libraries, or simply for general interest and review. You do not need to be aware of this protocol in order to use Lightbus .","title":"Schema protocol (Redis)"},{"location":"reference/protocols/schema/#example-api","text":"We will use the following API within the examples in the remainder of this page: from lightbus import Api , Event , Parameter class AuthApi ( Api ): user_registered = Event ( parameters = ( Parameter ( 'username' , str ), Parameter ( 'email' , str ), Parameter ( 'is_admin' , bool , default = False ), )) class Meta : name = 'my_company.auth' def check_password ( self , username : str , password : str ) -> bool : return username == 'admin' and password == 'secret'","title":"Example API"},{"location":"reference/protocols/schema/#schema-format","text":"The Lightbus schema format is a JSON structure containing multiple JSON schemas. Below is the schema for the example my_company.auth API shown above (generalised format follows): // Auto-generated schema for auth API { \"my_company.auth\" : { // Events specify only parameters \"events\" : { \"user_registered\" : { \"parameters\" : { \"$schema\" : \"http://json-schema.org/draft-07/schema#\" , \"title\" : \"Event my_company.auth.user_registered parameters\" , \"type\" : \"object\" , \"username\" : { \"type\" : \"string\" }, \"properties\" : { \"email\" : { \"type\" : \"string\" }, \"is_admin\" : { \"default\" : false , \"type\" : \"boolean\" } }, \"required\" : [ \"username\" , \"email\" ], \"additionalProperties\" : false } } }, // RPCs specify both parameters and response \"rpcs\" : { \"check_password\" : { \"parameters\" : { \"$schema\" : \"http://json-schema.org/draft-07/schema#\" , \"title\" : \"RPC my_company.auth.check_password() parameters\" , \"type\" : \"object\" , \"properties\" : { \"username\" : { \"type\" : \"string\" }, \"password\" : { \"type\" : \"string\" } }, \"required\" : [ \"username\" , \"password\" ], \"additionalProperties\" : false }, \"response\" : { \"$schema\" : \"http://json-schema.org/draft-07/schema#\" , \"title\" : \"RPC my_company.auth.check_password() response\" , \"type\" : \"boolean\" } } } } } The generalised format is as follows: // Generalised Lightbus schema format { \"\" : { \"events\" : { \"\" : { \"parameters\" : { /* json schema */ } } // additional events }, \"rpcs\" : { \"\" : { \"parameters\" : { /* json schema */ }, \"response\" : { /* json schema */ } } // additional procedures } } // additional APIs }","title":"Schema format"},{"location":"reference/protocols/schema/#per-schema-redis-key","text":"Each schema is stored in redis as a string-ified JSON blob using a key formed as follows: schema:{api_name} For example: schema:my_company.auth","title":"Per-schema Redis key"},{"location":"reference/protocols/schema/#storing-schemas","text":"A schema can be stored on the bus as follows: SET \"schema:{api_name}\" \"{json_schema}\" SADD \"schemas\" \"{api_name}\" EXPIRE \"schema:{api_name}\" \"{json_schema}\" {ttl_seconds} Where: {api_name} is replaced with the fully qualified API name (e.g. my_company.auth ) {json_schema} is replaced with the string-ified JSON structure detailed above {ttl_seconds} is replaced with the maximum time the schema should persist before being expired. Schemas expire to ensure shutdown Lightbus processes no longer advertise their APIs. Lightbus has a default value of 60 seconds. This process must be repeated at least every ttl_seconds in order to keep the schema active on the bus.","title":"Storing schemas"},{"location":"reference/protocols/schema/#loading-schemas","text":"Available schemas are loaded as follows: schemas = SMEMBERS \"schemas\" for api_name in scheams: GET \"schema:{api_name}\" A schema is only available if the GET succeeds. The data returned by the GET should be JSON decoded, and will contain the structure detailed above.","title":"Loading schemas"},{"location":"tutorial/","text":"Tutorial overview \u00b6 These tutorials will give you a practical concrete introduction to Lightbus . We will link to concepts as we go, but the aim here is to get you up and running quickly. Do you prefer to read the theory first? Feel free to start with the explanation section and come back here later. We recommend you approach the tutorials in the following order: Installation Quick start Worked example After completing these tutorials you should make sure you look over the explanation section.","title":"Overview"},{"location":"tutorial/#tutorial-overview","text":"These tutorials will give you a practical concrete introduction to Lightbus . We will link to concepts as we go, but the aim here is to get you up and running quickly. Do you prefer to read the theory first? Feel free to start with the explanation section and come back here later. We recommend you approach the tutorials in the following order: Installation Quick start Worked example After completing these tutorials you should make sure you look over the explanation section.","title":"Tutorial overview"},{"location":"tutorial/django-and-lightbus/","text":"If you get stuck... It is really useful to hear from people who have encountered a problem or got stuck. Hearing from you means we can improve our documentation and error messages. If you get stuck drop then please drop an email to adam@adamcharnock.com , visit the Lightbus discord server , or call me (Adam) on +442032896620. The more information you can include the better (problem description, screenshots, and code are all useful) Lightbus is designed to work smoothly with Django. A few of additions are needed to your bus.py file in order to setup django correctly. You should perform these at the very start of your @bus.client.on_start() function within your bus.py : Set your DJANGO_SETTINGS_MODULE environment variable 1 Call django.setup() Decorate RPCs and handlers which use the Django ORM with @uses_django_db . This will ensure database errors are cleaned up correctly. Simple example \u00b6 The following example demonstrates: Setting DJANGO_SETTINGS_MODULE Calling django.setup() # bus.py import lightbus import django import os # Do NOT import your django models here bus = lightbus . create () # ...Define and register APIs here... @bus . client . on_start () def on_start ( ** kwargs ): # DJANGO STEP 1: Set DJANGO_SETTINGS_MODULE\\ # Customise this value (if necessary) to point to your settings module. # This should be the same as the corresponding line in your manage.py file. os . environ . setdefault ( \"DJANGO_SETTINGS_MODULE\" , \"settings\" ) # DJANGO STEP 2: Call django.setup() # We must do this before we import any of our django models django . setup () # You can now safely import your django models here if needed, # or within any event handlers we setup (see next example) from my_app.models import MyModel Practical example \u00b6 Here we build upon the simple example (above). This example includes an API and an event handler, and is loosely based upon our work in the quick start tutorial . Here we demonstrate: Setting DJANGO_SETTINGS_MODULE Calling django.setup() Applying @uses_django_db to an RPC and event handler import lightbus from lightbus.utilities.django import uses_django_db import django import os bus = lightbus . create () class AuthApi ( lightbus . Api ): user_registered = lightbus . Event ( parameters = ( 'username' , 'email' )) class Meta : name = 'auth' # Decorating our RPC with @uses_django_db # ensures database connections are cleaned up @uses_django_db def check_password ( self , username , password ): # Import django models here, once django.setup() has been called from django.contrib.auth.models import User try : user = User . objects . get ( username = username ) return user . check_password ( password ) except User . DoesNotExist : return False bus . client . register_api ( AuthApi ()) # Decorating our event handler with @uses_django_db # ensures database connections are cleaned up @uses_django_db def handle_new_user ( event , username , email ): # For example, we may want to create an audit log entry from my_app.models import AuditLogEntry AuditLogEntry . objects . create ( message = f \"User { username } created\" ) @bus . client . on_start () def on_start ( ** kwargs ): os . environ . setdefault ( \"DJANGO_SETTINGS_MODULE\" , \"my_company.a_project.settings\" ) django . setup () # Setup a listener for the 'auth.user_registered' event, to be # handled by handle_new_user() (above) bus . auth . user_registered . listen ( handle_new_user , listener_name = \"create_audit_entry\" ) Further development Simplification to the Django setup process is tracked in GitHub . You will see that your Django project' manage.py file also performs this same step \u21a9","title":"4. Django & Lightbus"},{"location":"tutorial/django-and-lightbus/#simple-example","text":"The following example demonstrates: Setting DJANGO_SETTINGS_MODULE Calling django.setup() # bus.py import lightbus import django import os # Do NOT import your django models here bus = lightbus . create () # ...Define and register APIs here... @bus . client . on_start () def on_start ( ** kwargs ): # DJANGO STEP 1: Set DJANGO_SETTINGS_MODULE\\ # Customise this value (if necessary) to point to your settings module. # This should be the same as the corresponding line in your manage.py file. os . environ . setdefault ( \"DJANGO_SETTINGS_MODULE\" , \"settings\" ) # DJANGO STEP 2: Call django.setup() # We must do this before we import any of our django models django . setup () # You can now safely import your django models here if needed, # or within any event handlers we setup (see next example) from my_app.models import MyModel","title":"Simple example"},{"location":"tutorial/django-and-lightbus/#practical-example","text":"Here we build upon the simple example (above). This example includes an API and an event handler, and is loosely based upon our work in the quick start tutorial . Here we demonstrate: Setting DJANGO_SETTINGS_MODULE Calling django.setup() Applying @uses_django_db to an RPC and event handler import lightbus from lightbus.utilities.django import uses_django_db import django import os bus = lightbus . create () class AuthApi ( lightbus . Api ): user_registered = lightbus . Event ( parameters = ( 'username' , 'email' )) class Meta : name = 'auth' # Decorating our RPC with @uses_django_db # ensures database connections are cleaned up @uses_django_db def check_password ( self , username , password ): # Import django models here, once django.setup() has been called from django.contrib.auth.models import User try : user = User . objects . get ( username = username ) return user . check_password ( password ) except User . DoesNotExist : return False bus . client . register_api ( AuthApi ()) # Decorating our event handler with @uses_django_db # ensures database connections are cleaned up @uses_django_db def handle_new_user ( event , username , email ): # For example, we may want to create an audit log entry from my_app.models import AuditLogEntry AuditLogEntry . objects . create ( message = f \"User { username } created\" ) @bus . client . on_start () def on_start ( ** kwargs ): os . environ . setdefault ( \"DJANGO_SETTINGS_MODULE\" , \"my_company.a_project.settings\" ) django . setup () # Setup a listener for the 'auth.user_registered' event, to be # handled by handle_new_user() (above) bus . auth . user_registered . listen ( handle_new_user , listener_name = \"create_audit_entry\" ) Further development Simplification to the Django setup process is tracked in GitHub . You will see that your Django project' manage.py file also performs this same step \u21a9","title":"Practical example"},{"location":"tutorial/getting-involved/","text":"Getting involved (even if you don't code) \u00b6 Lightbus can only thrive as a project if it has new contributors getting involved, this means we need you! \"But I've never contributed to open source before,\" I hear you cry. Well, it turns out most people have never contributed to open source before either, so don't worry, you're not alone. The Lightbus team (which currently consists of me, Adam) is here to support you. Code of Conduct \u00b6 Make sure you read over the Code of Conduct . This will outline how you should engage with others, and how you should expect to be treated yourself. How to get started (non-coding) \u00b6 There are lots of ways you can get even without getting stuck into code. Even if you are a Python developer, this can still be a good place to start: Say hello in our community chat . We can have a chat about ways in which you can help. Raise an issue and describe something you find confusing about Lightbus. Something you think is not clear or missing from the documentation. Submit a pull request to improve the documentation (for example, I'm sure you can find some typos) Tell me how you are using (or would like to use) Lightbus. This kind of insight is invaluable. How to get started (coding) \u00b6 If you want to get stuck in to writing some code here are some good places to start: See how to modify Lightbus for a guide to getting your local development environment setup. If you have a specific idea or bug you would like to fix then come and discuss it , either in our community chat or in a GitHub issue . It would be great to meet you, and I may be able to advise you of any pitfalls to be aware of. Read over the code to get a feel for how things work. For a high-level view take a look at the BusClient class. For a low-level view of the messaging system take a look at the redis transports ( lightbus/transports/redis ). You'll likely find some typo's as you go, so feel free to submit a pull request ! Work on one of the project's issues . I recommend you talk your plan over me first, that way we can make sure you develop something that fits with the the project as a whole.","title":"Getting involved (even if you don't code)"},{"location":"tutorial/getting-involved/#getting-involved-even-if-you-dont-code","text":"Lightbus can only thrive as a project if it has new contributors getting involved, this means we need you! \"But I've never contributed to open source before,\" I hear you cry. Well, it turns out most people have never contributed to open source before either, so don't worry, you're not alone. The Lightbus team (which currently consists of me, Adam) is here to support you.","title":"Getting involved (even if you don't code)"},{"location":"tutorial/getting-involved/#code-of-conduct","text":"Make sure you read over the Code of Conduct . This will outline how you should engage with others, and how you should expect to be treated yourself.","title":"Code of Conduct"},{"location":"tutorial/getting-involved/#how-to-get-started-non-coding","text":"There are lots of ways you can get even without getting stuck into code. Even if you are a Python developer, this can still be a good place to start: Say hello in our community chat . We can have a chat about ways in which you can help. Raise an issue and describe something you find confusing about Lightbus. Something you think is not clear or missing from the documentation. Submit a pull request to improve the documentation (for example, I'm sure you can find some typos) Tell me how you are using (or would like to use) Lightbus. This kind of insight is invaluable.","title":"How to get started (non-coding)"},{"location":"tutorial/getting-involved/#how-to-get-started-coding","text":"If you want to get stuck in to writing some code here are some good places to start: See how to modify Lightbus for a guide to getting your local development environment setup. If you have a specific idea or bug you would like to fix then come and discuss it , either in our community chat or in a GitHub issue . It would be great to meet you, and I may be able to advise you of any pitfalls to be aware of. Read over the code to get a feel for how things work. For a high-level view take a look at the BusClient class. For a low-level view of the messaging system take a look at the redis transports ( lightbus/transports/redis ). You'll likely find some typo's as you go, so feel free to submit a pull request ! Work on one of the project's issues . I recommend you talk your plan over me first, that way we can make sure you develop something that fits with the the project as a whole.","title":"How to get started (coding)"},{"location":"tutorial/installation/","text":"If you get stuck... It is really useful to hear from people who have encountered a problem or got stuck. Hearing from you means we can improve our documentation and error messages. If you get stuck drop then please drop an email to adam@adamcharnock.com , visit the Lightbus discord server , or call me (Adam) on +442032896620. The more information you can include the better (problem description, screenshots, and code are all useful) 1.1. Preface: Installing Python 3.7 (or above) \u00b6 Lightbus requires Python 3.7 or newer. This is available for all major operating systems. Python 3.7 on macOS \u00b6 You can check your current version of Python as follows: $ python3 --version Python 3.7.2 You need version 3.7 or above to run Lightbus. If you are running an older version of Python you can install a newer version via one of the following methods: Install Python 3.7 using Homebrew \u2013 This is the easiest option, you will install that latest version of Python 3. Install Python 3.7 using Homebrew + pyenv \u2013 This option has some additional steps, but you will have complete control over the Python versions available to you. If you work on multiple Python projects this may be more suitable. Install Python 3.7 manually \u2013 Not recommended Python 3.7 on Linux \u00b6 Your Linux distribution may already come with Python 3.7 installed. You can check your Python version as follows: $ python3 --version Python 3.7.2 You need version 3.7 or above to run Lightbus. Digital Ocean has a beginner-suitable guide on installing Python 3 which you may find useful. If you require more granular control of your python versions you may find pyenv more suitable. Windows \u00b6 Lightbus is not currently tested for deployment on Windows, so your millage may vary. The Hitchhiker's Guide to Python covers installing Python 3 on Windows . 1.2. Installing Lightbus \u00b6 Installing using pip (recommended) \u00b6 $ pip3 install lightbus Installing using git \u00b6 This will clone the bleeding-edge version Lightbus and install it ready to use. This is useful if you need the latest (albeit unstable) changes, or if you wish to modify the Lightbus source. $ pip install https://github.com/adamcharnock/lightbus.git#egg=lightbus 1.3. Installing Redis \u00b6 You will need Redis 5.0 or above in order to use Lightbus. You can install Redis 5.0 on macOS by either: Using Homebrew ( brew install redis ), or Using docker ( docker run --rm -p 6379:6379 -d redis ) 1.4. Check it works \u00b6 You should now have: Python 3.7 or above installed Lightbus installed Redis installed and running You check check everything is setup correctly by starting up lightbus. First create a file name bus.py which contains: # Create a bus.py file with the following contents import lightbus bus = lightbus . create () Now run the command: $ lightbus run Lightbus should start without errors and wait for messages. You can exit using Ctrl + C . This Lightbus process isn't setup to do anything yet, but its a start! 1.5 Next \u00b6 Now you are up and running, take a look at the quick start tutorial .","title":"1. Installation"},{"location":"tutorial/installation/#11-preface-installing-python-37-or-above","text":"Lightbus requires Python 3.7 or newer. This is available for all major operating systems.","title":"1.1. Preface: Installing Python 3.7 (or above)"},{"location":"tutorial/installation/#python-37-on-macos","text":"You can check your current version of Python as follows: $ python3 --version Python 3.7.2 You need version 3.7 or above to run Lightbus. If you are running an older version of Python you can install a newer version via one of the following methods: Install Python 3.7 using Homebrew \u2013 This is the easiest option, you will install that latest version of Python 3. Install Python 3.7 using Homebrew + pyenv \u2013 This option has some additional steps, but you will have complete control over the Python versions available to you. If you work on multiple Python projects this may be more suitable. Install Python 3.7 manually \u2013 Not recommended","title":"Python 3.7 on macOS"},{"location":"tutorial/installation/#python-37-on-linux","text":"Your Linux distribution may already come with Python 3.7 installed. You can check your Python version as follows: $ python3 --version Python 3.7.2 You need version 3.7 or above to run Lightbus. Digital Ocean has a beginner-suitable guide on installing Python 3 which you may find useful. If you require more granular control of your python versions you may find pyenv more suitable.","title":"Python 3.7 on Linux"},{"location":"tutorial/installation/#windows","text":"Lightbus is not currently tested for deployment on Windows, so your millage may vary. The Hitchhiker's Guide to Python covers installing Python 3 on Windows .","title":"Windows"},{"location":"tutorial/installation/#12-installing-lightbus","text":"","title":"1.2. Installing Lightbus"},{"location":"tutorial/installation/#installing-using-pip-recommended","text":"$ pip3 install lightbus","title":"Installing using pip (recommended)"},{"location":"tutorial/installation/#installing-using-git","text":"This will clone the bleeding-edge version Lightbus and install it ready to use. This is useful if you need the latest (albeit unstable) changes, or if you wish to modify the Lightbus source. $ pip install https://github.com/adamcharnock/lightbus.git#egg=lightbus","title":"Installing using git"},{"location":"tutorial/installation/#13-installing-redis","text":"You will need Redis 5.0 or above in order to use Lightbus. You can install Redis 5.0 on macOS by either: Using Homebrew ( brew install redis ), or Using docker ( docker run --rm -p 6379:6379 -d redis )","title":"1.3. Installing Redis"},{"location":"tutorial/installation/#14-check-it-works","text":"You should now have: Python 3.7 or above installed Lightbus installed Redis installed and running You check check everything is setup correctly by starting up lightbus. First create a file name bus.py which contains: # Create a bus.py file with the following contents import lightbus bus = lightbus . create () Now run the command: $ lightbus run Lightbus should start without errors and wait for messages. You can exit using Ctrl + C . This Lightbus process isn't setup to do anything yet, but its a start!","title":"1.4. Check it works"},{"location":"tutorial/installation/#15-next","text":"Now you are up and running, take a look at the quick start tutorial .","title":"1.5 Next"},{"location":"tutorial/quick-start/","text":"If you get stuck... It is really useful to hear from people who have encountered a problem or got stuck. Hearing from you means we can improve our documentation and error messages. If you get stuck drop then please drop an email to adam@adamcharnock.com , visit the Lightbus discord server , or call me (Adam) on +442032896620. The more information you can include the better (problem description, screenshots, and code are all useful) 2.1 Requirements \u00b6 Before continuing, ensure you have completed the following steps detailed in the installation section : Installed Python 3.7 or above Installed Lightbus Running Redis 5 locally on the default port (6379) Optionally, you can read some additional explanation in the anatomy lesson section. 2.2 Define your API \u00b6 First we will define an API for Lightbus to serve. This will be an authentication API and will live in an authentication service. Create an auth_service directory and within there create the following in a bus.py file: # File: auth_service/bus.py import lightbus # Create your service's bus client. You can import this elsewere # in your service's codebase in order to access the bus bus = lightbus . create () class AuthApi ( lightbus . Api ): user_registered = lightbus . Event ( parameters = ( 'username' , 'email' )) class Meta : name = 'auth' def check_password ( self , username , password ): return username == 'admin' and password == 'secret' # Register this API with Lightbus. Lightbus will respond to # remote procedure calls for registered APIs, as well as allow you # as the developer to fire events on any registered APIs. bus . client . register_api ( AuthApi ()) You should now be able to startup Lightbus as follows: cd ./auth_service/ lightbus run Lightbus will output some logging data which will include a list of registered APIs, including your new auth API: Congratulations, you have now created an authentication service which contains a single API named auth . Leave Lightbus running and open a new terminal window for the next stage. 2.3 Remote procedure calls \u00b6 With Lightbus still running in another terminal window, create a new directory alongside auth_service called another_service . This will be an example service which will interact with the auth service. You should now have the following structure: ./auth_service/bus.py , created above ./another_service/ , which we will create files within now. We always define a service's bus client within a bus.py file. Therefore, create ./another_service/bus.py containing the following: # File: ./another_service/bus.py import lightbus bus = lightbus . create () Now create ./another_service/check_password.py . We will use this to experiment with making RPC calls: # File: ./another_service/check_password.py # Import our service's bus client from .bus import bus # Call the check_password() procedure on our auth API valid = bus . auth . check_password ( username = 'admin' , password = 'secret' ) # Show the result if valid : print ( 'Password valid!' ) else : print ( 'Oops, bad username or password' ) Running this script should show you the following: cd ./another_service/ python3 ./check_password.py Password valid! Looking at the other terminal window you have open you should see that Lightbus has also reported that it is handled a remote procedure call. 2.4 Events \u00b6 Events allow services to broadcast a message to any other services which care to listen. Your service can fire events on any API which has been registered. We have already created and registered our AuthApi in ./auth_service/bus.py which provides the user_registered event. Firing events \u00b6 Let's write a simple script to manually register users and fire events. Note we can only fire events for APIs we have registered. We have registered the auth API in auth_service , so it is within this service that this new script must reside. We could not put the script within another_service as this service does not include the AuthApi class and can therefore not register it. # ./auth_service/manually_register_user.py # Import the service's bus client from bus.py from bus import bus print ( \"New user creation\" ) new_username = input ( \"Enter a username: \" ) . strip () new_email = input ( \"Enter the user's email address: \" ) . strip () # You would normally store the new user in your database # at this point. We don't show this here for simplicity. # Let the bus know a user has been registered by firing the event bus . auth . user_registered . fire ( username = new_username , email = new_email ) print ( \"Done\" ) Run this script using: cd ./auth_service/ python3 manually_register_user.py You should be prompted for a new username & email address, at which point an event will be fired onto the bus. We will make use of this event in the next section. Listening for events \u00b6 Perhaps another_service needs to be notified when a user is created. To achieve this we can setup an event listener by modifying ./another_service/bus.py : # File: ./another_service/bus.py import lightbus bus = lightbus . create () def handle_new_user ( event , username , email ): print ( f \"A new user was created in the authentication service:\" ) print ( f \" Username: { username } \" ) print ( f \" Email: { email } \" ) @bus . client . on_start () def bus_start ( client ): bus . auth . user_registered . listen ( handle_new_user , listener_name = \"print_on_new_registration\" ) Listening for events requires your service to sit waiting for something to happen. Sitting around and waiting for something to happen is precisely what the lightbus run console command is for. Therefore, in one terminal window startup the lightbus run command for another_service : # In the first terminal window: cd ./another_service/ lightbus run Your service is now waiting for events. In the second terminal window let's cause an event to be fired using the script we wrote in the previous section: cd ./auth_service/ python3 manually_register_user.py You should see that the event gets sent by the manually_register_user.py script within the auth_service , and received by by the lightbus run process within the another_service . Listener names \u00b6 You may have noticed we specified the listener_name parameter in the above example, and we set it to an arbitrary value of \"print_on_new_registration\" . This is important, because Lightbus tracks which listeners have received which events. It is this listener name which is used to uniquely identify each listener. You don't need to worry about this often, just follow the rule: Ensure you specify a unique listener_name every time you setup a new listener . The listener name should be unique within the service. Two listeners with the same name will only receive a partial set of events each. 2.5 Next \u00b6 This was a simple example to get you started. The worked example considers a more realistic scenario involving multiple services.","title":"2. Quick start"},{"location":"tutorial/quick-start/#21-requirements","text":"Before continuing, ensure you have completed the following steps detailed in the installation section : Installed Python 3.7 or above Installed Lightbus Running Redis 5 locally on the default port (6379) Optionally, you can read some additional explanation in the anatomy lesson section.","title":"2.1 Requirements"},{"location":"tutorial/quick-start/#22-define-your-api","text":"First we will define an API for Lightbus to serve. This will be an authentication API and will live in an authentication service. Create an auth_service directory and within there create the following in a bus.py file: # File: auth_service/bus.py import lightbus # Create your service's bus client. You can import this elsewere # in your service's codebase in order to access the bus bus = lightbus . create () class AuthApi ( lightbus . Api ): user_registered = lightbus . Event ( parameters = ( 'username' , 'email' )) class Meta : name = 'auth' def check_password ( self , username , password ): return username == 'admin' and password == 'secret' # Register this API with Lightbus. Lightbus will respond to # remote procedure calls for registered APIs, as well as allow you # as the developer to fire events on any registered APIs. bus . client . register_api ( AuthApi ()) You should now be able to startup Lightbus as follows: cd ./auth_service/ lightbus run Lightbus will output some logging data which will include a list of registered APIs, including your new auth API: Congratulations, you have now created an authentication service which contains a single API named auth . Leave Lightbus running and open a new terminal window for the next stage.","title":"2.2 Define your API"},{"location":"tutorial/quick-start/#23-remote-procedure-calls","text":"With Lightbus still running in another terminal window, create a new directory alongside auth_service called another_service . This will be an example service which will interact with the auth service. You should now have the following structure: ./auth_service/bus.py , created above ./another_service/ , which we will create files within now. We always define a service's bus client within a bus.py file. Therefore, create ./another_service/bus.py containing the following: # File: ./another_service/bus.py import lightbus bus = lightbus . create () Now create ./another_service/check_password.py . We will use this to experiment with making RPC calls: # File: ./another_service/check_password.py # Import our service's bus client from .bus import bus # Call the check_password() procedure on our auth API valid = bus . auth . check_password ( username = 'admin' , password = 'secret' ) # Show the result if valid : print ( 'Password valid!' ) else : print ( 'Oops, bad username or password' ) Running this script should show you the following: cd ./another_service/ python3 ./check_password.py Password valid! Looking at the other terminal window you have open you should see that Lightbus has also reported that it is handled a remote procedure call.","title":"2.3 Remote procedure calls"},{"location":"tutorial/quick-start/#24-events","text":"Events allow services to broadcast a message to any other services which care to listen. Your service can fire events on any API which has been registered. We have already created and registered our AuthApi in ./auth_service/bus.py which provides the user_registered event.","title":"2.4 Events"},{"location":"tutorial/quick-start/#firing-events","text":"Let's write a simple script to manually register users and fire events. Note we can only fire events for APIs we have registered. We have registered the auth API in auth_service , so it is within this service that this new script must reside. We could not put the script within another_service as this service does not include the AuthApi class and can therefore not register it. # ./auth_service/manually_register_user.py # Import the service's bus client from bus.py from bus import bus print ( \"New user creation\" ) new_username = input ( \"Enter a username: \" ) . strip () new_email = input ( \"Enter the user's email address: \" ) . strip () # You would normally store the new user in your database # at this point. We don't show this here for simplicity. # Let the bus know a user has been registered by firing the event bus . auth . user_registered . fire ( username = new_username , email = new_email ) print ( \"Done\" ) Run this script using: cd ./auth_service/ python3 manually_register_user.py You should be prompted for a new username & email address, at which point an event will be fired onto the bus. We will make use of this event in the next section.","title":"Firing events"},{"location":"tutorial/quick-start/#listening-for-events","text":"Perhaps another_service needs to be notified when a user is created. To achieve this we can setup an event listener by modifying ./another_service/bus.py : # File: ./another_service/bus.py import lightbus bus = lightbus . create () def handle_new_user ( event , username , email ): print ( f \"A new user was created in the authentication service:\" ) print ( f \" Username: { username } \" ) print ( f \" Email: { email } \" ) @bus . client . on_start () def bus_start ( client ): bus . auth . user_registered . listen ( handle_new_user , listener_name = \"print_on_new_registration\" ) Listening for events requires your service to sit waiting for something to happen. Sitting around and waiting for something to happen is precisely what the lightbus run console command is for. Therefore, in one terminal window startup the lightbus run command for another_service : # In the first terminal window: cd ./another_service/ lightbus run Your service is now waiting for events. In the second terminal window let's cause an event to be fired using the script we wrote in the previous section: cd ./auth_service/ python3 manually_register_user.py You should see that the event gets sent by the manually_register_user.py script within the auth_service , and received by by the lightbus run process within the another_service .","title":"Listening for events"},{"location":"tutorial/quick-start/#listener-names","text":"You may have noticed we specified the listener_name parameter in the above example, and we set it to an arbitrary value of \"print_on_new_registration\" . This is important, because Lightbus tracks which listeners have received which events. It is this listener name which is used to uniquely identify each listener. You don't need to worry about this often, just follow the rule: Ensure you specify a unique listener_name every time you setup a new listener . The listener name should be unique within the service. Two listeners with the same name will only receive a partial set of events each.","title":"Listener names"},{"location":"tutorial/quick-start/#25-next","text":"This was a simple example to get you started. The worked example considers a more realistic scenario involving multiple services.","title":"2.5 Next"},{"location":"tutorial/worked-example/","text":"If you get stuck... It is really useful to hear from people who have encountered a problem or got stuck. Hearing from you means we can improve our documentation and error messages. If you get stuck drop then please drop an email to adam@adamcharnock.com , visit the Lightbus discord server , or call me (Adam) on +442032896620. The more information you can include the better (problem description, screenshots, and code are all useful) In the following worked example we will create three services: An image resizing service An online store A stats dashboard This will involve a combination of web interfaces (using Flask ), and Lightbus APIs. The goal is to show how Lightbus can allow multiple services to interact. 3.1. Getting started \u00b6 The code created here can be found in Lightbus example ex03_worked_example , although the code will be repeated below. There is a directory for each service we will create \u2013 store/ , dashboard/ , and image/ . Before continuing ensure you have installed flask and honcho : pip3 install flask honcho A passing familiarity with Flask may be useful, but is not required. Honcho will assist us in running the various processes required for our services. We will assume you have already read and completed the installation and quick start tutorials. 3.2. Image resizing service \u00b6 The image resizing service will be a simple Lightbus API, the purpose of which is to allow our store to resize images prior to display: # File: ./image/bus.py import lightbus class ImageApi ( lightbus . Api ): class Meta : name = 'image' def resize ( self , url , width , height ): # This is a demo, so just return an animal picture of the correct size return f 'https://placeimg.com/ { width } / { height } /animals?_= { url } ' bus = lightbus . create () bus . client . register_api ( ImageApi ) There is no web interface for this service, so this is all we need. 3.3. Store service \u00b6 Our store will have both a Lightbus API and a web interface. We'll start with the API first: # File: ./store/bus.py import lightbus class StoreApi ( lightbus . Api ): page_viewed = lightbus . Event ( parameters = ( 'url' , )) class Meta : name = 'store' bus = lightbus . create () bus . client . register_api ( StoreApi ) This API has a single event called page_viewed . The store web interface will fire this event whenever a page is viewed. Our store web interface uses Flask and is a little longer: # File: ./store/web.py import lightbus from flask import Flask # Import our bus client from .bus import bus # Setup flask app = Flask ( __name__ ) # A dummy list of pets our store will sell PETS = ( 'http://store.company.com/image1.jpg' , 'http://store.company.com/image2.jpg' , 'http://store.company.com/image3.jpg' , ) @app . route ( '/' ) def home (): # A view to list all available pets html = '

Online pet store


' for pet_num , image_url in enumerate ( PETS ): # Get an image of the appropriate size resized_url = bus . image . resize ( url = image_url , width = 200 , height = 200 ) html += ( f '' f '' f ' ' ) # Fire the page view bus . store . page_viewed . fire ( url = '/' ) return html @app . route ( '/pet/' ) def pet ( pet_num ): # Show an individual pet resized_url = bus . image . resize ( url = PETS [ pet_num ], width = 200 , height = 200 ) # Fire the page view bus . store . page_viewed . fire ( url = f '/pet/ { pet_num } ' ) html = f '

Pet { pet_num }

' html = f '
' return html 3.4. Interlude: give it a go \u00b6 We're not quite done yet, but you can now startup the necessary processes and see the store. You will need to run each of these in a separate terminal window: $ ls image/ store/ # Start our image resizing service $ lightbus run --bus = image.bus # Start our store's web interface $ FLASK_APP = store/web.py flask run --port = 5001 Now open 127.0.0.1:5001 in your browser and you should see three animal pictures awaiting you. The URL for each image was fetched from the image resizing service. The flask web interface should also have some logging output akin to the following: Here you can see: image.resize was called three times, once for each image The store.page_viewed event was fired Next we will create the dashboard which will make use of the store.page_viewed event. 3.5. Dashboard service \u00b6 The dashboard service will provide internal reporting in the form of page view statistics for the online store. There dashboard will need to both receive events and provide a web interface. It will therefore need both a lightbus process and a web process. Fist we will start with the bus.py file: # File: ./dashboard/bus.py import json import lightbus bus = lightbus . client () page_views = {} def handle_page_view ( event_message , url ): # Record the page view in our page_views dictionary page_views . setdefault ( url , 0 ) page_views [ url ] += 1 # Store the dictionary on disk # (In reality you would probably use an external database) with open ( '/tmp/.dashboard.db.json' , 'w' ) as f : json . dump ( page_views , f ) @bus . client . on_start () def on_start ( ** kwargs ): # Called when lightbus starts up bus . store . page_viewed . listen ( handle_page_view ) This is a simple listener for the bus.store.page_viewed event. This event is fired by the store's web interface we created above. Note we do not define any APIs, instead we setup our event listener once the bus client has started up. Listening for this event is all the dashboard's Lightbus process will do, it will not provide any APIs. The handle_page_view() handler persists each view to the Dashboard services' local database. In a real service this would likely be a DBMS of some form (Postgres, MySQL, Redis, Mongo etc). For simplicity we just store JSON to a file. Now we'll define our dashboard's web interface: # File: ./dashboard/web.py import json from flask import Flask app = Flask ( __name__ ) @app . route ( '/' ) def home (): html = '

Dashboard

\\n ' html = '

Total store views

\\n ' with open ( '/tmp/.dashboard.db.json' , 'r' ) as f : page_views = json . load ( f ) html += '
    ' for url , total_views in page_views . items (): html += f '
  • URL { url } : { total_views }
    • ' html += '
' return html This reads the JSON data that was written by the event listener in dashboard/bus.py above, then render it to HTML. 3.7. Run it! \u00b6 You should now have the following python files: ./image/bus.py ./store/bus.py ./store/web.py ./dashboard/bus.py ./dashboard/web.py This translates into the following processes: Service Process type Purpose Image reszier Lightbus Will resize images and return a new URL Store Web Render the store UI. Use bus to resize image and fire events Store Lightbus While the store does have a bus.py , it does not have any RPCs to serve or events to listen for. We therefore do not need to run a lightbus service. Dashboard Web Render the dashboard web UI, read data from database Dashboard Lightbus Listen for page view events and store stats to database You can run each of these as follows: $ ls dashboard/ image/ store/ # Image resizer (Lightbus) $ lightbus run --bus = image.bus # Store (Web) $ FLASK_APP = store/web.py flask run --port = 5001 # Dashboard (Web + Lightbus) $ lightbus run --bus = dashboard.bus $ FLASK_APP = dashboard/web.py flask run --port = 5000 However, you may find it easier to startup these processes with the honcho tool we installed earlier. First, create a file called Procfile : # Procfile image_resizer_bus: lightbus run --bus = image.bus store_web: FLASK_APP = store/web.py flask run --port = 5001 dashboard_bus: lightbus run --bus = dashboard.bus dashboard_web: FLASK_APP = dashboard/web.py flask run --port = 5000 And now use honcho to startup all the processes together: $ ls Procfile dashboard/ image/ store/ $ honcho start If you see an error stating command not found , ensure you installed honcho as detailed above ( pip3 install honcho ). Once started, see the output for any errors. Each log line will state the process it came from. If all is well, you should see something like this: You should now be able to access the store's web interface at 127.0.0.1:5001 as you did previously. Upon viewing the page, the following will happen: The store web process will resize each image using the image resizing service The store web service will fire the store.page_viewed event. The dashboard will receive the store.page_viewed event and create the database for the first time. The logging output should reflect this: At this point you can view the dashboard at 127.0.0.1:5000 . Note that opening the dashboard before this point would have resulted in an error as the database would not have been created. The dashboard should show a simple list of URLs plus the total number of page views for each. Go back to the store and view a few pages. Now refresh the dashboard and note the new data. 3.7. Wrapping up \u00b6 While the services we have have created here are very crude, hopefully they have helped show how Lightbus can be used as a effective communications infrastructure. If you want to continue with practical learning, you should take a look at the How to or Reference sections. To learn more about the underlying concepts you should explore the Explanation section.","title":"3. Worked example"},{"location":"tutorial/worked-example/#31-getting-started","text":"The code created here can be found in Lightbus example ex03_worked_example , although the code will be repeated below. There is a directory for each service we will create \u2013 store/ , dashboard/ , and image/ . Before continuing ensure you have installed flask and honcho : pip3 install flask honcho A passing familiarity with Flask may be useful, but is not required. Honcho will assist us in running the various processes required for our services. We will assume you have already read and completed the installation and quick start tutorials.","title":"3.1. Getting started"},{"location":"tutorial/worked-example/#32-image-resizing-service","text":"The image resizing service will be a simple Lightbus API, the purpose of which is to allow our store to resize images prior to display: # File: ./image/bus.py import lightbus class ImageApi ( lightbus . Api ): class Meta : name = 'image' def resize ( self , url , width , height ): # This is a demo, so just return an animal picture of the correct size return f 'https://placeimg.com/ { width } / { height } /animals?_= { url } ' bus = lightbus . create () bus . client . register_api ( ImageApi ) There is no web interface for this service, so this is all we need.","title":"3.2. Image resizing service"},{"location":"tutorial/worked-example/#33-store-service","text":"Our store will have both a Lightbus API and a web interface. We'll start with the API first: # File: ./store/bus.py import lightbus class StoreApi ( lightbus . Api ): page_viewed = lightbus . Event ( parameters = ( 'url' , )) class Meta : name = 'store' bus = lightbus . create () bus . client . register_api ( StoreApi ) This API has a single event called page_viewed . The store web interface will fire this event whenever a page is viewed. Our store web interface uses Flask and is a little longer: # File: ./store/web.py import lightbus from flask import Flask # Import our bus client from .bus import bus # Setup flask app = Flask ( __name__ ) # A dummy list of pets our store will sell PETS = ( 'http://store.company.com/image1.jpg' , 'http://store.company.com/image2.jpg' , 'http://store.company.com/image3.jpg' , ) @app . route ( '/' ) def home (): # A view to list all available pets html = '

Online pet store


' for pet_num , image_url in enumerate ( PETS ): # Get an image of the appropriate size resized_url = bus . image . resize ( url = image_url , width = 200 , height = 200 ) html += ( f '' f '' f ' ' ) # Fire the page view bus . store . page_viewed . fire ( url = '/' ) return html @app . route ( '/pet/' ) def pet ( pet_num ): # Show an individual pet resized_url = bus . image . resize ( url = PETS [ pet_num ], width = 200 , height = 200 ) # Fire the page view bus . store . page_viewed . fire ( url = f '/pet/ { pet_num } ' ) html = f '

Pet { pet_num }

' html = f '
' return html","title":"3.3. Store service"},{"location":"tutorial/worked-example/#34-interlude-give-it-a-go","text":"We're not quite done yet, but you can now startup the necessary processes and see the store. You will need to run each of these in a separate terminal window: $ ls image/ store/ # Start our image resizing service $ lightbus run --bus = image.bus # Start our store's web interface $ FLASK_APP = store/web.py flask run --port = 5001 Now open 127.0.0.1:5001 in your browser and you should see three animal pictures awaiting you. The URL for each image was fetched from the image resizing service. The flask web interface should also have some logging output akin to the following: Here you can see: image.resize was called three times, once for each image The store.page_viewed event was fired Next we will create the dashboard which will make use of the store.page_viewed event.","title":"3.4. Interlude: give it a go"},{"location":"tutorial/worked-example/#35-dashboard-service","text":"The dashboard service will provide internal reporting in the form of page view statistics for the online store. There dashboard will need to both receive events and provide a web interface. It will therefore need both a lightbus process and a web process. Fist we will start with the bus.py file: # File: ./dashboard/bus.py import json import lightbus bus = lightbus . client () page_views = {} def handle_page_view ( event_message , url ): # Record the page view in our page_views dictionary page_views . setdefault ( url , 0 ) page_views [ url ] += 1 # Store the dictionary on disk # (In reality you would probably use an external database) with open ( '/tmp/.dashboard.db.json' , 'w' ) as f : json . dump ( page_views , f ) @bus . client . on_start () def on_start ( ** kwargs ): # Called when lightbus starts up bus . store . page_viewed . listen ( handle_page_view ) This is a simple listener for the bus.store.page_viewed event. This event is fired by the store's web interface we created above. Note we do not define any APIs, instead we setup our event listener once the bus client has started up. Listening for this event is all the dashboard's Lightbus process will do, it will not provide any APIs. The handle_page_view() handler persists each view to the Dashboard services' local database. In a real service this would likely be a DBMS of some form (Postgres, MySQL, Redis, Mongo etc). For simplicity we just store JSON to a file. Now we'll define our dashboard's web interface: # File: ./dashboard/web.py import json from flask import Flask app = Flask ( __name__ ) @app . route ( '/' ) def home (): html = '

Dashboard

\\n ' html = '

Total store views

\\n ' with open ( '/tmp/.dashboard.db.json' , 'r' ) as f : page_views = json . load ( f ) html += '
    ' for url , total_views in page_views . items (): html += f '
  • URL { url } : { total_views }
    • ' html += '
' return html This reads the JSON data that was written by the event listener in dashboard/bus.py above, then render it to HTML.","title":"3.5. Dashboard service"},{"location":"tutorial/worked-example/#37-run-it","text":"You should now have the following python files: ./image/bus.py ./store/bus.py ./store/web.py ./dashboard/bus.py ./dashboard/web.py This translates into the following processes: Service Process type Purpose Image reszier Lightbus Will resize images and return a new URL Store Web Render the store UI. Use bus to resize image and fire events Store Lightbus While the store does have a bus.py , it does not have any RPCs to serve or events to listen for. We therefore do not need to run a lightbus service. Dashboard Web Render the dashboard web UI, read data from database Dashboard Lightbus Listen for page view events and store stats to database You can run each of these as follows: $ ls dashboard/ image/ store/ # Image resizer (Lightbus) $ lightbus run --bus = image.bus # Store (Web) $ FLASK_APP = store/web.py flask run --port = 5001 # Dashboard (Web + Lightbus) $ lightbus run --bus = dashboard.bus $ FLASK_APP = dashboard/web.py flask run --port = 5000 However, you may find it easier to startup these processes with the honcho tool we installed earlier. First, create a file called Procfile : # Procfile image_resizer_bus: lightbus run --bus = image.bus store_web: FLASK_APP = store/web.py flask run --port = 5001 dashboard_bus: lightbus run --bus = dashboard.bus dashboard_web: FLASK_APP = dashboard/web.py flask run --port = 5000 And now use honcho to startup all the processes together: $ ls Procfile dashboard/ image/ store/ $ honcho start If you see an error stating command not found , ensure you installed honcho as detailed above ( pip3 install honcho ). Once started, see the output for any errors. Each log line will state the process it came from. If all is well, you should see something like this: You should now be able to access the store's web interface at 127.0.0.1:5001 as you did previously. Upon viewing the page, the following will happen: The store web process will resize each image using the image resizing service The store web service will fire the store.page_viewed event. The dashboard will receive the store.page_viewed event and create the database for the first time. The logging output should reflect this: At this point you can view the dashboard at 127.0.0.1:5000 . Note that opening the dashboard before this point would have resulted in an error as the database would not have been created. The dashboard should show a simple list of URLs plus the total number of page views for each. Go back to the store and view a few pages. Now refresh the dashboard and note the new data.","title":"3.7. Run it!"},{"location":"tutorial/worked-example/#37-wrapping-up","text":"While the services we have have created here are very crude, hopefully they have helped show how Lightbus can be used as a effective communications infrastructure. If you want to continue with practical learning, you should take a look at the How to or Reference sections. To learn more about the underlying concepts you should explore the Explanation section.","title":"3.7. Wrapping up"}]} \ No newline at end of file +{"config":{"indexing":"full","lang":["en"],"min_search_length":3,"prebuild_index":false,"separator":"[\\s\\-]+"},"docs":[{"location":"","text":"What is Lightbus? \u00b6 Lightbus is a powerful and intuitive messaging client for your backend Python services. Lightbus uses Redis 5 as its underlying transport , although support for other platforms will be added in future. Other languages can also communicate with Lightbus by interacting with Redis . How Lightbus works \u00b6 Lightbus provides you with two tools: A client with which to fire events, and make remote procedure calls (RPCs) from anywhere within your codebase. A stand-alone Lightbus worker process in which you can setup event listeners. This process will also respond to RPCs calls. For example, you could architect an e-commerce system as follows: In this example: Django serves pages using data from the database Django performs remote procedure calls to resize images. The Lightbus worker in the image resizing service performs the image resize and responds. The price monitoring service fires price_monitor.competitor_price_changed events The Lightbus worker in the online shop web service listens for price_monitor.competitor_price_changed events and updates prices in the database accordingly. See the anatomy lesson for further discussion. Designed for ease of use \u00b6 Lightbus is designed to be intuitive and familiar, and common problems are caught with clear and helpful error messages. For example, a na\u00efve authentication API: class AuthApi ( Api ): user_registered = Event ( parameters = ( 'user' , 'email' )) class Meta : name = 'auth' def check_password ( self , user , password ): return ( user == 'admin' and password == 'secret' ) The check_password procedure can be called remotely as follows: import lightbus bus = lightbus . create () is_valid = bus . auth . check_password ( user = 'admin' , password = 'secret' ) # is_valid is True You can also listen for events: # bus.py import lightbus bus = lightbus . create () # Our event handler def send_signup_email ( event_message , user , email ): send_mail ( email , subject = f 'Welcome { user } ' ) # Setup our listeners on startup @bus . client . on_start () def on_start (): bus . auth . user_registered . listen ( send_signup_email , listener_name = \"send_signup_email\" ) Where to start? \u00b6 Starting with the tutorials section will give you a practical introduction to Lightbus. Alternatively, the explanation section will give you a grounding in the high level concepts and theory . Start with whichever section suits you best. You should ultimately look through both sections for a complete understanding. In addition, the how to section gives solutions to common use cases , and the reference section provides detailed technical information regarding specific features. Questions? \u00b6 Get in touch via: Email: adam@adamcharnock.com Phone: +442032896620 (Skype, London/Lisbon timezone) Community chat GitHub: https://github.com/adamcharnock/lightbus/ If you are having a technical problem then the more information you can include the better (problem description, screenshots, and code are all useful).","title":"Home"},{"location":"#what-is-lightbus","text":"Lightbus is a powerful and intuitive messaging client for your backend Python services. Lightbus uses Redis 5 as its underlying transport , although support for other platforms will be added in future. Other languages can also communicate with Lightbus by interacting with Redis .","title":"What is Lightbus?"},{"location":"#how-lightbus-works","text":"Lightbus provides you with two tools: A client with which to fire events, and make remote procedure calls (RPCs) from anywhere within your codebase. A stand-alone Lightbus worker process in which you can setup event listeners. This process will also respond to RPCs calls. For example, you could architect an e-commerce system as follows: In this example: Django serves pages using data from the database Django performs remote procedure calls to resize images. The Lightbus worker in the image resizing service performs the image resize and responds. The price monitoring service fires price_monitor.competitor_price_changed events The Lightbus worker in the online shop web service listens for price_monitor.competitor_price_changed events and updates prices in the database accordingly. See the anatomy lesson for further discussion.","title":"How Lightbus works"},{"location":"#designed-for-ease-of-use","text":"Lightbus is designed to be intuitive and familiar, and common problems are caught with clear and helpful error messages. For example, a na\u00efve authentication API: class AuthApi ( Api ): user_registered = Event ( parameters = ( 'user' , 'email' )) class Meta : name = 'auth' def check_password ( self , user , password ): return ( user == 'admin' and password == 'secret' ) The check_password procedure can be called remotely as follows: import lightbus bus = lightbus . create () is_valid = bus . auth . check_password ( user = 'admin' , password = 'secret' ) # is_valid is True You can also listen for events: # bus.py import lightbus bus = lightbus . create () # Our event handler def send_signup_email ( event_message , user , email ): send_mail ( email , subject = f 'Welcome { user } ' ) # Setup our listeners on startup @bus . client . on_start () def on_start (): bus . auth . user_registered . listen ( send_signup_email , listener_name = \"send_signup_email\" )","title":"Designed for ease of use"},{"location":"#where-to-start","text":"Starting with the tutorials section will give you a practical introduction to Lightbus. Alternatively, the explanation section will give you a grounding in the high level concepts and theory . Start with whichever section suits you best. You should ultimately look through both sections for a complete understanding. In addition, the how to section gives solutions to common use cases , and the reference section provides detailed technical information regarding specific features.","title":"Where to start?"},{"location":"#questions","text":"Get in touch via: Email: adam@adamcharnock.com Phone: +442032896620 (Skype, London/Lisbon timezone) Community chat GitHub: https://github.com/adamcharnock/lightbus/ If you are having a technical problem then the more information you can include the better (problem description, screenshots, and code are all useful).","title":"Questions?"},{"location":"CODE_OF_CONDUCT/","text":"Code of Conduct \u00b6 Our Pledge \u00b6 We as members, contributors, and leaders pledge to make participation in our community a harassment-free experience for everyone, regardless of age, body size, visible or invisible disability, ethnicity, sex characteristics, gender identity and expression, level of experience, education, socio-economic status, nationality, personal appearance, race, religion, or sexual identity and orientation. We pledge to act and interact in ways that contribute to an open, welcoming, diverse, inclusive, and healthy community. Our Standards \u00b6 Examples of behavior that contributes to a positive environment for our community include: Demonstrating empathy and kindness toward other people Being respectful of differing opinions, viewpoints, and experiences Giving and gracefully accepting constructive feedback Accepting responsibility and apologizing to those affected by our mistakes, and learning from the experience Focusing on what is best not just for us as individuals, but for the overall community Examples of unacceptable behavior include: The use of sexualized language or imagery, and sexual attention or advances of any kind Trolling, insulting or derogatory comments, and personal or political attacks Public or private harassment Publishing others' private information, such as a physical or email address, without their explicit permission Other conduct which could reasonably be considered inappropriate in a professional setting Enforcement Responsibilities \u00b6 Community leaders are responsible for clarifying and enforcing our standards of acceptable behavior and will take appropriate and fair corrective action in response to any behavior that they deem inappropriate, threatening, offensive, or harmful. Community leaders have the right and responsibility to remove, edit, or reject comments, commits, code, wiki edits, issues, and other contributions that are not aligned to this Code of Conduct, and will communicate reasons for moderation decisions when appropriate. Scope \u00b6 This Code of Conduct applies within all community spaces, and also applies when an individual is officially representing the community in public spaces. Examples of representing our community include using an official e-mail address, posting via an official social media account, or acting as an appointed representative at an online or offline event. Enforcement \u00b6 Instances of abusive, harassing, or otherwise unacceptable behavior may be reported to the community leaders responsible for enforcement at adam@adamcharnock.com . All complaints will be reviewed and investigated promptly and fairly. All community leaders are obligated to respect the privacy and security of the reporter of any incident. Enforcement Guidelines \u00b6 Community leaders will follow these Community Impact Guidelines in determining the consequences for any action they deem in violation of this Code of Conduct: 1. Correction \u00b6 Community Impact : Use of inappropriate language or other behavior deemed unprofessional or unwelcome in the community. Consequence : A private, written warning from community leaders, providing clarity around the nature of the violation and an explanation of why the behavior was inappropriate. A public apology may be requested. 2. Warning \u00b6 Community Impact : A violation through a single incident or series of actions. Consequence : A warning with consequences for continued behavior. No interaction with the people involved, including unsolicited interaction with those enforcing the Code of Conduct, for a specified period of time. This includes avoiding interactions in community spaces as well as external channels like social media. Violating these terms may lead to a temporary or permanent ban. 3. Temporary Ban \u00b6 Community Impact : A serious violation of community standards, including sustained inappropriate behavior. Consequence : A temporary ban from any sort of interaction or public communication with the community for a specified period of time. No public or private interaction with the people involved, including unsolicited interaction with those enforcing the Code of Conduct, is allowed during this period. Violating these terms may lead to a permanent ban. 4. Permanent Ban \u00b6 Community Impact : Demonstrating a pattern of violation of community standards, including sustained inappropriate behavior, harassment of an individual, or aggression toward or disparagement of classes of individuals. Consequence : A permanent ban from any sort of public interaction within the project community. Attribution \u00b6 This Code of Conduct is adapted from the Contributor Covenant , version 2.0, available at https://www.contributor-covenant.org/version/2/0/code_of_conduct.html . Community Impact Guidelines were inspired by Mozilla's code of conduct enforcement ladder . For answers to common questions about this code of conduct, see the FAQ at https://www.contributor-covenant.org/faq . Translations are available at https://www.contributor-covenant.org/translations . For answers to common questions about this code of conduct, see https://www.contributor-covenant.org/faq","title":"Code of Conduct"},{"location":"CODE_OF_CONDUCT/#code-of-conduct","text":"","title":"Code of Conduct"},{"location":"CODE_OF_CONDUCT/#our-pledge","text":"We as members, contributors, and leaders pledge to make participation in our community a harassment-free experience for everyone, regardless of age, body size, visible or invisible disability, ethnicity, sex characteristics, gender identity and expression, level of experience, education, socio-economic status, nationality, personal appearance, race, religion, or sexual identity and orientation. We pledge to act and interact in ways that contribute to an open, welcoming, diverse, inclusive, and healthy community.","title":"Our Pledge"},{"location":"CODE_OF_CONDUCT/#our-standards","text":"Examples of behavior that contributes to a positive environment for our community include: Demonstrating empathy and kindness toward other people Being respectful of differing opinions, viewpoints, and experiences Giving and gracefully accepting constructive feedback Accepting responsibility and apologizing to those affected by our mistakes, and learning from the experience Focusing on what is best not just for us as individuals, but for the overall community Examples of unacceptable behavior include: The use of sexualized language or imagery, and sexual attention or advances of any kind Trolling, insulting or derogatory comments, and personal or political attacks Public or private harassment Publishing others' private information, such as a physical or email address, without their explicit permission Other conduct which could reasonably be considered inappropriate in a professional setting","title":"Our Standards"},{"location":"CODE_OF_CONDUCT/#enforcement-responsibilities","text":"Community leaders are responsible for clarifying and enforcing our standards of acceptable behavior and will take appropriate and fair corrective action in response to any behavior that they deem inappropriate, threatening, offensive, or harmful. Community leaders have the right and responsibility to remove, edit, or reject comments, commits, code, wiki edits, issues, and other contributions that are not aligned to this Code of Conduct, and will communicate reasons for moderation decisions when appropriate.","title":"Enforcement Responsibilities"},{"location":"CODE_OF_CONDUCT/#scope","text":"This Code of Conduct applies within all community spaces, and also applies when an individual is officially representing the community in public spaces. Examples of representing our community include using an official e-mail address, posting via an official social media account, or acting as an appointed representative at an online or offline event.","title":"Scope"},{"location":"CODE_OF_CONDUCT/#enforcement","text":"Instances of abusive, harassing, or otherwise unacceptable behavior may be reported to the community leaders responsible for enforcement at adam@adamcharnock.com . All complaints will be reviewed and investigated promptly and fairly. All community leaders are obligated to respect the privacy and security of the reporter of any incident.","title":"Enforcement"},{"location":"CODE_OF_CONDUCT/#enforcement-guidelines","text":"Community leaders will follow these Community Impact Guidelines in determining the consequences for any action they deem in violation of this Code of Conduct:","title":"Enforcement Guidelines"},{"location":"CODE_OF_CONDUCT/#1-correction","text":"Community Impact : Use of inappropriate language or other behavior deemed unprofessional or unwelcome in the community. Consequence : A private, written warning from community leaders, providing clarity around the nature of the violation and an explanation of why the behavior was inappropriate. A public apology may be requested.","title":"1. Correction"},{"location":"CODE_OF_CONDUCT/#2-warning","text":"Community Impact : A violation through a single incident or series of actions. Consequence : A warning with consequences for continued behavior. No interaction with the people involved, including unsolicited interaction with those enforcing the Code of Conduct, for a specified period of time. This includes avoiding interactions in community spaces as well as external channels like social media. Violating these terms may lead to a temporary or permanent ban.","title":"2. Warning"},{"location":"CODE_OF_CONDUCT/#3-temporary-ban","text":"Community Impact : A serious violation of community standards, including sustained inappropriate behavior. Consequence : A temporary ban from any sort of interaction or public communication with the community for a specified period of time. No public or private interaction with the people involved, including unsolicited interaction with those enforcing the Code of Conduct, is allowed during this period. Violating these terms may lead to a permanent ban.","title":"3. Temporary Ban"},{"location":"CODE_OF_CONDUCT/#4-permanent-ban","text":"Community Impact : Demonstrating a pattern of violation of community standards, including sustained inappropriate behavior, harassment of an individual, or aggression toward or disparagement of classes of individuals. Consequence : A permanent ban from any sort of public interaction within the project community.","title":"4. Permanent Ban"},{"location":"CODE_OF_CONDUCT/#attribution","text":"This Code of Conduct is adapted from the Contributor Covenant , version 2.0, available at https://www.contributor-covenant.org/version/2/0/code_of_conduct.html . Community Impact Guidelines were inspired by Mozilla's code of conduct enforcement ladder . For answers to common questions about this code of conduct, see the FAQ at https://www.contributor-covenant.org/faq . Translations are available at https://www.contributor-covenant.org/translations . For answers to common questions about this code of conduct, see https://www.contributor-covenant.org/faq","title":"Attribution"},{"location":"explanation/","text":"Explanation overview \u00b6 This section discusses the theoretical and conceptual aspects of Lightbus . This is in contrast to the more practical tutorial and how to sections.","title":"Overview"},{"location":"explanation/#explanation-overview","text":"This section discusses the theoretical and conceptual aspects of Lightbus . This is in contrast to the more practical tutorial and how to sections.","title":"Explanation overview"},{"location":"explanation/anatomy-lesson/","text":"Anatomy lesson \u00b6 Lightbus provides you with two tools: A client with which to fire events , listen for events and make remote procedure calls (RPCs). A stand-alone Lightbus worker process in which you can setup event listeners. This process will also respond to RPCs calls. The client \u00b6 The client allows you to interact with the bus from within your Python codebase. For example: # The bus is created in your bus.py file import lightbus bus = lightbus . create () # Elsewhere in codebase from my_project.bus import bus # Perform a remote procedure call is_valid = bus . auth . check_password ( user = \"admin\" , password = \"secret\" ) # Fire an event bus . auth . user_registered ( user = \"sally\" , email = \"sally@example.com\" ) You can use this client anywhere you need to, such as: Within your Django/Flask views Within scheduled jobs Within Lightbus event & RPC handlers (see below) Important Each service should create its bus client within the service's bus.py file (also known as the servies bus module ). Other code within the service should import the bus client from the bus module as needed. See how to access your bus client . The Lightbus worker process ( lightbus run ) \u00b6 The Lightbus worker is a long running process started using the lightbus run console command. This process serves two purposes: Listens for events and fires any executes any listeners you have created. Respond to incoming remote procedure calls for the service's registered APIs. This process imports your bus module (see the module loading configuration reference) in order to bootstrap itself. Your bus module should therefore: Instantiate the bus client in a module variable named bus Register any API definitions for your service Setup listeners for any events you wish to listen for For example, let's use the auth.create_user() remote procedure call to create a new user every time a customers.new_customer event appears on the bus: # bus.py import lightbus bus = lightbus . create () def create_user_for_customer ( event_message , customer_name , email ): # We can do something locally, or call an # RPC, or both. Here we call an RPC. bus . auth . create_user ( name = customer_name , email = email ) # Setup our listeners on startup @bus . client . on_start () def on_start (): # Create a new user for each new customer bus . customers . new_customer . listen ( create_user_for_customer ) You start this process using the command: lightbus run Lightbus will import the bus module (your bus.py file) and wait for incoming events and remote procedure calls. A service will only need a Lightbus process if it wishes to listen for events or provide any RPCs which can be called.","title":"Anatomy lesson"},{"location":"explanation/anatomy-lesson/#anatomy-lesson","text":"Lightbus provides you with two tools: A client with which to fire events , listen for events and make remote procedure calls (RPCs). A stand-alone Lightbus worker process in which you can setup event listeners. This process will also respond to RPCs calls.","title":"Anatomy lesson"},{"location":"explanation/anatomy-lesson/#the-client","text":"The client allows you to interact with the bus from within your Python codebase. For example: # The bus is created in your bus.py file import lightbus bus = lightbus . create () # Elsewhere in codebase from my_project.bus import bus # Perform a remote procedure call is_valid = bus . auth . check_password ( user = \"admin\" , password = \"secret\" ) # Fire an event bus . auth . user_registered ( user = \"sally\" , email = \"sally@example.com\" ) You can use this client anywhere you need to, such as: Within your Django/Flask views Within scheduled jobs Within Lightbus event & RPC handlers (see below) Important Each service should create its bus client within the service's bus.py file (also known as the servies bus module ). Other code within the service should import the bus client from the bus module as needed. See how to access your bus client .","title":"The client"},{"location":"explanation/anatomy-lesson/#the-lightbus-worker-process-lightbus-run","text":"The Lightbus worker is a long running process started using the lightbus run console command. This process serves two purposes: Listens for events and fires any executes any listeners you have created. Respond to incoming remote procedure calls for the service's registered APIs. This process imports your bus module (see the module loading configuration reference) in order to bootstrap itself. Your bus module should therefore: Instantiate the bus client in a module variable named bus Register any API definitions for your service Setup listeners for any events you wish to listen for For example, let's use the auth.create_user() remote procedure call to create a new user every time a customers.new_customer event appears on the bus: # bus.py import lightbus bus = lightbus . create () def create_user_for_customer ( event_message , customer_name , email ): # We can do something locally, or call an # RPC, or both. Here we call an RPC. bus . auth . create_user ( name = customer_name , email = email ) # Setup our listeners on startup @bus . client . on_start () def on_start (): # Create a new user for each new customer bus . customers . new_customer . listen ( create_user_for_customer ) You start this process using the command: lightbus run Lightbus will import the bus module (your bus.py file) and wait for incoming events and remote procedure calls. A service will only need a Lightbus process if it wishes to listen for events or provide any RPCs which can be called.","title":"The Lightbus worker process (lightbus run)"},{"location":"explanation/apis/","text":"APIs \u00b6 When we refer to an API , we are referring to an Api class definition. All functionality on the bus is defined using APIs. For example, consider an API for support tickets within a company's help desk: class TicketApi ( Api ): ticket_created = Event ( parameters = ( 'id' , 'sender' , 'subject' , 'body' )) class Meta : name = 'help_desk.ticket' def get ( self , id ): return get_ticket_from_db ( pk = id ) This API defines an event, a procedure, and the name used to address the API on the bus. The help desk service could define multiple additional APIs as needed (perhaps for listing help desk staff or retrieving reports). API registration & authoritative/non-authoritative APIs \u00b6 An API can be registered with your service's bus client as follows: import lightbus from my_apis import HelpDeskApi bus = lightbus . create () # Register the API with your service's client bus . client . register_api ( HelpDeskApi ()) Registering an API will: Allow you to fire events on the API using the service's client Cause the lightbus worker for this service (i.e. lightbus run ) to respond to remote procedure calls on the registered API We say that a service which registers an API is authoritative for that API. Services which do not register a given API are non-authoritative for the API. Both authoritative and non-authoritative services can listen for events on any API and call remote procedures on any API. For example, a separate online store service could not fire the help_desk.ticket_created event on the API we defined above. Nor would you reasonably expect the online store to services remote procedure calls for help_desk.ticket_created() . Why? \u00b6 Preventing the online store service from responding to remote procedure calls for the help desk service makes sense. There is no reason the online store should have any awareness of the help desk, so you would not expect it to respond to remote procedure calls regarding tickets. Therefore, the logic for allowing only authoritative services to respond to remote procedure calls is hopefully compelling. The case for limiting event firing to authoritative services is one of architecture, maintainability, and consistency: Allowing any event to be fired by any service within your organisation could quickly lead to spiraling complexity. The authoritative service will always have sufficient information to guarantee basic validity of an emitted message (for example, the event exists, required parameters are present etc). As a result errors can be caught earlier, rather than allowing them to propagate onto the bus and potentially impact distant services. We welcome discussion on this topic, open a GitHub issue if you would like to discuss this further.","title":"APIs"},{"location":"explanation/apis/#apis","text":"When we refer to an API , we are referring to an Api class definition. All functionality on the bus is defined using APIs. For example, consider an API for support tickets within a company's help desk: class TicketApi ( Api ): ticket_created = Event ( parameters = ( 'id' , 'sender' , 'subject' , 'body' )) class Meta : name = 'help_desk.ticket' def get ( self , id ): return get_ticket_from_db ( pk = id ) This API defines an event, a procedure, and the name used to address the API on the bus. The help desk service could define multiple additional APIs as needed (perhaps for listing help desk staff or retrieving reports).","title":"APIs"},{"location":"explanation/apis/#api-registration-authoritativenon-authoritative-apis","text":"An API can be registered with your service's bus client as follows: import lightbus from my_apis import HelpDeskApi bus = lightbus . create () # Register the API with your service's client bus . client . register_api ( HelpDeskApi ()) Registering an API will: Allow you to fire events on the API using the service's client Cause the lightbus worker for this service (i.e. lightbus run ) to respond to remote procedure calls on the registered API We say that a service which registers an API is authoritative for that API. Services which do not register a given API are non-authoritative for the API. Both authoritative and non-authoritative services can listen for events on any API and call remote procedures on any API. For example, a separate online store service could not fire the help_desk.ticket_created event on the API we defined above. Nor would you reasonably expect the online store to services remote procedure calls for help_desk.ticket_created() .","title":"API registration & authoritative/non-authoritative APIs"},{"location":"explanation/apis/#why","text":"Preventing the online store service from responding to remote procedure calls for the help desk service makes sense. There is no reason the online store should have any awareness of the help desk, so you would not expect it to respond to remote procedure calls regarding tickets. Therefore, the logic for allowing only authoritative services to respond to remote procedure calls is hopefully compelling. The case for limiting event firing to authoritative services is one of architecture, maintainability, and consistency: Allowing any event to be fired by any service within your organisation could quickly lead to spiraling complexity. The authoritative service will always have sufficient information to guarantee basic validity of an emitted message (for example, the event exists, required parameters are present etc). As a result errors can be caught earlier, rather than allowing them to propagate onto the bus and potentially impact distant services. We welcome discussion on this topic, open a GitHub issue if you would like to discuss this further.","title":"Why?"},{"location":"explanation/architecture-tips/","text":"Architecture tips \u00b6 These tips draw some core concepts from the Domain Driven Design and CQRS patterns. We only scape the surface of these patterns here, but the ideas covered below are the ones the Lightbus maintainers found most useful when using Lightbus. There is a lot more to be said on each point. The intent here is to provide a digestible starting point. Use common data structures \u00b6 Create structures which represent the data you wish to transmit. Both NamedTuples and dataclasses work well for this purpose. For example: from typing import NamedTuple class Customer ( NamedTuple ): name : str email : str age : int You can then take advantage of Lightbus' schema checking and data casting: import lightbus class CustomerApi ( lightbus . Api ): class Meta : name = \"customer\" def create ( customer : Customer ): # Lightbus will ensure the incoming data conforms to # Customer, and will cast the data to be a Customer object ... You would call the customer.create() API as follows: import lightbus bus = lightbus . create () bus . customer . create ( customer = Customer ( name = \"Joe\" , email = \"joe@gmail.com\" , age = 54 ) ) This provides a standard way of communicating shared concepts across your services. Note You can still send and receive events even if you do not have the data strucuture available. To send you can simply use a dictionary: bus . customer . create ( customer = { \"name\" : \"Joe\" , \"email\" : \"joe@gmail.com\" , \"age\" : 54 } ) Likewise, an RPC or event listener without type hints will simply receive a dictionary. See also use a monorepository . Decide on boundaries \u00b6 Your services will likely use an assortment of entities. It is common for these to map to database tables or [ORM] classes. For example, you may have Customer , Address , Order , and OrderLine entities (and probably many more). You will need to pass these between your services, but how should you structure them in order to do so? Do pass them individually, or combine them into a hierarchy? Do you sometimes omit some fields for brevity, or not? Take the following events for example: customer.signup(customer) \u2013 Does the customer object include the address(es)? order.shipped(order) \u2013 Does the order object contain each line item? order.line_item_deleted(order, line_item) \u2013 Do we need to pass line_item here, or should the line items be included within order ? This is a simple example with only four entities. In a real-world system it is very easy to end up passing around data in forms which become increasingly bloated, inconsistent, and complex. You can mitigate this by grouping your entities together in a way that makes logical sense for your situation . These are your aggregates . Decide on these aggregates in advance, and stick to it. For example, we may group the Customer , Address , Order , and OrderLine entities as follows: A Customer aggregate. Each Customer contains an address field, which is an Address entity. An Order aggregate. Each Order contains a line_items field, which is a list of OrderLine entities. The Customer and Order aggregates also will likely contain other fields not listed above (such as name, email, or quantity). You should only ever pass around the top level aggregates. Additionally: Identify aggregates with UUIDs Do not enforce foreign keys between aggregates. Your application code will need to deal with inconsistencies gracefully (likely in it's user interface). It is likely still a good idea to use sequential integer primary keys in your RDBMS Do not share database-level sequential integer primary keys with other servces Writes are special \u00b6 Writes are inherently different to reads in a distributed system. Reading data is generally straightforward, whereas data modifications need to be propagated to all concerned services. Reading data looks broadly like this: Read data from storage (disk, ORM, memory etc) Perhaps transform the data Display the data, or send it somewhere An initial attempt at writing data may look like this: Write the data locally Broadcast the change (Lightbus event or RPC) Other services receive & apply the change The downside of this approach is that you have two write paths (writing locally and broadcasting the change). If either of these fails then your services will have an inconsistent view of the world. A more CQRS-based approach to writing data looks like this: Broadcast the change (Lightbus event) Services (including the broadcasting service) receive & apply the change The benefit of this approach is that either the change happens or it does not. Additionally all applications share the same write path, which reduces overall complexity. A downside is that changes can take a moment to be applied, so changes made by a user may not be immediately visible. Use a monorepository \u00b6 A monorepository is a repository which stores all your services, rather than having one repository per service. There are pros and cons to this approach and you should perform your own research. The relevant benefits to Lightbus (and distributed architectures in general) are: Atomic commits across multiple services \u2013 useful when a change to one service affects another. Easier sharing of data structures \u2013 this provides a shared language across all of your services. These data structures can be used as type hints on your events and RPCs, which Lightbus will automatically cast data into. Sharing of APIs \u2013 in some cases you may wish for multiple services fire events on the same API. In this case each of these services will need to have access to the API class. (See API registration & authoritative/non-authoritative APIs ) Sharing of global bus configuration \u2013 similar to sharing data structures, your global bus configuration YAML file can likewise be available to all services.","title":"Architecture tips"},{"location":"explanation/architecture-tips/#architecture-tips","text":"These tips draw some core concepts from the Domain Driven Design and CQRS patterns. We only scape the surface of these patterns here, but the ideas covered below are the ones the Lightbus maintainers found most useful when using Lightbus. There is a lot more to be said on each point. The intent here is to provide a digestible starting point.","title":"Architecture tips"},{"location":"explanation/architecture-tips/#use-common-data-structures","text":"Create structures which represent the data you wish to transmit. Both NamedTuples and dataclasses work well for this purpose. For example: from typing import NamedTuple class Customer ( NamedTuple ): name : str email : str age : int You can then take advantage of Lightbus' schema checking and data casting: import lightbus class CustomerApi ( lightbus . Api ): class Meta : name = \"customer\" def create ( customer : Customer ): # Lightbus will ensure the incoming data conforms to # Customer, and will cast the data to be a Customer object ... You would call the customer.create() API as follows: import lightbus bus = lightbus . create () bus . customer . create ( customer = Customer ( name = \"Joe\" , email = \"joe@gmail.com\" , age = 54 ) ) This provides a standard way of communicating shared concepts across your services. Note You can still send and receive events even if you do not have the data strucuture available. To send you can simply use a dictionary: bus . customer . create ( customer = { \"name\" : \"Joe\" , \"email\" : \"joe@gmail.com\" , \"age\" : 54 } ) Likewise, an RPC or event listener without type hints will simply receive a dictionary. See also use a monorepository .","title":"Use common data structures"},{"location":"explanation/architecture-tips/#decide-on-boundaries","text":"Your services will likely use an assortment of entities. It is common for these to map to database tables or [ORM] classes. For example, you may have Customer , Address , Order , and OrderLine entities (and probably many more). You will need to pass these between your services, but how should you structure them in order to do so? Do pass them individually, or combine them into a hierarchy? Do you sometimes omit some fields for brevity, or not? Take the following events for example: customer.signup(customer) \u2013 Does the customer object include the address(es)? order.shipped(order) \u2013 Does the order object contain each line item? order.line_item_deleted(order, line_item) \u2013 Do we need to pass line_item here, or should the line items be included within order ? This is a simple example with only four entities. In a real-world system it is very easy to end up passing around data in forms which become increasingly bloated, inconsistent, and complex. You can mitigate this by grouping your entities together in a way that makes logical sense for your situation . These are your aggregates . Decide on these aggregates in advance, and stick to it. For example, we may group the Customer , Address , Order , and OrderLine entities as follows: A Customer aggregate. Each Customer contains an address field, which is an Address entity. An Order aggregate. Each Order contains a line_items field, which is a list of OrderLine entities. The Customer and Order aggregates also will likely contain other fields not listed above (such as name, email, or quantity). You should only ever pass around the top level aggregates. Additionally: Identify aggregates with UUIDs Do not enforce foreign keys between aggregates. Your application code will need to deal with inconsistencies gracefully (likely in it's user interface). It is likely still a good idea to use sequential integer primary keys in your RDBMS Do not share database-level sequential integer primary keys with other servces","title":"Decide on boundaries"},{"location":"explanation/architecture-tips/#writes-are-special","text":"Writes are inherently different to reads in a distributed system. Reading data is generally straightforward, whereas data modifications need to be propagated to all concerned services. Reading data looks broadly like this: Read data from storage (disk, ORM, memory etc) Perhaps transform the data Display the data, or send it somewhere An initial attempt at writing data may look like this: Write the data locally Broadcast the change (Lightbus event or RPC) Other services receive & apply the change The downside of this approach is that you have two write paths (writing locally and broadcasting the change). If either of these fails then your services will have an inconsistent view of the world. A more CQRS-based approach to writing data looks like this: Broadcast the change (Lightbus event) Services (including the broadcasting service) receive & apply the change The benefit of this approach is that either the change happens or it does not. Additionally all applications share the same write path, which reduces overall complexity. A downside is that changes can take a moment to be applied, so changes made by a user may not be immediately visible.","title":"Writes are special"},{"location":"explanation/architecture-tips/#use-a-monorepository","text":"A monorepository is a repository which stores all your services, rather than having one repository per service. There are pros and cons to this approach and you should perform your own research. The relevant benefits to Lightbus (and distributed architectures in general) are: Atomic commits across multiple services \u2013 useful when a change to one service affects another. Easier sharing of data structures \u2013 this provides a shared language across all of your services. These data structures can be used as type hints on your events and RPCs, which Lightbus will automatically cast data into. Sharing of APIs \u2013 in some cases you may wish for multiple services fire events on the same API. In this case each of these services will need to have access to the API class. (See API registration & authoritative/non-authoritative APIs ) Sharing of global bus configuration \u2013 similar to sharing data structures, your global bus configuration YAML file can likewise be available to all services.","title":"Use a monorepository"},{"location":"explanation/bus/","text":"The bus \u00b6 The bus is the communications channel which links all your services together. Currently this is Redis. You use lightbus.create() in your bus.py file to access this bus: # bus.py import lightbus bus = lightbus . create () This creates a high-level client through which you can perform remote procedure calls and fire events . About buses \u00b6 In computing, a bus is a shared communication medium. A bus allows any software/hardware connected to that medium to communicate, as long as common rules are obeyed. In this sense a bus is very similar to a conversation between a group of people. In electronics the communication medium can be a simple copper cable. In software the communication medium is itself defined by software. Lightbus uses Redis as its communication medium , although support for other mediums may be added in future.","title":"The bus"},{"location":"explanation/bus/#the-bus","text":"The bus is the communications channel which links all your services together. Currently this is Redis. You use lightbus.create() in your bus.py file to access this bus: # bus.py import lightbus bus = lightbus . create () This creates a high-level client through which you can perform remote procedure calls and fire events .","title":"The bus"},{"location":"explanation/bus/#about-buses","text":"In computing, a bus is a shared communication medium. A bus allows any software/hardware connected to that medium to communicate, as long as common rules are obeyed. In this sense a bus is very similar to a conversation between a group of people. In electronics the communication medium can be a simple copper cable. In software the communication medium is itself defined by software. Lightbus uses Redis as its communication medium , although support for other mediums may be added in future.","title":"About buses"},{"location":"explanation/configuration/","text":"Configuration \u00b6 As discussed in the configuration reference , Lightbus has three stages of configuration: Module loading Service-level configuration Global bus configuration See the configuration reference for details on how this works in practice. Here we will discuss the reasoning behind this system. 1. Module loading \u00b6 Lightbus needs to know how to bootstrap itself. It needs to know where to start. This module loading step is how we provide Lightbus with this information, via the LIGHTBUS_MODULE environment variable. The module loading was inspired by Django 's DJANGO_SETTINGS_MODULE environment variable. LIGHTBUS_MODULE has a sensible default of bus in the same way that DJANGO_SETTINGS_MODULE has a sensible default of settings . This default will work in many scenarios, but may also need to be customised depending on one's project structure. 2. Service-level configuration \u00b6 Some configuration must by its nature be specific to a service, and not global to the entire bus. These options are broadly: Configuration which distinguishes this service from any other service ( service_name / process_name ) Configuration related to the specific deployment of this service ( features ) A pointer to the global bus configuration 3. Global bus configuration \u00b6 The global bus configuration provides the bulk of the Lightbus options. This configuration should be consistent across all Lightbus clients. But what is the reasoning here? The reasoning is that the bus is a globally shared resource, therefore everything that uses the bus is going to need a follow a common configuration in order to function. For example, consider the case where one bus client is configured to connect to redis server A for the customers API, and another bus client is configured to connect to redis server B for the same API. What will happen? The result will be that you will have effectively created a network partition. Each bus client will operate in total ignorance of each other. Events could be lost or ignored, and RPCs may never be processed. Some configuration must therefore be common to all clients, and that is what the global configuration provides. Configuration loading over HTTP(S) \u00b6 To this end, Lightbus supports loading configuration over HTTP(S). The intention is not for you to host you Lightbus configuration on the public internet! . Rather, and if you wish, you may find is useful to host your configuration on an internal-only endpoint. Alternatively, you may decide to ensure your build/deploy process distributes a copy of the global configuration file to every service. The discussion on monorepositories in Architecture Tips is relevant here.","title":"Configuration"},{"location":"explanation/configuration/#configuration","text":"As discussed in the configuration reference , Lightbus has three stages of configuration: Module loading Service-level configuration Global bus configuration See the configuration reference for details on how this works in practice. Here we will discuss the reasoning behind this system.","title":"Configuration"},{"location":"explanation/configuration/#1-module-loading","text":"Lightbus needs to know how to bootstrap itself. It needs to know where to start. This module loading step is how we provide Lightbus with this information, via the LIGHTBUS_MODULE environment variable. The module loading was inspired by Django 's DJANGO_SETTINGS_MODULE environment variable. LIGHTBUS_MODULE has a sensible default of bus in the same way that DJANGO_SETTINGS_MODULE has a sensible default of settings . This default will work in many scenarios, but may also need to be customised depending on one's project structure.","title":"1. Module loading"},{"location":"explanation/configuration/#2-service-level-configuration","text":"Some configuration must by its nature be specific to a service, and not global to the entire bus. These options are broadly: Configuration which distinguishes this service from any other service ( service_name / process_name ) Configuration related to the specific deployment of this service ( features ) A pointer to the global bus configuration","title":"2. Service-level configuration"},{"location":"explanation/configuration/#3-global-bus-configuration","text":"The global bus configuration provides the bulk of the Lightbus options. This configuration should be consistent across all Lightbus clients. But what is the reasoning here? The reasoning is that the bus is a globally shared resource, therefore everything that uses the bus is going to need a follow a common configuration in order to function. For example, consider the case where one bus client is configured to connect to redis server A for the customers API, and another bus client is configured to connect to redis server B for the same API. What will happen? The result will be that you will have effectively created a network partition. Each bus client will operate in total ignorance of each other. Events could be lost or ignored, and RPCs may never be processed. Some configuration must therefore be common to all clients, and that is what the global configuration provides.","title":"3. Global bus configuration"},{"location":"explanation/configuration/#configuration-loading-over-https","text":"To this end, Lightbus supports loading configuration over HTTP(S). The intention is not for you to host you Lightbus configuration on the public internet! . Rather, and if you wish, you may find is useful to host your configuration on an internal-only endpoint. Alternatively, you may decide to ensure your build/deploy process distributes a copy of the global configuration file to every service. The discussion on monorepositories in Architecture Tips is relevant here.","title":"Configuration loading over HTTP(S)"},{"location":"explanation/events/","text":"Events \u00b6 Firing an event will place the event onto the bus and return immediately. No information is provided as to whether the event was processed, or indeed of it was received by any other service at all. No return value is provided when firing an event. This is useful when: You wish to allow non- authoritative services to receive information without needing to concern yourself with their implementation You wish the authoritative service to perform a known task in the background The quickstart provides an example of the latter case. At-least-once semantics \u00b6 Delivering a message exactly once is Very Difficult. Delivering a message at-most-once, or at-least-once is much more practical. Lightbus therefore provides at-least-once delivery for events . As a result you can assume your event listeners will always receive an event, but sometimes a listener may be called multiple times for the same event. You can handle this by ensuring your event listeners are idempotent. That is, implement your event listeners in such a way that it doesn't matter how many times they are executed. See how to write idempotent event handlers . Service names & listener names \u00b6 An event will be delivered once to each consumer group . A consumer group is identified by a name in the form: # Consumer group naming format {service_name}-{listener_name} The service name is specified in your service-level configuration . The listener name is setup when you create your event listener (see below). For example, this bus module sets up two listeners. Each listener is given a listener_name , thereby ensuring each listener receives a copy of every competitor_prices.changed event. # Example of setting service and listener names from my_handlers import send_price_alerts , update_db bus = lightbus . create ( service_name = 'price-monitor' , ) # Consumer group name will be: price-monitor-send-price-alerts bus . competitor_prices . changed . listen ( send_price_alerts , listener_name = \"send_price_alerts\" , ) # Consumer group name will be: price-monitor-update-db bus . competitor_prices . changed . listen ( update_db , listener_name = \"update_db\" , ) Process names \u00b6 Your service-level configuration also specifies a process name . This is less critical than the service name & listener name pairing described above, but still serves an important function. The process name is used to track which process within a consumer group is dealing with a given event. This is primarily useful when a service runs multiple Lightbus processes, normally as a result of scaling or reliability requirements. The purpose of a process name is twofold: Each process has a per-determined length of time to handle and acknowledge a given event. If this time is exceeded then failure will be assumed, and the event may be picked up by another process. When a Lightbus process starts up it will check for any outstanding events reserved for its process name. In which case it will process these messages first. This can happen if the process was killed prior to acknowledging messages it was processing. Providing there is no timeout, an event will only be delivered to one process per listener within a service . The default process name is a random 4 character string. If left unchanged, then clause 2 (above) will never be triggered (as a process name will not persist between process restarts). Any outstanding messages will always have to wait for the timeout period to expire, at which point the will be picked up by another process. Considerations \u00b6 Events are more complex, you may need maintain state as events are received. The source-of-truth regarding stored state may no longer be clear. Enforcing consistency can become difficult. Events are more robust. Your service will be able to fire events as long as the bus client can connect. Likewise, you service can listen for events until the cows come home. Incoming events may be delayed by problems in other services, but each service should be isolated from those problems. Concepts such as Domain Driven Design and Event Sourcing can help to tackle some of these problems. Best practices \u00b6 You may find some of these best practices & suggestions useful. Just remember that there can be exceptions to every rule. Note See architecture tips for further details. Event naming \u00b6 Name events using the past tense. Use page_viewed , not page_view . Use order_created , not create_order . This can apply to events which are commands as well: Use email_report_requested , not email_report . Where relevant, consider using domain-based naming rather than technical names. For example, use order_placed , not order_created . Use parcel_delivered , not parcel_updated . Parameter values \u00b6 Parameter values should have consistent meaning over time. Use complete datetimes, not 'tomorrow' or '6 days ago'. Decide on the boundaries between your relations. See Decide on Boundaries with the architecture tips section.","title":"Events"},{"location":"explanation/events/#events","text":"Firing an event will place the event onto the bus and return immediately. No information is provided as to whether the event was processed, or indeed of it was received by any other service at all. No return value is provided when firing an event. This is useful when: You wish to allow non- authoritative services to receive information without needing to concern yourself with their implementation You wish the authoritative service to perform a known task in the background The quickstart provides an example of the latter case.","title":"Events"},{"location":"explanation/events/#at-least-once-semantics","text":"Delivering a message exactly once is Very Difficult. Delivering a message at-most-once, or at-least-once is much more practical. Lightbus therefore provides at-least-once delivery for events . As a result you can assume your event listeners will always receive an event, but sometimes a listener may be called multiple times for the same event. You can handle this by ensuring your event listeners are idempotent. That is, implement your event listeners in such a way that it doesn't matter how many times they are executed. See how to write idempotent event handlers .","title":"At-least-once semantics"},{"location":"explanation/events/#service-names-listener-names","text":"An event will be delivered once to each consumer group . A consumer group is identified by a name in the form: # Consumer group naming format {service_name}-{listener_name} The service name is specified in your service-level configuration . The listener name is setup when you create your event listener (see below). For example, this bus module sets up two listeners. Each listener is given a listener_name , thereby ensuring each listener receives a copy of every competitor_prices.changed event. # Example of setting service and listener names from my_handlers import send_price_alerts , update_db bus = lightbus . create ( service_name = 'price-monitor' , ) # Consumer group name will be: price-monitor-send-price-alerts bus . competitor_prices . changed . listen ( send_price_alerts , listener_name = \"send_price_alerts\" , ) # Consumer group name will be: price-monitor-update-db bus . competitor_prices . changed . listen ( update_db , listener_name = \"update_db\" , )","title":"Service names & listener names"},{"location":"explanation/events/#process-names","text":"Your service-level configuration also specifies a process name . This is less critical than the service name & listener name pairing described above, but still serves an important function. The process name is used to track which process within a consumer group is dealing with a given event. This is primarily useful when a service runs multiple Lightbus processes, normally as a result of scaling or reliability requirements. The purpose of a process name is twofold: Each process has a per-determined length of time to handle and acknowledge a given event. If this time is exceeded then failure will be assumed, and the event may be picked up by another process. When a Lightbus process starts up it will check for any outstanding events reserved for its process name. In which case it will process these messages first. This can happen if the process was killed prior to acknowledging messages it was processing. Providing there is no timeout, an event will only be delivered to one process per listener within a service . The default process name is a random 4 character string. If left unchanged, then clause 2 (above) will never be triggered (as a process name will not persist between process restarts). Any outstanding messages will always have to wait for the timeout period to expire, at which point the will be picked up by another process.","title":"Process names"},{"location":"explanation/events/#considerations","text":"Events are more complex, you may need maintain state as events are received. The source-of-truth regarding stored state may no longer be clear. Enforcing consistency can become difficult. Events are more robust. Your service will be able to fire events as long as the bus client can connect. Likewise, you service can listen for events until the cows come home. Incoming events may be delayed by problems in other services, but each service should be isolated from those problems. Concepts such as Domain Driven Design and Event Sourcing can help to tackle some of these problems.","title":"Considerations"},{"location":"explanation/events/#best-practices","text":"You may find some of these best practices & suggestions useful. Just remember that there can be exceptions to every rule. Note See architecture tips for further details.","title":"Best practices"},{"location":"explanation/events/#event-naming","text":"Name events using the past tense. Use page_viewed , not page_view . Use order_created , not create_order . This can apply to events which are commands as well: Use email_report_requested , not email_report . Where relevant, consider using domain-based naming rather than technical names. For example, use order_placed , not order_created . Use parcel_delivered , not parcel_updated .","title":"Event naming"},{"location":"explanation/events/#parameter-values","text":"Parameter values should have consistent meaning over time. Use complete datetimes, not 'tomorrow' or '6 days ago'. Decide on the boundaries between your relations. See Decide on Boundaries with the architecture tips section.","title":"Parameter values"},{"location":"explanation/internal-architecture/","text":"Internal Architecture \u00b6 Lightbus' internal workings are composed of: The user-facing API . This is provided by the BusClient class, which then makes use of the EventClient and RpcResultClient classes. This is a friendly API that issues helpful errors where appropriate. This also orchestrates system startup and shutdown. An internal message queuing system . This includes four separate internal message queues plus the EventDock & RpcResultDock classes. The message queues provide the internal communication medium between the user-facing API and the Lightbus backend. The EventDock & RpcResultDock classes convert these messages into a simplified API for implementation by the transports. The EventDock contains the EventTransport , and the RpcResultDock contains both the RpcTransport and RpcResultTransport . The Event & RPC transports implement Lightbus functionality for a specific backend (e.g. Redis). The main transports shipped with Lightbus are the RedisEventTransport , RedisRpcTransport , and RedisResultTransport . Diagram \u00b6","title":"Internal Architecture"},{"location":"explanation/internal-architecture/#internal-architecture","text":"Lightbus' internal workings are composed of: The user-facing API . This is provided by the BusClient class, which then makes use of the EventClient and RpcResultClient classes. This is a friendly API that issues helpful errors where appropriate. This also orchestrates system startup and shutdown. An internal message queuing system . This includes four separate internal message queues plus the EventDock & RpcResultDock classes. The message queues provide the internal communication medium between the user-facing API and the Lightbus backend. The EventDock & RpcResultDock classes convert these messages into a simplified API for implementation by the transports. The EventDock contains the EventTransport , and the RpcResultDock contains both the RpcTransport and RpcResultTransport . The Event & RPC transports implement Lightbus functionality for a specific backend (e.g. Redis). The main transports shipped with Lightbus are the RedisEventTransport , RedisRpcTransport , and RedisResultTransport .","title":"Internal Architecture"},{"location":"explanation/internal-architecture/#diagram","text":"","title":"Diagram"},{"location":"explanation/lightbus-vs-celery/","text":"Lightbus vs Celery \u00b6 Lightbus was conceived as a result of using Celery to communicate between multiple Python services. Differences in principle \u00b6 Celery is a task queue: A task queue is tightly coupled. The dispatching code must know what needs to be done A task queue typically doesn't return results Lightbus is a bus: A bus provides loose coupling. The dispatching code says what did happen, not what should happen (events) A bus provides bi-directional communication (remote procedure calls) Differences in practice \u00b6 A number of pain points were identified with Celery that Lightbus aims to address. In particular: Single vs multi- service \u2013 Celery is targeted as being a task queue for a service, rather than a means for multiple services to interact. Conceptual overlap \u2013 The mapping between concepts in Celery and the underlying broker (AMQP at the time) is both unclear and overlapping. Lightbus provides a limited set of well defined concepts to avoid this confusion. Non-sane defaults \u2013 Some Celery settings have non-sane defaults, making setup somewhat perilous at times. Lightbus provides sane defaults for most circumstances, and documentation specifically geared to certain use cases ( metrics , event sourcing ) Tight coupling (as discussed above) \u2013 Celery tasks define the action to take, not what happened. Lightbus uses events, which describe happened, and listening services decide the action to take. General feeling \u2013 Celery feels large and opaque, debugging issues was challenging. Lightbus aims to feel lightweight, with clear logging and debugging tools.","title":"Lightbus vs Celery"},{"location":"explanation/lightbus-vs-celery/#lightbus-vs-celery","text":"Lightbus was conceived as a result of using Celery to communicate between multiple Python services.","title":"Lightbus vs Celery"},{"location":"explanation/lightbus-vs-celery/#differences-in-principle","text":"Celery is a task queue: A task queue is tightly coupled. The dispatching code must know what needs to be done A task queue typically doesn't return results Lightbus is a bus: A bus provides loose coupling. The dispatching code says what did happen, not what should happen (events) A bus provides bi-directional communication (remote procedure calls)","title":"Differences in principle"},{"location":"explanation/lightbus-vs-celery/#differences-in-practice","text":"A number of pain points were identified with Celery that Lightbus aims to address. In particular: Single vs multi- service \u2013 Celery is targeted as being a task queue for a service, rather than a means for multiple services to interact. Conceptual overlap \u2013 The mapping between concepts in Celery and the underlying broker (AMQP at the time) is both unclear and overlapping. Lightbus provides a limited set of well defined concepts to avoid this confusion. Non-sane defaults \u2013 Some Celery settings have non-sane defaults, making setup somewhat perilous at times. Lightbus provides sane defaults for most circumstances, and documentation specifically geared to certain use cases ( metrics , event sourcing ) Tight coupling (as discussed above) \u2013 Celery tasks define the action to take, not what happened. Lightbus uses events, which describe happened, and listening services decide the action to take. General feeling \u2013 Celery feels large and opaque, debugging issues was challenging. Lightbus aims to feel lightweight, with clear logging and debugging tools.","title":"Differences in practice"},{"location":"explanation/marshalling/","text":"Lightbus has four stages of data marshalling: Encode / Decode Serialize / Deserialize Validation Deform / Cast An inbound message will go through this process from top to bottom . An outbound message will go through this process from bottom to top . Inbound flow \u00b6 Messages arriving from the bus go through the following stages in order to prepare the data for use: Decode: Decode the incoming data (JSON decoding by default) Deserialise: Convert decoded data into a Message object Validate: Validate the incoming message against the JSON schema available on the bus. Cast: Best effort casting of parameters/results based on the locally available type hinting. This can be disabled with the cast_values configuration option . Outbound flow \u00b6 This is the reverse of the inbound flow. Messages being sent will go through the following process in order to prepare the data for transmission on bus: Deform: Lightbus handles NamedTuples , dataclasses and other classes by converting them into dictionaries. Other common types such as datetimes, Decimals etc are converted into strings. Internally this is referred to as the deform process and is the inverse of the cast process. Validate: Validate the outgoing message against the JSON schema available on the bus. Serialize: Structures the data in a way suitable for the transport. Encode: Converts the data to a form suitable for transmission. This typically means stringifying it, for which lightbus uses JSON encoding by default. About casting \u00b6 Casting is separate from validation, although both rely on type hints. Whereas validation uses a shared bus-wide schema to check data validity, casting uses any Python type hints available in the local codebase to marshall event and RPC parameters into a format useful to the service's developer.","title":"Marshalling"},{"location":"explanation/marshalling/#inbound-flow","text":"Messages arriving from the bus go through the following stages in order to prepare the data for use: Decode: Decode the incoming data (JSON decoding by default) Deserialise: Convert decoded data into a Message object Validate: Validate the incoming message against the JSON schema available on the bus. Cast: Best effort casting of parameters/results based on the locally available type hinting. This can be disabled with the cast_values configuration option .","title":"Inbound flow"},{"location":"explanation/marshalling/#outbound-flow","text":"This is the reverse of the inbound flow. Messages being sent will go through the following process in order to prepare the data for transmission on bus: Deform: Lightbus handles NamedTuples , dataclasses and other classes by converting them into dictionaries. Other common types such as datetimes, Decimals etc are converted into strings. Internally this is referred to as the deform process and is the inverse of the cast process. Validate: Validate the outgoing message against the JSON schema available on the bus. Serialize: Structures the data in a way suitable for the transport. Encode: Converts the data to a form suitable for transmission. This typically means stringifying it, for which lightbus uses JSON encoding by default.","title":"Outbound flow"},{"location":"explanation/marshalling/#about-casting","text":"Casting is separate from validation, although both rely on type hints. Whereas validation uses a shared bus-wide schema to check data validity, casting uses any Python type hints available in the local codebase to marshall event and RPC parameters into a format useful to the service's developer.","title":"About casting"},{"location":"explanation/performance/","text":"Performance \u00b6 Caveats Lightbus has yet to undergo any profiling or optimisation, therefore it is reasonable to expect performance to improve with time. The performance of Lightbus is primarily governed by the transports used. Lightbus currently only ships with Redis support. Individual process performance \u00b6 Simple benchmarking 1 on a 2018 MacBook Pro indicates the following execution times (plugins disabled, schema and validation enabled, no event/RPC parameters): Firing an event: \u2248 1.7ms (\u00b110%) Performing a remote procedure call: \u2248 6.9ms (\u00b110%) Redis performance \u00b6 The Redis server has the potential to be a central bottleneck in Lightbus' performance. You may start to run into these limits if you are sending tens thousands of events per second. In these cases you can either: Scale via Redis by setting up Redis Cluster Scale via Lightbus by specifying a different Redis instance per-API. See configuration See how to modify Lightbus for details on how to run the benchmarks via pytest \u21a9","title":"Performance"},{"location":"explanation/performance/#performance","text":"Caveats Lightbus has yet to undergo any profiling or optimisation, therefore it is reasonable to expect performance to improve with time. The performance of Lightbus is primarily governed by the transports used. Lightbus currently only ships with Redis support.","title":"Performance"},{"location":"explanation/performance/#individual-process-performance","text":"Simple benchmarking 1 on a 2018 MacBook Pro indicates the following execution times (plugins disabled, schema and validation enabled, no event/RPC parameters): Firing an event: \u2248 1.7ms (\u00b110%) Performing a remote procedure call: \u2248 6.9ms (\u00b110%)","title":"Individual process performance"},{"location":"explanation/performance/#redis-performance","text":"The Redis server has the potential to be a central bottleneck in Lightbus' performance. You may start to run into these limits if you are sending tens thousands of events per second. In these cases you can either: Scale via Redis by setting up Redis Cluster Scale via Lightbus by specifying a different Redis instance per-API. See configuration See how to modify Lightbus for details on how to run the benchmarks via pytest \u21a9","title":"Redis performance"},{"location":"explanation/rpcs/","text":"Remote Procedure Calls (RPCs) \u00b6 A remote procedure call is where you call a procedure available on the bus. The sequence of events is: You call the RPC, bus.auth.check_password() An autoratitive process for the auth API handles the request and sends the response. You receive the result Remote Procedure Calls are useful when: You require information from a service 1 You wish to wait until a remote procedure has completed an action You can perform an RPC as follows: support_case = bus . support . case . get ( pk = 123 ) RPCs do not provide a fire and forget mode of operation. Consider using events if you need this feature. At most once semantics \u00b6 Remote procedure calls will be processed at most once. In some situations the call will never be processed, in which case the client will raise a LigutbusTimeout exception. Considerations \u00b6 Whether to use RPCs or events for communication will depend upon your project's particular needs. Some considerations are: RPCs are conceptually simple . You call a procedure and wait for a response. You do not need to store any state locally, you can simply request data on demand (performance considerations aside). RPCs can be fragile . Any errors in the remote service will propagate to the client's service. You should handle these if possible. Their use within a codebase may be non-obvious, leading to poor performance. Lightbus tries to alleviate this somewhat by using the bus.api.method() calling format, making it clear that this is a bus-based operation. This is also achievable with events . However you will need to listen for the events and likely store the data locally. See the events section for further discussion. \u21a9","title":"Remote procedure calls"},{"location":"explanation/rpcs/#remote-procedure-calls-rpcs","text":"A remote procedure call is where you call a procedure available on the bus. The sequence of events is: You call the RPC, bus.auth.check_password() An autoratitive process for the auth API handles the request and sends the response. You receive the result Remote Procedure Calls are useful when: You require information from a service 1 You wish to wait until a remote procedure has completed an action You can perform an RPC as follows: support_case = bus . support . case . get ( pk = 123 ) RPCs do not provide a fire and forget mode of operation. Consider using events if you need this feature.","title":"Remote Procedure Calls (RPCs)"},{"location":"explanation/rpcs/#at-most-once-semantics","text":"Remote procedure calls will be processed at most once. In some situations the call will never be processed, in which case the client will raise a LigutbusTimeout exception.","title":"At most once semantics"},{"location":"explanation/rpcs/#considerations","text":"Whether to use RPCs or events for communication will depend upon your project's particular needs. Some considerations are: RPCs are conceptually simple . You call a procedure and wait for a response. You do not need to store any state locally, you can simply request data on demand (performance considerations aside). RPCs can be fragile . Any errors in the remote service will propagate to the client's service. You should handle these if possible. Their use within a codebase may be non-obvious, leading to poor performance. Lightbus tries to alleviate this somewhat by using the bus.api.method() calling format, making it clear that this is a bus-based operation. This is also achievable with events . However you will need to listen for the events and likely store the data locally. See the events section for further discussion. \u21a9","title":"Considerations"},{"location":"explanation/schema/","text":"Schema \u00b6 Lightbus creates a schema for each of your APIs using the type hints specified on the API. This schema is shared on the bus for consumption by other Lightbus clients. This provides a number of features: The availability of a particular API can be detected by remote clients RPCs, results, and events transmitted on the bus can be validated by both the sender and receiver Tooling can load the schema to provide additional functionality. For example, you can dump your production schema and run your development environment and tests against it. Note that an API's schema will only be available on the bus while there is a worker running to provides it. Once the worker process for an API shuts down the schema on the bus will be cleaned up shortly thereafter. See also \u00b6 See the schema reference section for details on how this works in practice. The schema is created using the JSON schema format, see the schema protocol for details of the transmission format.","title":"Schema"},{"location":"explanation/schema/#schema","text":"Lightbus creates a schema for each of your APIs using the type hints specified on the API. This schema is shared on the bus for consumption by other Lightbus clients. This provides a number of features: The availability of a particular API can be detected by remote clients RPCs, results, and events transmitted on the bus can be validated by both the sender and receiver Tooling can load the schema to provide additional functionality. For example, you can dump your production schema and run your development environment and tests against it. Note that an API's schema will only be available on the bus while there is a worker running to provides it. Once the worker process for an API shuts down the schema on the bus will be cleaned up shortly thereafter.","title":"Schema"},{"location":"explanation/schema/#see-also","text":"See the schema reference section for details on how this works in practice. The schema is created using the JSON schema format, see the schema protocol for details of the transmission format.","title":"See also"},{"location":"explanation/services/","text":"Services \u00b6 A service is one or more processes handling a common task. These processes operate as a tightly-coupled whole. All processes in a service will generally: Share the same API class definitions Moreover, they will normally share the same codebase Create a single instance of the bus client in bus.py using bus = lightbus.create() . For example, your company may have the following: An online store A price monitoring script An image resizing resizing process Each of these would be a service. The store service would have a web process and a Lightbus process. The image resizing service & and price monitoring services would each likely have a Lightbus process only. A simple lightbus deployment could therefore look something like this: In this example the following actions would take place: Django reads from the web service database in order to serve web content The online shop's Lightbus process receives pricing events from the price monitoring service. It updates products in the database using this new pricing data. When the Django app receives an image upload, it performs a RPC to the image resizing service to resize the image 1 . Making the Django process wait for an RPC to respond is probably a bad idea in this case, but it illustrates how it could be done. Using an event (which is fire-and-forget) could be more suitable in reality. \u21a9","title":"Services"},{"location":"explanation/services/#services","text":"A service is one or more processes handling a common task. These processes operate as a tightly-coupled whole. All processes in a service will generally: Share the same API class definitions Moreover, they will normally share the same codebase Create a single instance of the bus client in bus.py using bus = lightbus.create() . For example, your company may have the following: An online store A price monitoring script An image resizing resizing process Each of these would be a service. The store service would have a web process and a Lightbus process. The image resizing service & and price monitoring services would each likely have a Lightbus process only. A simple lightbus deployment could therefore look something like this: In this example the following actions would take place: Django reads from the web service database in order to serve web content The online shop's Lightbus process receives pricing events from the price monitoring service. It updates products in the database using this new pricing data. When the Django app receives an image upload, it performs a RPC to the image resizing service to resize the image 1 . Making the Django process wait for an RPC to respond is probably a bad idea in this case, but it illustrates how it could be done. Using an event (which is fire-and-forget) could be more suitable in reality. \u21a9","title":"Services"},{"location":"explanation/transports/","text":"Transports \u00b6 Transports provide the communications system for Lightbus. There are four types of transport: RPC transports \u2013 sends and consumes RPC calls Result transports \u2013 sends and receives RPC results Event transports \u2013 sends and consumes events Schema transports \u2013 stores and retrieves the bus schema Lightbus ships with a Redis-backed implementation of each of these transports. For configuration details see the transport configuration reference . Lightbus can be configured to use custom transports either globally, or on a per-API level.","title":"Transports"},{"location":"explanation/transports/#transports","text":"Transports provide the communications system for Lightbus. There are four types of transport: RPC transports \u2013 sends and consumes RPC calls Result transports \u2013 sends and receives RPC results Event transports \u2013 sends and consumes events Schema transports \u2013 stores and retrieves the bus schema Lightbus ships with a Redis-backed implementation of each of these transports. For configuration details see the transport configuration reference . Lightbus can be configured to use custom transports either globally, or on a per-API level.","title":"Transports"},{"location":"howto/","text":"Howto overview \u00b6 In this section we address specific problems and common use cases . As with the tutorials we will link to concepts as we go, but the priority here is to provide a clear path to a solution.","title":"Overview"},{"location":"howto/#howto-overview","text":"In this section we address specific problems and common use cases . As with the tutorials we will link to concepts as we go, but the priority here is to provide a clear path to a solution.","title":"Howto overview"},{"location":"howto/access-your-bus-client/","text":"How to access your bus client \u00b6 You create your bus client in your bus module (typically called bus.py ) with the line: # Creating your bus client in your bus.py file import lightbus bus = lightbus . create () However, you will often need to make use of your bus client in other areas of your codebase. For example, you may need to fire an event when a web form is submitted. You can access the bus client in two ways. Method 1: Direct import (recommended) \u00b6 The first approach is to import your bus client directly from your bus module, in the same way you would import anything else in your codebase: # For example from bus import bus # ...or if your service has its own package from my_service.bus import bus You should use this approach in code which is specific to your service (i.e. non-shared/non-library code). This approach is more explicit (good), but hard codes the path to your bus module (bad for shared code). Method 2: get_bus() \u00b6 The second approach uses the lightbus.get_bus() function. This will use the module loading configuration to determine the bus module location. If the bus module has already been imported, then the module's bus attribute will simply be returned. Otherwise the bus module will be imported first. # Anywhere in your codebase import lightbus bus = lightbus . get_bus () This approach is best suited to when you do not know where your bus module will be at runtime. This could be the case when: Writing shared code which is used by multiple services Writing a third party library","title":"Access your bus client"},{"location":"howto/access-your-bus-client/#how-to-access-your-bus-client","text":"You create your bus client in your bus module (typically called bus.py ) with the line: # Creating your bus client in your bus.py file import lightbus bus = lightbus . create () However, you will often need to make use of your bus client in other areas of your codebase. For example, you may need to fire an event when a web form is submitted. You can access the bus client in two ways.","title":"How to access your bus client"},{"location":"howto/access-your-bus-client/#method-1-direct-import-recommended","text":"The first approach is to import your bus client directly from your bus module, in the same way you would import anything else in your codebase: # For example from bus import bus # ...or if your service has its own package from my_service.bus import bus You should use this approach in code which is specific to your service (i.e. non-shared/non-library code). This approach is more explicit (good), but hard codes the path to your bus module (bad for shared code).","title":"Method 1: Direct import (recommended)"},{"location":"howto/access-your-bus-client/#method-2-get_bus","text":"The second approach uses the lightbus.get_bus() function. This will use the module loading configuration to determine the bus module location. If the bus module has already been imported, then the module's bus attribute will simply be returned. Otherwise the bus module will be imported first. # Anywhere in your codebase import lightbus bus = lightbus . get_bus () This approach is best suited to when you do not know where your bus module will be at runtime. This could be the case when: Writing shared code which is used by multiple services Writing a third party library","title":"Method 2: get_bus()"},{"location":"howto/event-sourcing/","text":"How to use Lightbus for event sourcing \u00b6 We won't go into the details of event sourcing here, but we can roughly describe our messaging needs as follows: We are optimising for reliability and completeness, performance is secondary Sent events must be valid Received events must be processed regardless of their validity Event history is very important Note Your needs may not precisely match this scenario, so be prepared to tweak the following configuration to your needs. Global configuration \u00b6 For the above event sourced scenario, a sample Lightbus configuration may look like something like this: # Lightbus config for event sourcing bus : schema : transport : redis : url : \"redis://redis_host:6379/0\" apis : # Here we specify the default for your entire bus, but you could # also specify the config for a specific API by using the API's name # instead of 'default'. default : validate : # Sent (outgoing) events must be valid outgoing : true # Received (incoming) events must be processed # regardless of their validity incoming : false event_transport : redis : url : 'redis://redis_host:6379/0' # Load few events into order to prioritise consistency batch_size : 1 # Do not truncate the event stream. We keep all events # as these events are our source of truth max_stream_length : null # Per-API streams, as we wish to prioritise # ordering (this is the default) stream_use : \"per_api\" Run a single Lightbus worker \u00b6 Running only a single Lightbus worker process (with a specific process name) will ensure messages are processed in order. lightbus run --service-name=example_service --process-name=worker Set process names \u00b6 If your event volume requires multiple workers then ensure you set a deterministic per-process name for each. This will allow restarted workers to immediately pickup any previously claimed messages without needing to wait for a timeout. For example, if you have three Lightbus workers you can start each as follows: lightbus run --service-name=example_service --process-name=worker1 lightbus run --service-name=example_service --process-name=worker2 lightbus run --service-name=example_service --process-name=worker3 Ordering will not be maintained when running multiple workers.","title":"Use Lightbus for event sourcing"},{"location":"howto/event-sourcing/#how-to-use-lightbus-for-event-sourcing","text":"We won't go into the details of event sourcing here, but we can roughly describe our messaging needs as follows: We are optimising for reliability and completeness, performance is secondary Sent events must be valid Received events must be processed regardless of their validity Event history is very important Note Your needs may not precisely match this scenario, so be prepared to tweak the following configuration to your needs.","title":"How to use Lightbus for event sourcing"},{"location":"howto/event-sourcing/#global-configuration","text":"For the above event sourced scenario, a sample Lightbus configuration may look like something like this: # Lightbus config for event sourcing bus : schema : transport : redis : url : \"redis://redis_host:6379/0\" apis : # Here we specify the default for your entire bus, but you could # also specify the config for a specific API by using the API's name # instead of 'default'. default : validate : # Sent (outgoing) events must be valid outgoing : true # Received (incoming) events must be processed # regardless of their validity incoming : false event_transport : redis : url : 'redis://redis_host:6379/0' # Load few events into order to prioritise consistency batch_size : 1 # Do not truncate the event stream. We keep all events # as these events are our source of truth max_stream_length : null # Per-API streams, as we wish to prioritise # ordering (this is the default) stream_use : \"per_api\"","title":"Global configuration"},{"location":"howto/event-sourcing/#run-a-single-lightbus-worker","text":"Running only a single Lightbus worker process (with a specific process name) will ensure messages are processed in order. lightbus run --service-name=example_service --process-name=worker","title":"Run a single Lightbus worker"},{"location":"howto/event-sourcing/#set-process-names","text":"If your event volume requires multiple workers then ensure you set a deterministic per-process name for each. This will allow restarted workers to immediately pickup any previously claimed messages without needing to wait for a timeout. For example, if you have three Lightbus workers you can start each as follows: lightbus run --service-name=example_service --process-name=worker1 lightbus run --service-name=example_service --process-name=worker2 lightbus run --service-name=example_service --process-name=worker3 Ordering will not be maintained when running multiple workers.","title":"Set process names"},{"location":"howto/metrics/","text":"How to use Lightbus for metrics \u00b6 When we talk about metrics we may mean all or any of the following: Current information is most important Previous events will become irrelevant as soon as new data is received Lost events are therefore tolerable, as long as we keep up with new events Events may be high volume, so optimisations may be needed Note Your needs may not precisely match this scenario, so be prepared to tweak the following configuration to your needs. For the above metrics-based scenario, a sample Lightbus configuration may look like something like this: # Lightbus config for metrics bus : schema : transport : redis : url : \"redis://redis_host:6379/0\" apis : # Here we specify the default for your entire bus, but you could # also specify the config for a specific API by using the API's name # instead of 'default'. default : # Disable validation to enhance performance validate : outgoing : false incoming : false # Assume we will be transmitting simple types, so we can bypass casting for performance cast_values : false event_transport : redis : url : 'redis://redis_host:6379/0' # Load in many events at once for performance improvements batch_size : 100 # No need to keep many historical events around max_stream_length : 10000 # Per-event streams, to allow selective consumption of metrics stream_use : \"per_event\"","title":"Use Lightbus for realtime metrics"},{"location":"howto/metrics/#how-to-use-lightbus-for-metrics","text":"When we talk about metrics we may mean all or any of the following: Current information is most important Previous events will become irrelevant as soon as new data is received Lost events are therefore tolerable, as long as we keep up with new events Events may be high volume, so optimisations may be needed Note Your needs may not precisely match this scenario, so be prepared to tweak the following configuration to your needs. For the above metrics-based scenario, a sample Lightbus configuration may look like something like this: # Lightbus config for metrics bus : schema : transport : redis : url : \"redis://redis_host:6379/0\" apis : # Here we specify the default for your entire bus, but you could # also specify the config for a specific API by using the API's name # instead of 'default'. default : # Disable validation to enhance performance validate : outgoing : false incoming : false # Assume we will be transmitting simple types, so we can bypass casting for performance cast_values : false event_transport : redis : url : 'redis://redis_host:6379/0' # Load in many events at once for performance improvements batch_size : 100 # No need to keep many historical events around max_stream_length : 10000 # Per-event streams, to allow selective consumption of metrics stream_use : \"per_event\"","title":"How to use Lightbus for metrics"},{"location":"howto/migrate-from-celery-to-lightbus/","text":"How to migrate from Celery to Lightbus \u00b6 Migration from Celery to Lightbus can be a very straightforward process depending on how you currently make use of celery. However, before you start out you should take time to familiarise yourself with the principles behind Lightbus, and its differences to Celery: Lightbus anatomy lesson Lightbus vs Celery Queued tasks \u00b6 Celery is often used to schedule background tasks for later processing. For example, a web server may schedule a task to send a welcome email to a new user. This task will go into your Celery queue and be handled by a Celery worker process. The web process does not need any response from the worker, it only needs to know that it will be handled. In Lightbus these are events . We can convert this to Lightbus as follows: # bus.py import lightbus # Create an API for everything user-related class UserApi ( lightbus . Api ): # We specify the event which has occurred (a user signed up) user_signed_up = lightbus . Event ( parameters = ( 'email' , 'username' )) class Meta : # Name our API # (used below to setup our listener) name = \"user\" # Create our bus client (this is a requirement of our bus.py file) bus = lightbus . create () # Register the API (see docs: Explanation -> APIs) bus . client . register_api ( UserApi ()) @bus . client . on_start () def bus_start ( ** kwargs ): from my_app.somewhere import send_welcome_email # Listen for this event being fired. # When the event is fired, send_welcome_email will be called as follows: # send_welcome_email(event, email, username) bus . user . user_signed_up . listen ( send_welcome_email , listener_name = \"send_welcome_email\" ) Then, within our web server code we can send an email as follows: # Your web server code # Import the bus client from your bus.py file from bus import bus # Your registration view @route ( '/register' , methods = ( 'GET' , 'POST' )) def register (): if request . method == 'POST' : # ...validate and create user... # Send registration email via the Lightbus process bus . user . user_signed_up . fire ( username = request . form [ 'username' ], email = request . form [ 'email' ] ) else : # ... This will fire an event on the bus, and our lightbus run process will receive this event and send the welcome email. Fetching Celery task results \u00b6 Sometimes you wish to have a Celery worker process execute a task for you, and then return the result. For example, perhaps you need to get the URL to a resized user avatar image. Image resizing is computationally expensive, so you prefer to do this on your Celery workers. You want the worker to do this task for you, and the web server will wait for it to be done, and the web server needs a response (the image URL) from the worker. In Lightbus these are remote procedure calls (RPCs). We can implement the above example as follows: # bus.py (as above, with one addition) import lightbus class UserApi ( lightbus . Api ): user_signed_up = lightbus . Event ( parameters = ( \"email\" , \"username\" )) class Meta : name = \"user\" # This is our new remote procedure def get_avatar_url ( self , username , width , height ): # ... do some resizing & uploading ... return f \"https://example.com/images/ { width } / { height } / { username } .png\" bus = lightbus . create () bus . client . register_api ( UserApi ()) @bus . client . on_start () def bus_start ( ** kwargs ): from my_app.somewhere import send_welcome_email bus . user . user_signed_up . listen ( send_welcome_email , listener_name = \"send_welcome_email\" ) We can then use this within our web server as follows: # Your web server code # Import the bus client from your bus.py file from bus import bus # Your registration view @route ( '/user-profile/' , methods = ( 'GET' ,)) def user_profile ( username ): # Call our new remote procedure image_url = bus . user . get_avatar_url ( username = username , width = 500 , height = 500 ) return f \"

{ username }

\" Now, as long as our lightbus run process is running, the web server will be able to call the remote procedure and render the user profile HTML. Scheduled tasks \u00b6 Lightbus can replace Celery's periodic tasks through the use of the @bus.client.every() and/or @bus.client.schedule() decorators (see how to schedule recurring tasks ). For example: # bus.py import lightbus bus = lightbus . create () @bus . client . every ( hours = 1 ) def every_hour (): # Will be called every hour refresh_customer_data () The lightbus run command will then execute this function every hour. See how to schedule recurring tasks for more details on the available scheduling options.","title":"Migrate from Celery to Lightbus"},{"location":"howto/migrate-from-celery-to-lightbus/#how-to-migrate-from-celery-to-lightbus","text":"Migration from Celery to Lightbus can be a very straightforward process depending on how you currently make use of celery. However, before you start out you should take time to familiarise yourself with the principles behind Lightbus, and its differences to Celery: Lightbus anatomy lesson Lightbus vs Celery","title":"How to migrate from Celery to Lightbus"},{"location":"howto/migrate-from-celery-to-lightbus/#queued-tasks","text":"Celery is often used to schedule background tasks for later processing. For example, a web server may schedule a task to send a welcome email to a new user. This task will go into your Celery queue and be handled by a Celery worker process. The web process does not need any response from the worker, it only needs to know that it will be handled. In Lightbus these are events . We can convert this to Lightbus as follows: # bus.py import lightbus # Create an API for everything user-related class UserApi ( lightbus . Api ): # We specify the event which has occurred (a user signed up) user_signed_up = lightbus . Event ( parameters = ( 'email' , 'username' )) class Meta : # Name our API # (used below to setup our listener) name = \"user\" # Create our bus client (this is a requirement of our bus.py file) bus = lightbus . create () # Register the API (see docs: Explanation -> APIs) bus . client . register_api ( UserApi ()) @bus . client . on_start () def bus_start ( ** kwargs ): from my_app.somewhere import send_welcome_email # Listen for this event being fired. # When the event is fired, send_welcome_email will be called as follows: # send_welcome_email(event, email, username) bus . user . user_signed_up . listen ( send_welcome_email , listener_name = \"send_welcome_email\" ) Then, within our web server code we can send an email as follows: # Your web server code # Import the bus client from your bus.py file from bus import bus # Your registration view @route ( '/register' , methods = ( 'GET' , 'POST' )) def register (): if request . method == 'POST' : # ...validate and create user... # Send registration email via the Lightbus process bus . user . user_signed_up . fire ( username = request . form [ 'username' ], email = request . form [ 'email' ] ) else : # ... This will fire an event on the bus, and our lightbus run process will receive this event and send the welcome email.","title":"Queued tasks"},{"location":"howto/migrate-from-celery-to-lightbus/#fetching-celery-task-results","text":"Sometimes you wish to have a Celery worker process execute a task for you, and then return the result. For example, perhaps you need to get the URL to a resized user avatar image. Image resizing is computationally expensive, so you prefer to do this on your Celery workers. You want the worker to do this task for you, and the web server will wait for it to be done, and the web server needs a response (the image URL) from the worker. In Lightbus these are remote procedure calls (RPCs). We can implement the above example as follows: # bus.py (as above, with one addition) import lightbus class UserApi ( lightbus . Api ): user_signed_up = lightbus . Event ( parameters = ( \"email\" , \"username\" )) class Meta : name = \"user\" # This is our new remote procedure def get_avatar_url ( self , username , width , height ): # ... do some resizing & uploading ... return f \"https://example.com/images/ { width } / { height } / { username } .png\" bus = lightbus . create () bus . client . register_api ( UserApi ()) @bus . client . on_start () def bus_start ( ** kwargs ): from my_app.somewhere import send_welcome_email bus . user . user_signed_up . listen ( send_welcome_email , listener_name = \"send_welcome_email\" ) We can then use this within our web server as follows: # Your web server code # Import the bus client from your bus.py file from bus import bus # Your registration view @route ( '/user-profile/' , methods = ( 'GET' ,)) def user_profile ( username ): # Call our new remote procedure image_url = bus . user . get_avatar_url ( username = username , width = 500 , height = 500 ) return f \"

{ username }

\" Now, as long as our lightbus run process is running, the web server will be able to call the remote procedure and render the user profile HTML.","title":"Fetching Celery task results"},{"location":"howto/migrate-from-celery-to-lightbus/#scheduled-tasks","text":"Lightbus can replace Celery's periodic tasks through the use of the @bus.client.every() and/or @bus.client.schedule() decorators (see how to schedule recurring tasks ). For example: # bus.py import lightbus bus = lightbus . create () @bus . client . every ( hours = 1 ) def every_hour (): # Will be called every hour refresh_customer_data () The lightbus run command will then execute this function every hour. See how to schedule recurring tasks for more details on the available scheduling options.","title":"Scheduled tasks"},{"location":"howto/modify-lightbus/","text":"How to modify Lightbus \u00b6 Contributions to Lightbus are very welcome. This will talk you though setting up a development installation of Lightbus. Using this installation you will be able to: Modify the Lightbus source code and/or documentation Run the Lightbus test suite View any modified documentation locally Use your development Lightbus install within another project Prerequisites \u00b6 You will need: Redis running locally Getting the code \u00b6 Checkout the Lightbus repository from GitHub: git clone https://github.com/adamcharnock/lightbus.git cd lightbus Environment setup \u00b6 It is a good idea to put asyncio into debug mode. You can do this by setting the following in your shell's environment: PYTHONASYNCIODEBUG=1 The testing framework will also need to know where your redis instance is running. This is set using the REDIS_URL and REDIS_URL_B environment variables: # Default values shown below REDIS_URL=redis://127.0.0.1:6379/10 REDIS_URL_B=redis://127.0.0.1:6379/11 Installation \u00b6 You will need to install Lightbus' standard dependencies, as well as Lightbus' development dependencies. Note that you may need to install poetry if you do not already have it: You can install both of these groups as follows: # Install standard & dev dependencies into a virtual environment poetry install # Enter the virtual environment you have created, # thereby giving you access the the pytest and mkdocs commands (below) poetry shell Running the tests \u00b6 You can run the tests once you have completed the above steps: pytest Note that you can run subsets of the tests as follows: pytest -m unit # Fast with high coverage pytest -m integration pytest -m reliability pytest -m benchmark Viewing the Lightbus documentation locally \u00b6 You can view the documentation of your local Lightbus install as follows: # Serve the docs locally mkdocs serve You can now view the documentation http://127.0.0.1:8000 . The documentation source can be found in docs/ . You can also check for broken links within the docs by running the check_links.sh script: # Check for broken links ./docs/check_links.sh Using within your project \u00b6 You can install your development Lightbus install within your project as follows: # Within your own project # Make sure you remove any existing lightbus version pip uninstall lightbus # Install your local development lightbus pip install --editable /path/to/your/local/lightbus See also \u00b6 Being familiar with the explanation section is highly recommended if modifying the Lightbus source","title":"Modify Lightbus"},{"location":"howto/modify-lightbus/#how-to-modify-lightbus","text":"Contributions to Lightbus are very welcome. This will talk you though setting up a development installation of Lightbus. Using this installation you will be able to: Modify the Lightbus source code and/or documentation Run the Lightbus test suite View any modified documentation locally Use your development Lightbus install within another project","title":"How to modify Lightbus"},{"location":"howto/modify-lightbus/#prerequisites","text":"You will need: Redis running locally","title":"Prerequisites"},{"location":"howto/modify-lightbus/#getting-the-code","text":"Checkout the Lightbus repository from GitHub: git clone https://github.com/adamcharnock/lightbus.git cd lightbus","title":"Getting the code"},{"location":"howto/modify-lightbus/#environment-setup","text":"It is a good idea to put asyncio into debug mode. You can do this by setting the following in your shell's environment: PYTHONASYNCIODEBUG=1 The testing framework will also need to know where your redis instance is running. This is set using the REDIS_URL and REDIS_URL_B environment variables: # Default values shown below REDIS_URL=redis://127.0.0.1:6379/10 REDIS_URL_B=redis://127.0.0.1:6379/11","title":"Environment setup"},{"location":"howto/modify-lightbus/#installation","text":"You will need to install Lightbus' standard dependencies, as well as Lightbus' development dependencies. Note that you may need to install poetry if you do not already have it: You can install both of these groups as follows: # Install standard & dev dependencies into a virtual environment poetry install # Enter the virtual environment you have created, # thereby giving you access the the pytest and mkdocs commands (below) poetry shell","title":"Installation"},{"location":"howto/modify-lightbus/#running-the-tests","text":"You can run the tests once you have completed the above steps: pytest Note that you can run subsets of the tests as follows: pytest -m unit # Fast with high coverage pytest -m integration pytest -m reliability pytest -m benchmark","title":"Running the tests"},{"location":"howto/modify-lightbus/#viewing-the-lightbus-documentation-locally","text":"You can view the documentation of your local Lightbus install as follows: # Serve the docs locally mkdocs serve You can now view the documentation http://127.0.0.1:8000 . The documentation source can be found in docs/ . You can also check for broken links within the docs by running the check_links.sh script: # Check for broken links ./docs/check_links.sh","title":"Viewing the Lightbus documentation locally"},{"location":"howto/modify-lightbus/#using-within-your-project","text":"You can install your development Lightbus install within your project as follows: # Within your own project # Make sure you remove any existing lightbus version pip uninstall lightbus # Install your local development lightbus pip install --editable /path/to/your/local/lightbus","title":"Using within your project"},{"location":"howto/modify-lightbus/#see-also","text":"Being familiar with the explanation section is highly recommended if modifying the Lightbus source","title":"See also"},{"location":"howto/run-background-tasks/","text":"How to run background tasks \u00b6 Sometimes you may wish to run arbitrary asyncio tasks in the background of the lightbus run process. You can set these up in your bus.py file: # bus.py import asyncio import lightbus bus = lightbus . create () async def my_background_task (): while True : await asyncio . sleep ( 1 ) print ( \"Hello!\" ) @bus . client . on_start () def on_startup ( ** kwargs ): bus . client . add_background_task ( my_background_task ()) Important points to note are: The background task will be automatically cancelled when the bus is closed. Any errors in the background task will be bubbled up and cause the Lightbus process to exit. If this is not desired you can implement your own try/except handling within the function being executed. Note If you wish to schedule a recurring task then you should probably use @bus.client.every() or @bus.client.schedule() . See how to schedule recurring tasks .","title":"Run background tasks"},{"location":"howto/run-background-tasks/#how-to-run-background-tasks","text":"Sometimes you may wish to run arbitrary asyncio tasks in the background of the lightbus run process. You can set these up in your bus.py file: # bus.py import asyncio import lightbus bus = lightbus . create () async def my_background_task (): while True : await asyncio . sleep ( 1 ) print ( \"Hello!\" ) @bus . client . on_start () def on_startup ( ** kwargs ): bus . client . add_background_task ( my_background_task ()) Important points to note are: The background task will be automatically cancelled when the bus is closed. Any errors in the background task will be bubbled up and cause the Lightbus process to exit. If this is not desired you can implement your own try/except handling within the function being executed. Note If you wish to schedule a recurring task then you should probably use @bus.client.every() or @bus.client.schedule() . See how to schedule recurring tasks .","title":"How to run background tasks"},{"location":"howto/schedule-recurring-tasks/","text":"How to schedule recurring tasks \u00b6 Recurring tasks can be scheduled in two ways: The @bus.client.every() decorator \u2013 Will execute a function or coroutine at a given interval The @bus.client.schedule() decorator \u2013 Similar to every() , but takes complex schedules as provided by the schedule library. Simple recurring tasks using @bus.client.every() \u00b6 Lightbus natively supports simple recurring tasks using the @bus.client.every() decorator: # bus.py import lightbus bus = lightbus . create () @bus . client . every ( seconds = 1 ) def do_it (): print ( \"Hello!\" ) The interval can be specified using the seconds , minutes , hours , and days keys. Pass also_run_immediately=True to execute the function/coroutine immediately, as well as at the given interval. Complex schedules using @bus.client.schedule() \u00b6 Lightbus also supports using schedules specified using the schedule library. This allows for schedules such as 'every Monday at 1am', rather than simple intervals. For example: import lightbus import schedule bus = lightbus . create () # Run the task every 1-3 seconds, varying randomly @bus . client . schedule ( schedule . every ( 1 ) . to ( 3 ) . seconds ) def do_it (): print ( \"Hello using schedule library\" ) Long running tasks \u00b6 If your tasks are long running you may prefer to handle these in a separate Lightbus process. This will avoid blocking the processing of incoming events and RPCs. For example, the default RPC timeout is 5 seconds. Any task which runs for longer than this has the possibility of causing incoming RPCs to timeout. You can move your task processing to a separate process as follows: # Process 1: Handles scheduled tasks only lightbus run --only tasks # Process 2: Handles everything else (events and rpcs) lightbus run --skip tasks","title":"Schedule recurring tasks"},{"location":"howto/schedule-recurring-tasks/#how-to-schedule-recurring-tasks","text":"Recurring tasks can be scheduled in two ways: The @bus.client.every() decorator \u2013 Will execute a function or coroutine at a given interval The @bus.client.schedule() decorator \u2013 Similar to every() , but takes complex schedules as provided by the schedule library.","title":"How to schedule recurring tasks"},{"location":"howto/schedule-recurring-tasks/#simple-recurring-tasks-using-busclientevery","text":"Lightbus natively supports simple recurring tasks using the @bus.client.every() decorator: # bus.py import lightbus bus = lightbus . create () @bus . client . every ( seconds = 1 ) def do_it (): print ( \"Hello!\" ) The interval can be specified using the seconds , minutes , hours , and days keys. Pass also_run_immediately=True to execute the function/coroutine immediately, as well as at the given interval.","title":"Simple recurring tasks using @bus.client.every()"},{"location":"howto/schedule-recurring-tasks/#complex-schedules-using-busclientschedule","text":"Lightbus also supports using schedules specified using the schedule library. This allows for schedules such as 'every Monday at 1am', rather than simple intervals. For example: import lightbus import schedule bus = lightbus . create () # Run the task every 1-3 seconds, varying randomly @bus . client . schedule ( schedule . every ( 1 ) . to ( 3 ) . seconds ) def do_it (): print ( \"Hello using schedule library\" )","title":"Complex schedules using @bus.client.schedule()"},{"location":"howto/schedule-recurring-tasks/#long-running-tasks","text":"If your tasks are long running you may prefer to handle these in a separate Lightbus process. This will avoid blocking the processing of incoming events and RPCs. For example, the default RPC timeout is 5 seconds. Any task which runs for longer than this has the possibility of causing incoming RPCs to timeout. You can move your task processing to a separate process as follows: # Process 1: Handles scheduled tasks only lightbus run --only tasks # Process 2: Handles everything else (events and rpcs) lightbus run --skip tasks","title":"Long running tasks"},{"location":"howto/write-idempotent-event-handlers/","text":"How to write idempotent event handlers \u00b6 An idempotent function is a function which, when called multiple times with the same parameters, will not result in any change after the first call. As Wikipedia puts it : Idempotence is the property of certain operations in mathematics and computer science whereby they can be applied multiple times without changing the result beyond the initial application Why is idempotency useful? \u00b6 Consider that Lightbus events will be delivered at least once . This is in comparison to RPCs which are called at most once . This is not Lightbus trying to be awkward, rather it is because it is theoretically impossible to ensure exactly-once delivery. Lightbus therefore guarantees you that events will always arrive, but with the trade-off that sometimes the event may arrive multiple times. In the wost case, your event handlers will be executed multiple times. In some cases this will not cause a problem. For example, an event handler which performs a simple update to a record with a specific ID. Executing this update a second time will leave the record in the same state it was in after the first execution. However, an event handler which creates a new record will face a problem. If the event handler is executed twice then two records will be created, even though the event was identical in both cases. Ensuring idempotency for database operations \u00b6 A simple and reliable way to ensure idempotency is through the use of UUIDs and upserts . A UUID is a standard way of producing a globally unique ID. An upsert is a combination of an 'update' and an 'insert' which says: *'Insert this record, but if you find a duplicate then update it instead'. Take the following event handler as an example: # BAD, not idempotent def handle_page_view ( event , url ): db . execute ( \"INSERT INTO views (url)\" ) This is not an idempotent event handler because multiple invocations for the same event will result in multiple records being created. An idempotent version of the above can be rewritten as: # GOOD, idempotent def handle_page_view ( event , uuid , url ): try : # Try to perform an insert first (this should normally work fine) db . execute ( \"INSERT INTO views ( %s , %s )\" , [ uuid , url ]) except IntegrityError : # UUID already exists, so do an update db . execute ( \"UPDATE views SET url = %s WHERE view_uuid = %s \" , [ url , uuid ]) This handler can be executed any number of times for the same parameters without ever creating duplicate records. It is idempotent. Note that the views table must include a UUID column which is UNIQUE , otherwise an error would not be raised. Additionally, the code which fires the event must now provide a value for the UUID parameter. Many databases and frameworks provide direct support for performing upsert operations which will make the above simpler (and therefore less error prone). Upserts in Django \u00b6 Django provides the update_or_create() method which serves exactly this purpose: # Idempotent def handle_page_view ( event , uuid , url ): Views . objects . update_or_create ( uuid = uuid , defaults = { 'url' : url }) Upserts in SQL \u00b6 Many relational databases can also perform upsert in a single SQL query: INSERT INTO views ( uuid , url ) VALUES ( 'uuid-here' , 'http://lightbus.org' ) ON CONFLICT ( uuid ) DO UPDATE SET url = 'http://lightbus.org' ; Ensuring idempotency in other cases \u00b6 There are additional areas where idempotency can become relevant: Event handlers which call external APIs Event handlers which send email Handling these cases is beyond the scope of this documentation. Rest APIs which support idempotency keys will likely help here 1 . You may also decide that multiple invocations are acceptable in some situations as long as they are sufficiently rare (for example, sending emails). Blog post from Stripe: Designing robust and predictable APIs with idempotency \u21a9","title":"Write idempotent event handlers"},{"location":"howto/write-idempotent-event-handlers/#how-to-write-idempotent-event-handlers","text":"An idempotent function is a function which, when called multiple times with the same parameters, will not result in any change after the first call. As Wikipedia puts it : Idempotence is the property of certain operations in mathematics and computer science whereby they can be applied multiple times without changing the result beyond the initial application","title":"How to write idempotent event handlers"},{"location":"howto/write-idempotent-event-handlers/#why-is-idempotency-useful","text":"Consider that Lightbus events will be delivered at least once . This is in comparison to RPCs which are called at most once . This is not Lightbus trying to be awkward, rather it is because it is theoretically impossible to ensure exactly-once delivery. Lightbus therefore guarantees you that events will always arrive, but with the trade-off that sometimes the event may arrive multiple times. In the wost case, your event handlers will be executed multiple times. In some cases this will not cause a problem. For example, an event handler which performs a simple update to a record with a specific ID. Executing this update a second time will leave the record in the same state it was in after the first execution. However, an event handler which creates a new record will face a problem. If the event handler is executed twice then two records will be created, even though the event was identical in both cases.","title":"Why is idempotency useful?"},{"location":"howto/write-idempotent-event-handlers/#ensuring-idempotency-for-database-operations","text":"A simple and reliable way to ensure idempotency is through the use of UUIDs and upserts . A UUID is a standard way of producing a globally unique ID. An upsert is a combination of an 'update' and an 'insert' which says: *'Insert this record, but if you find a duplicate then update it instead'. Take the following event handler as an example: # BAD, not idempotent def handle_page_view ( event , url ): db . execute ( \"INSERT INTO views (url)\" ) This is not an idempotent event handler because multiple invocations for the same event will result in multiple records being created. An idempotent version of the above can be rewritten as: # GOOD, idempotent def handle_page_view ( event , uuid , url ): try : # Try to perform an insert first (this should normally work fine) db . execute ( \"INSERT INTO views ( %s , %s )\" , [ uuid , url ]) except IntegrityError : # UUID already exists, so do an update db . execute ( \"UPDATE views SET url = %s WHERE view_uuid = %s \" , [ url , uuid ]) This handler can be executed any number of times for the same parameters without ever creating duplicate records. It is idempotent. Note that the views table must include a UUID column which is UNIQUE , otherwise an error would not be raised. Additionally, the code which fires the event must now provide a value for the UUID parameter. Many databases and frameworks provide direct support for performing upsert operations which will make the above simpler (and therefore less error prone).","title":"Ensuring idempotency for database operations"},{"location":"howto/write-idempotent-event-handlers/#upserts-in-django","text":"Django provides the update_or_create() method which serves exactly this purpose: # Idempotent def handle_page_view ( event , uuid , url ): Views . objects . update_or_create ( uuid = uuid , defaults = { 'url' : url })","title":"Upserts in Django"},{"location":"howto/write-idempotent-event-handlers/#upserts-in-sql","text":"Many relational databases can also perform upsert in a single SQL query: INSERT INTO views ( uuid , url ) VALUES ( 'uuid-here' , 'http://lightbus.org' ) ON CONFLICT ( uuid ) DO UPDATE SET url = 'http://lightbus.org' ;","title":"Upserts in SQL"},{"location":"howto/write-idempotent-event-handlers/#ensuring-idempotency-in-other-cases","text":"There are additional areas where idempotency can become relevant: Event handlers which call external APIs Event handlers which send email Handling these cases is beyond the scope of this documentation. Rest APIs which support idempotency keys will likely help here 1 . You may also decide that multiple invocations are acceptable in some situations as long as they are sufficiently rare (for example, sending emails). Blog post from Stripe: Designing robust and predictable APIs with idempotency \u21a9","title":"Ensuring idempotency in other cases"},{"location":"includes/if-you-get-stuck/","text":"If you get stuck... It is really useful to hear from people who have encountered a problem or got stuck. Hearing from you means we can improve our documentation and error messages. If you get stuck drop then please drop an email to adam@adamcharnock.com , visit the Lightbus discord server , or call me (Adam) on +442032896620. The more information you can include the better (problem description, screenshots, and code are all useful)","title":"If you get stuck"},{"location":"includes/note-configuration-auto-complete/","text":"Configuration auto-complete using JSON Schema Many code editors support using a JSON schema to provide auto-complete and validation when editing a JSON file. If you wish, you can write your configuration in JSON (rather than YAML), and load the following JSON schema into your editor: https://lightbus.org/static/default-config-schema.json This will provide you with autocomplete and validation for Lightbus' various configuration options. If you are using custom transports or plugins you should generate your own config schema .","title":"Note configuration auto complete"},{"location":"reference/","text":"Reference overview \u00b6 This section provides detailed information regarding the specific features of Lightbus . A grasp of the tutorial and explantion sections will be useful here.","title":"Overview"},{"location":"reference/#reference-overview","text":"This section provides detailed information regarding the specific features of Lightbus . A grasp of the tutorial and explantion sections will be useful here.","title":"Reference overview"},{"location":"reference/apis/","text":"APIs specify the functionality available on the bus. To do this you define API classes within your bus.py file. You can also define your API elsewhere and import it into your bus.py file. For further discussion of APIs see the concepts section . An example API \u00b6 # An example API. You can define this in your bus.py, # or import into your bus.py file from elsewhere class SupportCaseApi ( Api ): # An event, # available at bus.support.case.case_created case_created = Event ( parameters = ( 'id' , 'sender' , 'subject' , 'body' )) # Options for this API class Meta : # API name on the bus name = 'support.case' # Will be available as a remote procedure call at # bus.support.case.get() def get ( self , id ): return get_case_from_db ( pk = id ) A service can define zero or more APIs, and each API can contain zero or more events and zero or more procedures. The Meta class specifies options regarding the API, with name being the only required option. The name specifies how the API will be accessed on the bus. You could call an RPC on the above API as follows: bus = lightbus . create () # Call the get_case() RPC. case = bus . support . case . get_case ( id = 123 ) You can also fire an event on this API: bus = lightbus . create () # Fire the case_created event bus . support . case . case_created . fire ( id = 123 , sender = 'Joe' , subject = 'I need support please!' , body = '...' , ) Options \u00b6 name (str) \u00b6 Specifies the name of the API. This will determine how the API is addressed on the bus. See naming , below. name is a required option. Naming your APIs \u00b6 As you can from the Meta.name option in the example above, API names can contain periods which allow you to structure your bus in a suitable form for your situation. Some example API naming schemes may look like: # Example API naming schemes for use within Meta.name Format : Example : support.get_case() support.get_activity() Format : . Example : support.case.get() support.activity.get() Format : .. Example : marketing.website.stats.get() ops.monitoring.servers.get_status()","title":"APIs"},{"location":"reference/apis/#an-example-api","text":"# An example API. You can define this in your bus.py, # or import into your bus.py file from elsewhere class SupportCaseApi ( Api ): # An event, # available at bus.support.case.case_created case_created = Event ( parameters = ( 'id' , 'sender' , 'subject' , 'body' )) # Options for this API class Meta : # API name on the bus name = 'support.case' # Will be available as a remote procedure call at # bus.support.case.get() def get ( self , id ): return get_case_from_db ( pk = id ) A service can define zero or more APIs, and each API can contain zero or more events and zero or more procedures. The Meta class specifies options regarding the API, with name being the only required option. The name specifies how the API will be accessed on the bus. You could call an RPC on the above API as follows: bus = lightbus . create () # Call the get_case() RPC. case = bus . support . case . get_case ( id = 123 ) You can also fire an event on this API: bus = lightbus . create () # Fire the case_created event bus . support . case . case_created . fire ( id = 123 , sender = 'Joe' , subject = 'I need support please!' , body = '...' , )","title":"An example API"},{"location":"reference/apis/#options","text":"","title":"Options"},{"location":"reference/apis/#name-str","text":"Specifies the name of the API. This will determine how the API is addressed on the bus. See naming , below. name is a required option.","title":"name (str)"},{"location":"reference/apis/#naming-your-apis","text":"As you can from the Meta.name option in the example above, API names can contain periods which allow you to structure your bus in a suitable form for your situation. Some example API naming schemes may look like: # Example API naming schemes for use within Meta.name Format : Example : support.get_case() support.get_activity() Format : . Example : support.case.get() support.activity.get() Format : .. Example : marketing.website.stats.get() ops.monitoring.servers.get_status()","title":"Naming your APIs"},{"location":"reference/authors/","text":"Authors \u00b6 Current team \u00b6 Adam Charnock , Portugal (Maintainer) Alumni \u00b6 None yet... Thanks \u00b6 Lightbus would not have been possible without the intellectual, emotional, and logistical support of the following individuals and companies: Louis Thibault \u2013 For helping Adam work through many of the knottier problems, and relentlessly cheering him on. Futurepump \u2013 For being an early testing ground and ongoing user of Lightbus. Louis Pilfold \u2013 For being an early sounding board, and persuading Adam to take schemas seriously. Presscast \u2013 For being an early adopter and proving invaluable feedback. Additional thanks to everyone (technical and non-technical) who kindly listened to Adam talk about the reasons and ideas behind Lightbus.","title":"Authors"},{"location":"reference/authors/#authors","text":"","title":"Authors"},{"location":"reference/authors/#current-team","text":"Adam Charnock , Portugal (Maintainer)","title":"Current team"},{"location":"reference/authors/#alumni","text":"None yet...","title":"Alumni"},{"location":"reference/authors/#thanks","text":"Lightbus would not have been possible without the intellectual, emotional, and logistical support of the following individuals and companies: Louis Thibault \u2013 For helping Adam work through many of the knottier problems, and relentlessly cheering him on. Futurepump \u2013 For being an early testing ground and ongoing user of Lightbus. Louis Pilfold \u2013 For being an early sounding board, and persuading Adam to take schemas seriously. Presscast \u2013 For being an early adopter and proving invaluable feedback. Additional thanks to everyone (technical and non-technical) who kindly listened to Adam talk about the reasons and ideas behind Lightbus.","title":"Thanks"},{"location":"reference/code-of-conduct/","text":"Code of Conduct \u00b6 Our Pledge \u00b6 We as members, contributors, and leaders pledge to make participation in our community a harassment-free experience for everyone, regardless of age, body size, visible or invisible disability, ethnicity, sex characteristics, gender identity and expression, level of experience, education, socio-economic status, nationality, personal appearance, race, religion, or sexual identity and orientation. We pledge to act and interact in ways that contribute to an open, welcoming, diverse, inclusive, and healthy community. Our Standards \u00b6 Examples of behavior that contributes to a positive environment for our community include: Demonstrating empathy and kindness toward other people Being respectful of differing opinions, viewpoints, and experiences Giving and gracefully accepting constructive feedback Accepting responsibility and apologizing to those affected by our mistakes, and learning from the experience Focusing on what is best not just for us as individuals, but for the overall community Examples of unacceptable behavior include: The use of sexualized language or imagery, and sexual attention or advances of any kind Trolling, insulting or derogatory comments, and personal or political attacks Public or private harassment Publishing others' private information, such as a physical or email address, without their explicit permission Other conduct which could reasonably be considered inappropriate in a professional setting Enforcement Responsibilities \u00b6 Community leaders are responsible for clarifying and enforcing our standards of acceptable behavior and will take appropriate and fair corrective action in response to any behavior that they deem inappropriate, threatening, offensive, or harmful. Community leaders have the right and responsibility to remove, edit, or reject comments, commits, code, wiki edits, issues, and other contributions that are not aligned to this Code of Conduct, and will communicate reasons for moderation decisions when appropriate. Scope \u00b6 This Code of Conduct applies within all community spaces, and also applies when an individual is officially representing the community in public spaces. Examples of representing our community include using an official e-mail address, posting via an official social media account, or acting as an appointed representative at an online or offline event. Enforcement \u00b6 Instances of abusive, harassing, or otherwise unacceptable behavior may be reported to the community leaders responsible for enforcement at adam@adamcharnock.com . All complaints will be reviewed and investigated promptly and fairly. All community leaders are obligated to respect the privacy and security of the reporter of any incident. Enforcement Guidelines \u00b6 Community leaders will follow these Community Impact Guidelines in determining the consequences for any action they deem in violation of this Code of Conduct: 1. Correction \u00b6 Community Impact : Use of inappropriate language or other behavior deemed unprofessional or unwelcome in the community. Consequence : A private, written warning from community leaders, providing clarity around the nature of the violation and an explanation of why the behavior was inappropriate. A public apology may be requested. 2. Warning \u00b6 Community Impact : A violation through a single incident or series of actions. Consequence : A warning with consequences for continued behavior. No interaction with the people involved, including unsolicited interaction with those enforcing the Code of Conduct, for a specified period of time. This includes avoiding interactions in community spaces as well as external channels like social media. Violating these terms may lead to a temporary or permanent ban. 3. Temporary Ban \u00b6 Community Impact : A serious violation of community standards, including sustained inappropriate behavior. Consequence : A temporary ban from any sort of interaction or public communication with the community for a specified period of time. No public or private interaction with the people involved, including unsolicited interaction with those enforcing the Code of Conduct, is allowed during this period. Violating these terms may lead to a permanent ban. 4. Permanent Ban \u00b6 Community Impact : Demonstrating a pattern of violation of community standards, including sustained inappropriate behavior, harassment of an individual, or aggression toward or disparagement of classes of individuals. Consequence : A permanent ban from any sort of public interaction within the project community. Attribution \u00b6 This Code of Conduct is adapted from the Contributor Covenant , version 2.0, available at https://www.contributor-covenant.org/version/2/0/code_of_conduct.html . Community Impact Guidelines were inspired by Mozilla's code of conduct enforcement ladder . For answers to common questions about this code of conduct, see the FAQ at https://www.contributor-covenant.org/faq . Translations are available at https://www.contributor-covenant.org/translations . For answers to common questions about this code of conduct, see https://www.contributor-covenant.org/faq","title":"Code of Conduct"},{"location":"reference/code-of-conduct/#code-of-conduct","text":"","title":"Code of Conduct"},{"location":"reference/code-of-conduct/#our-pledge","text":"We as members, contributors, and leaders pledge to make participation in our community a harassment-free experience for everyone, regardless of age, body size, visible or invisible disability, ethnicity, sex characteristics, gender identity and expression, level of experience, education, socio-economic status, nationality, personal appearance, race, religion, or sexual identity and orientation. We pledge to act and interact in ways that contribute to an open, welcoming, diverse, inclusive, and healthy community.","title":"Our Pledge"},{"location":"reference/code-of-conduct/#our-standards","text":"Examples of behavior that contributes to a positive environment for our community include: Demonstrating empathy and kindness toward other people Being respectful of differing opinions, viewpoints, and experiences Giving and gracefully accepting constructive feedback Accepting responsibility and apologizing to those affected by our mistakes, and learning from the experience Focusing on what is best not just for us as individuals, but for the overall community Examples of unacceptable behavior include: The use of sexualized language or imagery, and sexual attention or advances of any kind Trolling, insulting or derogatory comments, and personal or political attacks Public or private harassment Publishing others' private information, such as a physical or email address, without their explicit permission Other conduct which could reasonably be considered inappropriate in a professional setting","title":"Our Standards"},{"location":"reference/code-of-conduct/#enforcement-responsibilities","text":"Community leaders are responsible for clarifying and enforcing our standards of acceptable behavior and will take appropriate and fair corrective action in response to any behavior that they deem inappropriate, threatening, offensive, or harmful. Community leaders have the right and responsibility to remove, edit, or reject comments, commits, code, wiki edits, issues, and other contributions that are not aligned to this Code of Conduct, and will communicate reasons for moderation decisions when appropriate.","title":"Enforcement Responsibilities"},{"location":"reference/code-of-conduct/#scope","text":"This Code of Conduct applies within all community spaces, and also applies when an individual is officially representing the community in public spaces. Examples of representing our community include using an official e-mail address, posting via an official social media account, or acting as an appointed representative at an online or offline event.","title":"Scope"},{"location":"reference/code-of-conduct/#enforcement","text":"Instances of abusive, harassing, or otherwise unacceptable behavior may be reported to the community leaders responsible for enforcement at adam@adamcharnock.com . All complaints will be reviewed and investigated promptly and fairly. All community leaders are obligated to respect the privacy and security of the reporter of any incident.","title":"Enforcement"},{"location":"reference/code-of-conduct/#enforcement-guidelines","text":"Community leaders will follow these Community Impact Guidelines in determining the consequences for any action they deem in violation of this Code of Conduct:","title":"Enforcement Guidelines"},{"location":"reference/code-of-conduct/#1-correction","text":"Community Impact : Use of inappropriate language or other behavior deemed unprofessional or unwelcome in the community. Consequence : A private, written warning from community leaders, providing clarity around the nature of the violation and an explanation of why the behavior was inappropriate. A public apology may be requested.","title":"1. Correction"},{"location":"reference/code-of-conduct/#2-warning","text":"Community Impact : A violation through a single incident or series of actions. Consequence : A warning with consequences for continued behavior. No interaction with the people involved, including unsolicited interaction with those enforcing the Code of Conduct, for a specified period of time. This includes avoiding interactions in community spaces as well as external channels like social media. Violating these terms may lead to a temporary or permanent ban.","title":"2. Warning"},{"location":"reference/code-of-conduct/#3-temporary-ban","text":"Community Impact : A serious violation of community standards, including sustained inappropriate behavior. Consequence : A temporary ban from any sort of interaction or public communication with the community for a specified period of time. No public or private interaction with the people involved, including unsolicited interaction with those enforcing the Code of Conduct, is allowed during this period. Violating these terms may lead to a permanent ban.","title":"3. Temporary Ban"},{"location":"reference/code-of-conduct/#4-permanent-ban","text":"Community Impact : Demonstrating a pattern of violation of community standards, including sustained inappropriate behavior, harassment of an individual, or aggression toward or disparagement of classes of individuals. Consequence : A permanent ban from any sort of public interaction within the project community.","title":"4. Permanent Ban"},{"location":"reference/code-of-conduct/#attribution","text":"This Code of Conduct is adapted from the Contributor Covenant , version 2.0, available at https://www.contributor-covenant.org/version/2/0/code_of_conduct.html . Community Impact Guidelines were inspired by Mozilla's code of conduct enforcement ladder . For answers to common questions about this code of conduct, see the FAQ at https://www.contributor-covenant.org/faq . Translations are available at https://www.contributor-covenant.org/translations . For answers to common questions about this code of conduct, see https://www.contributor-covenant.org/faq","title":"Attribution"},{"location":"reference/configuration/","text":"Configuration \u00b6 Lightbus' configuration happens in three stages: Module loading \u2013 Lightbus discovers where your bus.py file can found via the LIGHTBUS_MODULE environment variable. Service-level configuration \u2013 Your bus.py file specifies service-level settings ( service_name and process_name ) Global bus configuration \u2013 Your bus.py provides the location to the global config for your bus. This can be a local file path, or a HTTP(S) URL. See the configuration explanation for further discussion and reasoning around this approach. Configuration auto-complete using JSON Schema Many code editors support using a JSON schema to provide auto-complete and validation when editing a JSON file. If you wish, you can write your configuration in JSON (rather than YAML), and load the following JSON schema into your editor: https://lightbus.org/static/default-config-schema.json This will provide you with autocomplete and validation for Lightbus' various configuration options. If you are using custom transports or plugins you should generate your own config schema . 1. Module loading \u00b6 The first stage in Lightbus' startup is to import your bus.py module. By default Lightbus will attempt to import a module named bus , but you can modify this by specifying the LIGHTBUS_MODULE environment variable. This stage is only required when starting a Lightbus worker process (i.e. lightbus run ). Non-lightbus processes will import the bus module manually in order to access the bus client within (see next stage, below). Note See anatomy lesson for further discusison of the distinction between processes. 2. Service-level configuration \u00b6 The bus module discovered in the module loading stage ( above ) must define a Lightbus client as follows: # Must be in your bus.py file bus = lightbus . create () The above statement serves several purposes: The lightbus run command will use this client to access the bus. You can (and should) import this client elsewhere in the service in order to call RPCs and fire events (see how to access your bus client ). You can configure service-level configuration options for your Lightbus client. Service-level configuration is different to your global configuration because the values will vary between services. The following service-level options are available: # Available service-level configuration options bus = lightbus . create ( # Relevant to event consumption service_name = 'my_service' , # Will be replaced 4 random characters. Default process_name = ' {random4} ' , # Path to the global bus config. # Can be .yaml or .json, and http(s) URLs are supported. config = '/path/to/lightbus.yaml' , # Features to enable. Default is to enable all features. # Can be configured using the --skip and --only arguments features = [ 'rpcs' , 'events' , 'tasks' ], ) The above configuration options can also be set using the following environment variables or command line arguments: Configuration option Environment Variable Command line argument Notes Service name LIGHTBUS_SERVICE_NAME --service-name See service name explanation Process name LIGHTBUS_PROCESS_NAME --process-name See process name explanation Configuration path LIGHTBUS_CONFIG --config Path or URL to global configuration yaml/json file. See global bus configuration . Features LIGHTBUS_FEATURES --only , --skip Features to enable/disable. Comma separated list of rpcs , events , tasks Service & process name placeholders \u00b6 The following placeholders may be used within your service and process name values: Paceholder Example value Notes {hostname} my-host Lower case hostname {pid} 12345 The process' ID {random4} abcd Random 4-character string {random8} abcdefgh Random 8-character string {random16} abcdefghijklmnop Random 16-character string {friendly} delicate-wave-915 Human-friendly random name Event delivery \u00b6 See the events explanation section for a discussion on how service & process names affect event delivery. 3. Global bus configuration \u00b6 The global bus configuration specifies the bus' overall architecture. This configuration takes the form of a YAML or JSON file, and accordingly the filename should end with .yaml or .json . This file is typically shared by all lightbus clients and can be specified as a path on disk, or as a HTTP(S) URL ( see explanation ). A basic default configuration file is as follows: # Root configuration bus : # Bus configuration schema : # Schema configuration transport : # Transport selector config redis : url : \"redis://redis.svc.cluster.local:6379/0\" apis : # API configuration listing default : # Api config event_transport : # Transport selector configuration redis : url : \"redis://redis.svc.cluster.local:6379/0\" rpc_transport : # Transport selector configuration redis : url : \"redis://redis.svc.cluster.local:6379/0\" result_transport : # Transport selector configuration redis : url : \"redis://redis.svc.cluster.local:6379/0\" plugins : # Plugin configuring listing # These plugins ship with Lightbus internal_metrics : # Plugin configuration # Note that plugins are disabled by default, and must be enabled here # in your bus' global configuration enabled : true internal_state : enabled : true ping_enabled : true ping_interval : 60 Each section is detailed below. Root config \u00b6 The available root-level configuration keys are: bus \u2013 Contains the bus config apis \u2013 Contains the API configuration listing plugins - Contains the plugin configuration listing The following keys are also present, but should generally not be specified in your global yaml file . Instead they should be specified for each service, as per the service-level setup : service_name \u2013 Service name process_name \u2013 Process name Bus config \u00b6 The bus config resides under the root config . It contains the following keys: log_level (default: info ) - The log level for the lightbus logger. One of debug , info , warning , error , critical . info is a good level for development purposes, warning will be more suited to production. schema - Contains the schema config Schema config \u00b6 The schema config resides under the bus config . human_readable (default: True ) \u2013 Should the schema JSON be transmitted with human-friendly indentation and spacing? ttl (default: 60 ) \u2013 Integer number of seconds that an API schema should live on the bus. Each schema will be pinged every ttl * 0.8 seconds in order to keep it alive. The bus will also check for new remote schemas every ttl * 0.8 seconds. transport \u2013 Contains the schema transport selector API configuration listing \u00b6 The API configuration listing config resides under the root config . This is a key/value association between APIs and their configurations. The reserved API name of default provides a catch-all configuration for any APIs without specific configurations. Specifically configured APIs do not inherit from the default configuration. For example: ... apis : # Catch-all api config default : # See 'API config' ... default config as above ... # Specific config for the 'marketing.analytics' API. # Use a different Redis instance for the high # volume marketing analytics marketing.analytics : # See 'API config' validate : false cast_values : false event_transport : redis : url : \"redis://redis-marketing.svc.cluster.local:6379/0\" rpc_transport : redis : url : \"redis://redis-marketing.svc.cluster.local:6379/0\" result_transport : redis : url : \"redis://redis-marketing.svc.cluster.local:6379/0\" See API config for further details on the API options available. API config \u00b6 The API config resides under the API configuration listing . APIs are configured using the options below: rpc_timeout (default: 5 ) \u2013 Timeout when calling RPCs on this API (must also be specified on the RPC and result transport) event_listener_setup_timeout (default: 1 ) \u2013 Timeout seconds when setting up event listeners (only applies when using the blocking api) event_fire_timeout (default: 1 ) \u2013 Timeout seconds when firing events on the bus (only applies when using the blocking api) validate \u2013 Contains the api validation config . May also be set to boolean true or false to blanket enable/disable. strict_validation (default: false ) \u2013 Raise an exception if we receive a message from an API for which there is no schema available on the bus. If false a warning will be emitted instead. event_transport \u2013 Contains the transport selector . rpc_transport \u2013 Contains the transport selector . result_transport \u2013 Contains the transport selector . cast_values (default: true ) \u2013 If enabled, incoming values will be best-effort casted based on the annotations of the RPC method signature or event listener. See typing . Transport selector \u00b6 The schema config resides under both the API config and the schema config . Transports are specified as follows: ... parent yaml ... [transport-name]: option: \"value\" option: \"value\" Where [transport-name] can be one of: redis \u2013 The redis-backed transport. debug \u2013 A debug transport which logs what happens but takes no further action. The table below details which transports can be used in which situations Transport RPC Result Event Schema redis \u2714 \u2714 \u2714 \u2714 debug \u2714 \u2714 \u2714 \u2714 Additional transports may be added in future. A single API can use different types for each of its rpc , result , and event needs. The schema transport is global to the bus, and is not configurable on a per-api level. For configuration details see the transport configuration reference . API validation config \u00b6 The schema config resides under the validate key within the API config . Available options are: outgoing (default true ) \u2013 Validate outgoing messages against any available API schema. incoming (default true ) \u2013 Validate incoming messages against any available API schema. A warning will be emitted if validation is enabled and the schema is not present on the bus. You can turn this into an error by enabling strict_validation within the API config . Plugin configuration listing \u00b6 The plugin configuration listing config resides under the root config . This configuration section is a mapping between plugin names and their configuration options. Plugins are made available to Lightbus via the lightbus_plugins entry point (see Lightbus' pyproject.toml for an example). As a result, installing a plugin should be sufficient for it to be made available for configuration. Plugin developers should see the plugins reference for further details. Plugin configuration \u00b6 The Plugin configuration resides under the Plugin configuration listing . This section provides the configuration for an individual plugin. The available configuration options will vary from plugin to plugin, so you should read the plugin's documentation for details. However, all plugins support the enabled property, which defaults to false . You must there set enabled: true if you wish to use the plugin. For example: bus : ... apis : ... plugins : internal_metrics : # Plugin configuration enabled : true","title":"Configuration"},{"location":"reference/configuration/#configuration","text":"Lightbus' configuration happens in three stages: Module loading \u2013 Lightbus discovers where your bus.py file can found via the LIGHTBUS_MODULE environment variable. Service-level configuration \u2013 Your bus.py file specifies service-level settings ( service_name and process_name ) Global bus configuration \u2013 Your bus.py provides the location to the global config for your bus. This can be a local file path, or a HTTP(S) URL. See the configuration explanation for further discussion and reasoning around this approach. Configuration auto-complete using JSON Schema Many code editors support using a JSON schema to provide auto-complete and validation when editing a JSON file. If you wish, you can write your configuration in JSON (rather than YAML), and load the following JSON schema into your editor: https://lightbus.org/static/default-config-schema.json This will provide you with autocomplete and validation for Lightbus' various configuration options. If you are using custom transports or plugins you should generate your own config schema .","title":"Configuration"},{"location":"reference/configuration/#1-module-loading","text":"The first stage in Lightbus' startup is to import your bus.py module. By default Lightbus will attempt to import a module named bus , but you can modify this by specifying the LIGHTBUS_MODULE environment variable. This stage is only required when starting a Lightbus worker process (i.e. lightbus run ). Non-lightbus processes will import the bus module manually in order to access the bus client within (see next stage, below). Note See anatomy lesson for further discusison of the distinction between processes.","title":"1. Module loading"},{"location":"reference/configuration/#2-service-level-configuration","text":"The bus module discovered in the module loading stage ( above ) must define a Lightbus client as follows: # Must be in your bus.py file bus = lightbus . create () The above statement serves several purposes: The lightbus run command will use this client to access the bus. You can (and should) import this client elsewhere in the service in order to call RPCs and fire events (see how to access your bus client ). You can configure service-level configuration options for your Lightbus client. Service-level configuration is different to your global configuration because the values will vary between services. The following service-level options are available: # Available service-level configuration options bus = lightbus . create ( # Relevant to event consumption service_name = 'my_service' , # Will be replaced 4 random characters. Default process_name = ' {random4} ' , # Path to the global bus config. # Can be .yaml or .json, and http(s) URLs are supported. config = '/path/to/lightbus.yaml' , # Features to enable. Default is to enable all features. # Can be configured using the --skip and --only arguments features = [ 'rpcs' , 'events' , 'tasks' ], ) The above configuration options can also be set using the following environment variables or command line arguments: Configuration option Environment Variable Command line argument Notes Service name LIGHTBUS_SERVICE_NAME --service-name See service name explanation Process name LIGHTBUS_PROCESS_NAME --process-name See process name explanation Configuration path LIGHTBUS_CONFIG --config Path or URL to global configuration yaml/json file. See global bus configuration . Features LIGHTBUS_FEATURES --only , --skip Features to enable/disable. Comma separated list of rpcs , events , tasks","title":"2. Service-level configuration"},{"location":"reference/configuration/#service-process-name-placeholders","text":"The following placeholders may be used within your service and process name values: Paceholder Example value Notes {hostname} my-host Lower case hostname {pid} 12345 The process' ID {random4} abcd Random 4-character string {random8} abcdefgh Random 8-character string {random16} abcdefghijklmnop Random 16-character string {friendly} delicate-wave-915 Human-friendly random name","title":"Service & process name placeholders"},{"location":"reference/configuration/#event-delivery","text":"See the events explanation section for a discussion on how service & process names affect event delivery.","title":"Event delivery"},{"location":"reference/configuration/#3-global-bus-configuration","text":"The global bus configuration specifies the bus' overall architecture. This configuration takes the form of a YAML or JSON file, and accordingly the filename should end with .yaml or .json . This file is typically shared by all lightbus clients and can be specified as a path on disk, or as a HTTP(S) URL ( see explanation ). A basic default configuration file is as follows: # Root configuration bus : # Bus configuration schema : # Schema configuration transport : # Transport selector config redis : url : \"redis://redis.svc.cluster.local:6379/0\" apis : # API configuration listing default : # Api config event_transport : # Transport selector configuration redis : url : \"redis://redis.svc.cluster.local:6379/0\" rpc_transport : # Transport selector configuration redis : url : \"redis://redis.svc.cluster.local:6379/0\" result_transport : # Transport selector configuration redis : url : \"redis://redis.svc.cluster.local:6379/0\" plugins : # Plugin configuring listing # These plugins ship with Lightbus internal_metrics : # Plugin configuration # Note that plugins are disabled by default, and must be enabled here # in your bus' global configuration enabled : true internal_state : enabled : true ping_enabled : true ping_interval : 60 Each section is detailed below.","title":"3. Global bus configuration"},{"location":"reference/configuration/#root-config","text":"The available root-level configuration keys are: bus \u2013 Contains the bus config apis \u2013 Contains the API configuration listing plugins - Contains the plugin configuration listing The following keys are also present, but should generally not be specified in your global yaml file . Instead they should be specified for each service, as per the service-level setup : service_name \u2013 Service name process_name \u2013 Process name","title":"Root config"},{"location":"reference/configuration/#bus-config","text":"The bus config resides under the root config . It contains the following keys: log_level (default: info ) - The log level for the lightbus logger. One of debug , info , warning , error , critical . info is a good level for development purposes, warning will be more suited to production. schema - Contains the schema config","title":"Bus config"},{"location":"reference/configuration/#schema-config","text":"The schema config resides under the bus config . human_readable (default: True ) \u2013 Should the schema JSON be transmitted with human-friendly indentation and spacing? ttl (default: 60 ) \u2013 Integer number of seconds that an API schema should live on the bus. Each schema will be pinged every ttl * 0.8 seconds in order to keep it alive. The bus will also check for new remote schemas every ttl * 0.8 seconds. transport \u2013 Contains the schema transport selector","title":"Schema config"},{"location":"reference/configuration/#api-configuration-listing","text":"The API configuration listing config resides under the root config . This is a key/value association between APIs and their configurations. The reserved API name of default provides a catch-all configuration for any APIs without specific configurations. Specifically configured APIs do not inherit from the default configuration. For example: ... apis : # Catch-all api config default : # See 'API config' ... default config as above ... # Specific config for the 'marketing.analytics' API. # Use a different Redis instance for the high # volume marketing analytics marketing.analytics : # See 'API config' validate : false cast_values : false event_transport : redis : url : \"redis://redis-marketing.svc.cluster.local:6379/0\" rpc_transport : redis : url : \"redis://redis-marketing.svc.cluster.local:6379/0\" result_transport : redis : url : \"redis://redis-marketing.svc.cluster.local:6379/0\" See API config for further details on the API options available.","title":"API configuration listing"},{"location":"reference/configuration/#api-config","text":"The API config resides under the API configuration listing . APIs are configured using the options below: rpc_timeout (default: 5 ) \u2013 Timeout when calling RPCs on this API (must also be specified on the RPC and result transport) event_listener_setup_timeout (default: 1 ) \u2013 Timeout seconds when setting up event listeners (only applies when using the blocking api) event_fire_timeout (default: 1 ) \u2013 Timeout seconds when firing events on the bus (only applies when using the blocking api) validate \u2013 Contains the api validation config . May also be set to boolean true or false to blanket enable/disable. strict_validation (default: false ) \u2013 Raise an exception if we receive a message from an API for which there is no schema available on the bus. If false a warning will be emitted instead. event_transport \u2013 Contains the transport selector . rpc_transport \u2013 Contains the transport selector . result_transport \u2013 Contains the transport selector . cast_values (default: true ) \u2013 If enabled, incoming values will be best-effort casted based on the annotations of the RPC method signature or event listener. See typing .","title":"API config"},{"location":"reference/configuration/#transport-selector","text":"The schema config resides under both the API config and the schema config . Transports are specified as follows: ... parent yaml ... [transport-name]: option: \"value\" option: \"value\" Where [transport-name] can be one of: redis \u2013 The redis-backed transport. debug \u2013 A debug transport which logs what happens but takes no further action. The table below details which transports can be used in which situations Transport RPC Result Event Schema redis \u2714 \u2714 \u2714 \u2714 debug \u2714 \u2714 \u2714 \u2714 Additional transports may be added in future. A single API can use different types for each of its rpc , result , and event needs. The schema transport is global to the bus, and is not configurable on a per-api level. For configuration details see the transport configuration reference .","title":"Transport selector"},{"location":"reference/configuration/#api-validation-config","text":"The schema config resides under the validate key within the API config . Available options are: outgoing (default true ) \u2013 Validate outgoing messages against any available API schema. incoming (default true ) \u2013 Validate incoming messages against any available API schema. A warning will be emitted if validation is enabled and the schema is not present on the bus. You can turn this into an error by enabling strict_validation within the API config .","title":"API validation config"},{"location":"reference/configuration/#plugin-configuration-listing","text":"The plugin configuration listing config resides under the root config . This configuration section is a mapping between plugin names and their configuration options. Plugins are made available to Lightbus via the lightbus_plugins entry point (see Lightbus' pyproject.toml for an example). As a result, installing a plugin should be sufficient for it to be made available for configuration. Plugin developers should see the plugins reference for further details.","title":"Plugin configuration listing"},{"location":"reference/configuration/#plugin-configuration","text":"The Plugin configuration resides under the Plugin configuration listing . This section provides the configuration for an individual plugin. The available configuration options will vary from plugin to plugin, so you should read the plugin's documentation for details. However, all plugins support the enabled property, which defaults to false . You must there set enabled: true if you wish to use the plugin. For example: bus : ... apis : ... plugins : internal_metrics : # Plugin configuration enabled : true","title":"Plugin configuration"},{"location":"reference/events/","text":"Events are defined as properties on your API classes. Events are useful when: You need asynchronous communication between services You wish to loosely-couple your services You need a service to perform a task in the background See event considerations in the explanations section for further discussion. Events provide at-least-once delivery semantics. Given this, your event handlers should be idempotent . Defining events \u00b6 You can define events using the Event class. For example, you could define the following bus.py in your authenication service: # auth_service/bus.py from lightbus import Api , Event class AuthApi ( Api ): user_created = Event ( parameters = ( 'username' , 'email' )) user_updated = Event ( parameters = ( 'username' , 'new_email' )) user_deleted = Event ( parameters = ( 'username' )) class Meta : name = 'auth' Firing events \u00b6 You can fire events as follows: # Anywhere in your code # Import your project's bus instance from bus import bus bus . auth . user_created . fire ( username = 'adam' , password = 'adam@example.com' ) Firing events (asynchronously) \u00b6 You can also fire events asynchronously using asyncio: # Anywhere in your code # Import your project's bus instance from bus import bus await bus . auth . user_created . fire_async ( username = 'adam' , password = 'adam@example.com' ) Listening for events \u00b6 Listening for events is typically a long-running background activity, and is therefore dealt with by the lightbus run command. You can setup event listeners in your services' bus module as follows: # other_service/bus.py import lightbus bus = lightbus . create () user_db = {} def handle_created ( username , email ): user_db [ username ] = email print ( user_db ) def handle_updated ( username , email ): user_db [ username ] = email print ( user_db ) def handle_deleted ( username , email ): user_db . pop ( username ) print ( user_db ) @bus . client . on_start () def on_start (): # Bus client has started up, so register our listeners bus . auth . user_created . listen ( handle_created , listener_name = \"user_created\" ) bus . auth . user_updated . listen ( handle_updated , listener_name = \"user_updated\" , ) bus . auth . user_deleted . listen ( handle_deleted , listener_name = \"user_deleted\" ) Specifying listener_name is required in order to ensure each listeners receives all events it is due. See the events explanation page for further discussion. You cannot have two listeners with the same name for the same API. For example, this will not work (it will raise a DuplicateListenerName exception: bus . competitor_prices . changed . listen ( send_price_alerts , listener_name = \"price_listener\" , ) ### ERROR ## # This will raise a DuplicateListenerName exception because # we have already created a listener named 'price_listener' # on the 'competitor_prices' API (above). bus . competitor_prices . changed . listen ( update_prices , listener_name = \"price_listener\" , # \u21e0 DuplicateListenerName exception ) This restriction only applies to listeners within the same service. Errors in event listeners \u00b6 By default, the Lightbus worker will exit if an event listener encounters an error. The event being handled will remain unacknowledged, and will be re-attempted after acknowledgement_timeout seconds. You can customise this behaviour by passing the on_error parameter to listen() . Possible values are: lightbus.OnError.SHUTDOWN (Default) - Shutdown the Lightbus work upon encountering an error lightbus.OnError.ACKNOWLEDGE_AND_LOG (Default) - Acknowledge erroring messages as being successfully processed, and log the error. For example: # Taken from the above example # Worker shutdown on error (the default) bus . auth . user_created . listen ( handle_created , listener_name = \"user_created\" , on_error = lightbus . OnError . SHUTDOWN , ) # Or, log the error and acknowlegde the message as being processed bus . auth . user_created . listen ( handle_created , listener_name = \"user_created\" , on_error = lightbus . OnError . ACKNOWLEDGE_AND_LOG , ) Listening for events (asynchronous handlers) \u00b6 Event handlers may also be asynchronous. For example: import lightbus import asyncio bus = lightbus . create () # Asynchronous handles can be used too async def handle_created ( username , email ): await asyncio . sleep ( 0.1 ) print ( f \"Username { username } created\" ) @bus . client . on_start () def on_start (): # Bus client has started up, so register our listeners bus . auth . user_created . listen ( handle_created , listener_name = \"user_created\" ) A note on threads Asynchronous handlers will be executed in the same thread as Lightbus, whereas synchronous handlers will be executed in their own thread. Type hints \u00b6 See the typing reference .","title":"Events"},{"location":"reference/events/#defining-events","text":"You can define events using the Event class. For example, you could define the following bus.py in your authenication service: # auth_service/bus.py from lightbus import Api , Event class AuthApi ( Api ): user_created = Event ( parameters = ( 'username' , 'email' )) user_updated = Event ( parameters = ( 'username' , 'new_email' )) user_deleted = Event ( parameters = ( 'username' )) class Meta : name = 'auth'","title":"Defining events"},{"location":"reference/events/#firing-events","text":"You can fire events as follows: # Anywhere in your code # Import your project's bus instance from bus import bus bus . auth . user_created . fire ( username = 'adam' , password = 'adam@example.com' )","title":"Firing events"},{"location":"reference/events/#firing-events-asynchronously","text":"You can also fire events asynchronously using asyncio: # Anywhere in your code # Import your project's bus instance from bus import bus await bus . auth . user_created . fire_async ( username = 'adam' , password = 'adam@example.com' )","title":"Firing events (asynchronously)"},{"location":"reference/events/#listening-for-events","text":"Listening for events is typically a long-running background activity, and is therefore dealt with by the lightbus run command. You can setup event listeners in your services' bus module as follows: # other_service/bus.py import lightbus bus = lightbus . create () user_db = {} def handle_created ( username , email ): user_db [ username ] = email print ( user_db ) def handle_updated ( username , email ): user_db [ username ] = email print ( user_db ) def handle_deleted ( username , email ): user_db . pop ( username ) print ( user_db ) @bus . client . on_start () def on_start (): # Bus client has started up, so register our listeners bus . auth . user_created . listen ( handle_created , listener_name = \"user_created\" ) bus . auth . user_updated . listen ( handle_updated , listener_name = \"user_updated\" , ) bus . auth . user_deleted . listen ( handle_deleted , listener_name = \"user_deleted\" ) Specifying listener_name is required in order to ensure each listeners receives all events it is due. See the events explanation page for further discussion. You cannot have two listeners with the same name for the same API. For example, this will not work (it will raise a DuplicateListenerName exception: bus . competitor_prices . changed . listen ( send_price_alerts , listener_name = \"price_listener\" , ) ### ERROR ## # This will raise a DuplicateListenerName exception because # we have already created a listener named 'price_listener' # on the 'competitor_prices' API (above). bus . competitor_prices . changed . listen ( update_prices , listener_name = \"price_listener\" , # \u21e0 DuplicateListenerName exception ) This restriction only applies to listeners within the same service.","title":"Listening for events"},{"location":"reference/events/#errors-in-event-listeners","text":"By default, the Lightbus worker will exit if an event listener encounters an error. The event being handled will remain unacknowledged, and will be re-attempted after acknowledgement_timeout seconds. You can customise this behaviour by passing the on_error parameter to listen() . Possible values are: lightbus.OnError.SHUTDOWN (Default) - Shutdown the Lightbus work upon encountering an error lightbus.OnError.ACKNOWLEDGE_AND_LOG (Default) - Acknowledge erroring messages as being successfully processed, and log the error. For example: # Taken from the above example # Worker shutdown on error (the default) bus . auth . user_created . listen ( handle_created , listener_name = \"user_created\" , on_error = lightbus . OnError . SHUTDOWN , ) # Or, log the error and acknowlegde the message as being processed bus . auth . user_created . listen ( handle_created , listener_name = \"user_created\" , on_error = lightbus . OnError . ACKNOWLEDGE_AND_LOG , )","title":"Errors in event listeners"},{"location":"reference/events/#listening-for-events-asynchronous-handlers","text":"Event handlers may also be asynchronous. For example: import lightbus import asyncio bus = lightbus . create () # Asynchronous handles can be used too async def handle_created ( username , email ): await asyncio . sleep ( 0.1 ) print ( f \"Username { username } created\" ) @bus . client . on_start () def on_start (): # Bus client has started up, so register our listeners bus . auth . user_created . listen ( handle_created , listener_name = \"user_created\" ) A note on threads Asynchronous handlers will be executed in the same thread as Lightbus, whereas synchronous handlers will be executed in their own thread.","title":"Listening for events (asynchronous handlers)"},{"location":"reference/events/#type-hints","text":"See the typing reference .","title":"Type hints"},{"location":"reference/plugins-development/","text":"Plugins provide hooks into Lightbus' inner workings. For example, the bundled StatePlugin hooks into the before_worker_start and after_worker_stopped hooks. The plugin uses these hooks to bus events indicating the state of the worker. A internal.state.worker_started event indicates a worker has started, and a internal.state.worker_stopped event indicates a worker has stopped. Consuming these events will provide a picture of the current state of workers on the bus. Example plugin \u00b6 Let's create a simple plugin which will print a configurable message to the console every time a message is sent. This requires three steps: Define the plugin Make Lightbus aware of the plugin Configure the plugin 1. Define the plugin \u00b6 Create the following in a python module (we'll assume it is at my_project.greeting_plugin ): from lightbus.plugins import LightbusPlugin class GreetingPlugin ( LightbusPlugin ): \"\"\"Print a greeting after every event it sent\"\"\" priority = 200 def __init__ ( self , greeting : str ): self . greeting = greeting @classmethod def from_config ( cls , config : \"Config\" , greeting : str = \"Hello world!\" ): # The arguments to this method define the configuration options which # can be set in the bus' yaml configuration (see below) return cls ( greeting = greeting ) async def after_event_sent ( self , * , event_message , client ): # Print a simple greeting after an event is sent print ( self . greeting ) 2. Make Lightbus aware of the plugin \u00b6 Lightbus is made aware of the plugin via an entrypoint in your project's pyproject.toml file: # Your pyproject.toml file ... [tool.poetry.plugins.lightbus_plugins] # `greeting_plugin` defines the plugin name in the config # `my_project.greeting_plugin` is your plugin's python module # `GreetingPlugin` is your plugins class name greeting_plugin = \"my_project.greeting_plugin:GreetingPlugin\" Once you've made this change (or for subsequent modifications) you will need to run: pip install . This will setup the entry_points you have specified 3. Configure the plugin \u00b6 You can configure your plugin in your bus' configuration YAML file (see the configuration reference ). For example: bus : ... apis : ... plugins : greeting_plugin : # The 'enabled' configuration is available for all plugins, # if set to 'false' the plugin will not be loaded. # Default is false enabled : true # Lightbus is aware of our `greeting` option as it reads it # from our `from_config()` method above. # If we omit this it will have the default value of \"Hello world!\" greeting : \"Hello world, I sent an event!\" Plugin hooks/methods \u00b6 The following hooks are available. Each of these should be implemented as an asynchronous method on your plugin class. For full reference see the LightbusPlugin class before_parse_args receive_args before_worker_start after_worker_stopped before_rpc_call after_rpc_call before_rpc_execution after_rpc_execution before_event_sent after_event_sent before_event_execution after_event_execution exception","title":"Plugin development"},{"location":"reference/plugins-development/#example-plugin","text":"Let's create a simple plugin which will print a configurable message to the console every time a message is sent. This requires three steps: Define the plugin Make Lightbus aware of the plugin Configure the plugin","title":"Example plugin"},{"location":"reference/plugins-development/#1-define-the-plugin","text":"Create the following in a python module (we'll assume it is at my_project.greeting_plugin ): from lightbus.plugins import LightbusPlugin class GreetingPlugin ( LightbusPlugin ): \"\"\"Print a greeting after every event it sent\"\"\" priority = 200 def __init__ ( self , greeting : str ): self . greeting = greeting @classmethod def from_config ( cls , config : \"Config\" , greeting : str = \"Hello world!\" ): # The arguments to this method define the configuration options which # can be set in the bus' yaml configuration (see below) return cls ( greeting = greeting ) async def after_event_sent ( self , * , event_message , client ): # Print a simple greeting after an event is sent print ( self . greeting )","title":"1. Define the plugin"},{"location":"reference/plugins-development/#2-make-lightbus-aware-of-the-plugin","text":"Lightbus is made aware of the plugin via an entrypoint in your project's pyproject.toml file: # Your pyproject.toml file ... [tool.poetry.plugins.lightbus_plugins] # `greeting_plugin` defines the plugin name in the config # `my_project.greeting_plugin` is your plugin's python module # `GreetingPlugin` is your plugins class name greeting_plugin = \"my_project.greeting_plugin:GreetingPlugin\" Once you've made this change (or for subsequent modifications) you will need to run: pip install . This will setup the entry_points you have specified","title":"2. Make Lightbus aware of the plugin"},{"location":"reference/plugins-development/#3-configure-the-plugin","text":"You can configure your plugin in your bus' configuration YAML file (see the configuration reference ). For example: bus : ... apis : ... plugins : greeting_plugin : # The 'enabled' configuration is available for all plugins, # if set to 'false' the plugin will not be loaded. # Default is false enabled : true # Lightbus is aware of our `greeting` option as it reads it # from our `from_config()` method above. # If we omit this it will have the default value of \"Hello world!\" greeting : \"Hello world, I sent an event!\"","title":"3. Configure the plugin"},{"location":"reference/plugins-development/#plugin-hooksmethods","text":"The following hooks are available. Each of these should be implemented as an asynchronous method on your plugin class. For full reference see the LightbusPlugin class before_parse_args receive_args before_worker_start after_worker_stopped before_rpc_call after_rpc_call before_rpc_execution after_rpc_execution before_event_sent after_event_sent before_event_execution after_event_execution exception","title":"Plugin hooks/methods"},{"location":"reference/plugins/","text":"Lightbus ships with two plugins, both of which are disabled by default. State Plugin \u00b6 Every Lightbus worker process which has the state plugin enabled will report it's state to the bus. This state information is available as the following events on the internal.state API. This plugin should add minimal load to the bus and may be useful in developing tooling around the bus. Events \u00b6 worker_started \u00b6 Parameters: process_name , metrics_enabled , api_names , listening_for , timestamp , ping_interval Fired when the worker starts up. worker_ping \u00b6 Parameters: process_name , metrics_enabled , api_names , listening_for , timestamp , ping_interval Fires every 60 seconds after worker startup. This indicates that the worker is still alive and has not died unexpectedly. This interval is configurable (see below). worker_stopped \u00b6 Parameters: process_name , timestamp Fires when a worker shuts down cleanly. Configuration \u00b6 The following configuration options are available: enabled (bool) \u00b6 Default: False Should the plugin be enabled? ping_enabled (bool) \u00b6 Default: True Should ping messages be sent? ping_enabled (int, seconds) \u00b6 Default: 60 How often (in seconds) should a ping event be sent. A lower interval means more frequent messages, but reduces the time it takes any listeners to discover dead workers. Example configuration \u00b6 bus : ... apis : ... plugins : internal_state : enabled : true ping_enabled : true ping_interval : 60 Metrics Plugin \u00b6 The metrics plugin sends metric events for every event & RPC processes. It therefore has a much bigger impact on performance than the state plugin, but also provides much more detailed information. Events \u00b6 The following events will be fired on the internal.metrics API: Event Parameters rpc_call_sent process_name , id , api_name , procedure_name , kwargs , timestamp rpc_call_received process_name , id , api_name , procedure_name , timestamp rpc_response_sent process_name , id , api_name , procedure_name , result , timestamp rpc_response_received process_name , id , api_name , procedure_name , timestamp event_fired process_name , event_id , api_name , event_name , kwargs , timestamp event_received process_name , event_id , api_name , event_name , kwargs , timestamp event_processed process_name , event_id , api_name , event_name , kwargs , timestamp Configuration \u00b6 The metrics plugin only includes the enabled configuration option. Configuration should therefore be: bus : ... apis : ... plugins : internal_metrics : enabled : true","title":"Plugins"},{"location":"reference/plugins/#state-plugin","text":"Every Lightbus worker process which has the state plugin enabled will report it's state to the bus. This state information is available as the following events on the internal.state API. This plugin should add minimal load to the bus and may be useful in developing tooling around the bus.","title":"State Plugin"},{"location":"reference/plugins/#events","text":"","title":"Events"},{"location":"reference/plugins/#worker_started","text":"Parameters: process_name , metrics_enabled , api_names , listening_for , timestamp , ping_interval Fired when the worker starts up.","title":"worker_started"},{"location":"reference/plugins/#worker_ping","text":"Parameters: process_name , metrics_enabled , api_names , listening_for , timestamp , ping_interval Fires every 60 seconds after worker startup. This indicates that the worker is still alive and has not died unexpectedly. This interval is configurable (see below).","title":"worker_ping"},{"location":"reference/plugins/#worker_stopped","text":"Parameters: process_name , timestamp Fires when a worker shuts down cleanly.","title":"worker_stopped"},{"location":"reference/plugins/#configuration","text":"The following configuration options are available:","title":"Configuration"},{"location":"reference/plugins/#enabled-bool","text":"Default: False Should the plugin be enabled?","title":"enabled (bool)"},{"location":"reference/plugins/#ping_enabled-bool","text":"Default: True Should ping messages be sent?","title":"ping_enabled (bool)"},{"location":"reference/plugins/#ping_enabled-int-seconds","text":"Default: 60 How often (in seconds) should a ping event be sent. A lower interval means more frequent messages, but reduces the time it takes any listeners to discover dead workers.","title":"ping_enabled (int, seconds)"},{"location":"reference/plugins/#example-configuration","text":"bus : ... apis : ... plugins : internal_state : enabled : true ping_enabled : true ping_interval : 60","title":"Example configuration"},{"location":"reference/plugins/#metrics-plugin","text":"The metrics plugin sends metric events for every event & RPC processes. It therefore has a much bigger impact on performance than the state plugin, but also provides much more detailed information.","title":"Metrics Plugin"},{"location":"reference/plugins/#events_1","text":"The following events will be fired on the internal.metrics API: Event Parameters rpc_call_sent process_name , id , api_name , procedure_name , kwargs , timestamp rpc_call_received process_name , id , api_name , procedure_name , timestamp rpc_response_sent process_name , id , api_name , procedure_name , result , timestamp rpc_response_received process_name , id , api_name , procedure_name , timestamp event_fired process_name , event_id , api_name , event_name , kwargs , timestamp event_received process_name , event_id , api_name , event_name , kwargs , timestamp event_processed process_name , event_id , api_name , event_name , kwargs , timestamp","title":"Events"},{"location":"reference/plugins/#configuration_1","text":"The metrics plugin only includes the enabled configuration option. Configuration should therefore be: bus : ... apis : ... plugins : internal_metrics : enabled : true","title":"Configuration"},{"location":"reference/release-process/","text":"Lightbus release process \u00b6 Lightbus releases are performed as follows: # Ensure poetry.lock is up to date poetry lock # Version bump poetry version { patch,minor,major,prepatch,preminor,premajor,prerelease } export VERSION =( lightbus version --pyproject ) # v1.2.3 export VERSION_DOCS =( lightbus version --pyproject --docs ) # v1.2 # Commit git add . git commit -m \"Releasing version $VERSION \" # Make docs git checkout gh-pages git pull origin gh-pages git checkout master mike deploy v $VERSION_DOCS --message = \"Build docs for release of $VERSION [ci skip]\" mike delete latest mike alias v $VERSION_DOCS latest # Tagging and branching git tag \"v $VERSION \" git branch \"v $VERSION \" git push origin \\ refs/tags/ \"v $VERSION \" \\ refs/heads/ \"v $VERSION \" \\ master \\ gh-pages # Wait for CI to pass: https://circleci.com/gh/adamcharnock/lightbus # Build and publish poetry publish --build","title":"Release process"},{"location":"reference/release-process/#lightbus-release-process","text":"Lightbus releases are performed as follows: # Ensure poetry.lock is up to date poetry lock # Version bump poetry version { patch,minor,major,prepatch,preminor,premajor,prerelease } export VERSION =( lightbus version --pyproject ) # v1.2.3 export VERSION_DOCS =( lightbus version --pyproject --docs ) # v1.2 # Commit git add . git commit -m \"Releasing version $VERSION \" # Make docs git checkout gh-pages git pull origin gh-pages git checkout master mike deploy v $VERSION_DOCS --message = \"Build docs for release of $VERSION [ci skip]\" mike delete latest mike alias v $VERSION_DOCS latest # Tagging and branching git tag \"v $VERSION \" git branch \"v $VERSION \" git push origin \\ refs/tags/ \"v $VERSION \" \\ refs/heads/ \"v $VERSION \" \\ master \\ gh-pages # Wait for CI to pass: https://circleci.com/gh/adamcharnock/lightbus # Build and publish poetry publish --build","title":"Lightbus release process"},{"location":"reference/rpcs/","text":"Remote procedures calls (RPCs) are defined as methods on your API classes. They are useful when either: You require information from a service You wish to wait until a remote procedure has completed an action RPCs provide at-most-once delivery semantics. If you need at-least-once semantics you should consder using events instead. Definition \u00b6 As covered in previous sections, you define RPCs as follows: # bus.py from lightbus import Api class AuthApi ( Api ): class Meta : name = 'auth' def check_password ( self , username , password ): return username == 'admin' and password == 'secret' def reset_password ( self , username ): reset_users_password_somehow ( username ) def get_user ( self , username ): return get_user ( username ) def promote_to_admin ( username ): user = get_user ( username ) user . admin = True user . save () Calling \u00b6 RPCs are called simply as follows: # Anywhere in your code # Import your project's bus instance from bus import bus # Call the RPC is_valid = bus . auth . check_password ( username = \"adam\" , password = \"secr3t\" ) Calling (asynchronously) \u00b6 You can also perform the call asynchronously using asyncio: # Anywhere in your code # Import your project's bus instance from bus import bus is_valid = await bus . auth . check_password . call_async ( username = \"adam\" , password = \"secr3t\" ) Type hints \u00b6 See the typing reference . Best practices \u00b6 RPC implementation \u00b6 It is best to keep your RPCs simple and easy to understand. In anything but the simplest service it will probably be best to use the API definition as a presentational layer which wraps up the business logic located elsewhere. If you find business logic creeping into your RPC definitions, consider factoring it out and invoking it from the RPC definition. For smaller services this will be less important, but as functionality is shared and used elsewhere within your service you may find it keeps your code more managable. This also leaves your RPCs definitions free do any API-specific legwork such as data marshalling. For example, converting incoming natural keys (e.g. usernames) into the primary keys (i.e. user IDs) which your service's may use internally. Architecture & coupling \u00b6 RPCs often represent a tight coupling between your code and the service you are calling. This may be acceptable to you, but it is worth being aware of potential pitfalls: Failures & timeouts may occurr, which should ideally be handled gracefully Modifications to the remote RPC may require updates to the code which calls the RPC RPCs incur much greater overhead than regular function calls. Utility functions that use RPCs should therefore make it clear that they will be incurring this overhead (either through naming convention or documentation) The more services call to for a given action, the less reliable the action will typically be. (i.e. if any service is unavailable the action will potentially fail) Using events will provide a different set of trade-offs, and ultimately you will need to decide on what is right for your particular scenario. Parameter values \u00b6 When deciding the values your RPC should receive, consider: Can this value become out of date? For example, an entire user object could become out of date, whereas a username or user ID would not. Packing and unpacking large data structures is computationally expensive. Will the meaning of the value change over time? For example, the meaning of 'today' or 'now' will change, but the meaning of a specific date & time will remain the same. Limitations \u00b6 RPCs can only be called with keyword arguments. For example: # Raises an InvalidParameters exception result = bus . auth . check_password ( 'admin' , 'secret' ) # ERROR # Success result = bus . auth . check_password ( username = 'admin' , password = 'secret' )","title":"Remote procedure calls"},{"location":"reference/rpcs/#definition","text":"As covered in previous sections, you define RPCs as follows: # bus.py from lightbus import Api class AuthApi ( Api ): class Meta : name = 'auth' def check_password ( self , username , password ): return username == 'admin' and password == 'secret' def reset_password ( self , username ): reset_users_password_somehow ( username ) def get_user ( self , username ): return get_user ( username ) def promote_to_admin ( username ): user = get_user ( username ) user . admin = True user . save ()","title":"Definition"},{"location":"reference/rpcs/#calling","text":"RPCs are called simply as follows: # Anywhere in your code # Import your project's bus instance from bus import bus # Call the RPC is_valid = bus . auth . check_password ( username = \"adam\" , password = \"secr3t\" )","title":"Calling"},{"location":"reference/rpcs/#calling-asynchronously","text":"You can also perform the call asynchronously using asyncio: # Anywhere in your code # Import your project's bus instance from bus import bus is_valid = await bus . auth . check_password . call_async ( username = \"adam\" , password = \"secr3t\" )","title":"Calling (asynchronously)"},{"location":"reference/rpcs/#type-hints","text":"See the typing reference .","title":"Type hints"},{"location":"reference/rpcs/#best-practices","text":"","title":"Best practices"},{"location":"reference/rpcs/#rpc-implementation","text":"It is best to keep your RPCs simple and easy to understand. In anything but the simplest service it will probably be best to use the API definition as a presentational layer which wraps up the business logic located elsewhere. If you find business logic creeping into your RPC definitions, consider factoring it out and invoking it from the RPC definition. For smaller services this will be less important, but as functionality is shared and used elsewhere within your service you may find it keeps your code more managable. This also leaves your RPCs definitions free do any API-specific legwork such as data marshalling. For example, converting incoming natural keys (e.g. usernames) into the primary keys (i.e. user IDs) which your service's may use internally.","title":"RPC implementation"},{"location":"reference/rpcs/#architecture-coupling","text":"RPCs often represent a tight coupling between your code and the service you are calling. This may be acceptable to you, but it is worth being aware of potential pitfalls: Failures & timeouts may occurr, which should ideally be handled gracefully Modifications to the remote RPC may require updates to the code which calls the RPC RPCs incur much greater overhead than regular function calls. Utility functions that use RPCs should therefore make it clear that they will be incurring this overhead (either through naming convention or documentation) The more services call to for a given action, the less reliable the action will typically be. (i.e. if any service is unavailable the action will potentially fail) Using events will provide a different set of trade-offs, and ultimately you will need to decide on what is right for your particular scenario.","title":"Architecture & coupling"},{"location":"reference/rpcs/#parameter-values","text":"When deciding the values your RPC should receive, consider: Can this value become out of date? For example, an entire user object could become out of date, whereas a username or user ID would not. Packing and unpacking large data structures is computationally expensive. Will the meaning of the value change over time? For example, the meaning of 'today' or 'now' will change, but the meaning of a specific date & time will remain the same.","title":"Parameter values"},{"location":"reference/rpcs/#limitations","text":"RPCs can only be called with keyword arguments. For example: # Raises an InvalidParameters exception result = bus . auth . check_password ( 'admin' , 'secret' ) # ERROR # Success result = bus . auth . check_password ( username = 'admin' , password = 'secret' )","title":"Limitations"},{"location":"reference/schema/","text":"Lightbus processes automatically generate and share schemas for their available APIs. These schemes can be used to validate the following: Remote procedure call parameters Remote procedure call return values Event parameters These schemas are shared using the configured SchemaTransprt (Redis, by default). Each Lightbus process will monitor for any schema changes. Specifying types \u00b6 Lightbus will create a schema by inspecting the parameters and type hints of your APIs' events and procedures. You can use the schema functionality without type hints, but the level of validation provided will be limited to ensuring parameter names match what is expected. The schema protocol reference covers the specifics of the schema data format. Supported data types \u00b6 Lightbus maps Python types to JSON types. While Python-specific values can be sent using Lightbus, these values will arrive in their JSON form. For example, if you send a string then a string will arrive. However, if you send the Decimal value 3.124 , then you will receive the string value 3.124 instead. The following types are reasonably interoperable: Python type sent JSON schema interpretation Type received str string str int , float number int , float bool boolean bool list , tuple array list None null None dict , Mapping , etc object dict Mapping[str, ...] object , with pattern properties set dict Tuple[A, B, C] array with maxItems/minItems and items set. list The following types will be successfully encoded and sent, but will arrive as their encoded equivalent: Python type JSON Schema type Value arrives as bytes , Decimal , complex string str datetime , date str str (ISO 8601) NamedTuple with annotations object with specific typed properties dict object with annotations object with specific typed properties dict Lightbus can also handle the following: Python type JSON Schema type Any {} (any value) Union[...] oneOf{...} (see oneOf ) Enum Sets enum property Automatic validation \u00b6 By default this validation will be validated in both the incoming and outgoing directions. Outgoing refers to the dispatching of events or procedure calls to the bus. Incoming refers to the processing of procedure calls or handling of received events. You can configuring this using the validate configuration option. Validation configuration \u00b6 You can configure the validation behaviour in your bus' config.yaml . validate (bool) = true \u00b6 You can enable/disable validation using a boolean true/false flag: # In config.yaml apis : default : validate : false For finer grained control you can specify individual flags for incoming/outgoing validation: # In config.yaml apis : default : validate : outgoing : true incoming : false strict_validation (bool) = false \u00b6 If strict_validation is true then calling a procedure for which no schema exists will result in an error: # In config.yaml apis : default : strict_validation : true Manual validation \u00b6 Lightbus supports manually loading your bus' schema into your bus client. For example, you may take a dump of your production bus schema and use it to test/develop against either in your local development environment or in any automated testing system. You can load a local schema as follows: from bus import bus # Either load form a single file bus . schema . load_local ( \"/path/to/local/bus-schema.json\" ) # ... or load all schemas in a directory bus . schema . load_local ( \"/path/to/local/schema/\" ) Your bus client will now use the specified schema to validate RPCs and events. Parameters and responses may also be directly validated so you need: Event (parameters): bus.my_api.my_event.validate_parameters(parameters) RPC (parameters): bus.my_api.my_rpc.validate_parameters(parameters) RPC (response): bus.my_api.my_rpc.validate_response(response) You can use these methods to manually validate parameters or response values against the locally loaded schema.","title":"Schema"},{"location":"reference/schema/#specifying-types","text":"Lightbus will create a schema by inspecting the parameters and type hints of your APIs' events and procedures. You can use the schema functionality without type hints, but the level of validation provided will be limited to ensuring parameter names match what is expected. The schema protocol reference covers the specifics of the schema data format.","title":"Specifying types"},{"location":"reference/schema/#supported-data-types","text":"Lightbus maps Python types to JSON types. While Python-specific values can be sent using Lightbus, these values will arrive in their JSON form. For example, if you send a string then a string will arrive. However, if you send the Decimal value 3.124 , then you will receive the string value 3.124 instead. The following types are reasonably interoperable: Python type sent JSON schema interpretation Type received str string str int , float number int , float bool boolean bool list , tuple array list None null None dict , Mapping , etc object dict Mapping[str, ...] object , with pattern properties set dict Tuple[A, B, C] array with maxItems/minItems and items set. list The following types will be successfully encoded and sent, but will arrive as their encoded equivalent: Python type JSON Schema type Value arrives as bytes , Decimal , complex string str datetime , date str str (ISO 8601) NamedTuple with annotations object with specific typed properties dict object with annotations object with specific typed properties dict Lightbus can also handle the following: Python type JSON Schema type Any {} (any value) Union[...] oneOf{...} (see oneOf ) Enum Sets enum property","title":"Supported data types"},{"location":"reference/schema/#automatic-validation","text":"By default this validation will be validated in both the incoming and outgoing directions. Outgoing refers to the dispatching of events or procedure calls to the bus. Incoming refers to the processing of procedure calls or handling of received events. You can configuring this using the validate configuration option.","title":"Automatic validation"},{"location":"reference/schema/#validation-configuration","text":"You can configure the validation behaviour in your bus' config.yaml .","title":"Validation configuration"},{"location":"reference/schema/#validate-bool-true","text":"You can enable/disable validation using a boolean true/false flag: # In config.yaml apis : default : validate : false For finer grained control you can specify individual flags for incoming/outgoing validation: # In config.yaml apis : default : validate : outgoing : true incoming : false","title":"validate (bool) = true"},{"location":"reference/schema/#strict_validation-bool-false","text":"If strict_validation is true then calling a procedure for which no schema exists will result in an error: # In config.yaml apis : default : strict_validation : true","title":"strict_validation (bool) = false"},{"location":"reference/schema/#manual-validation","text":"Lightbus supports manually loading your bus' schema into your bus client. For example, you may take a dump of your production bus schema and use it to test/develop against either in your local development environment or in any automated testing system. You can load a local schema as follows: from bus import bus # Either load form a single file bus . schema . load_local ( \"/path/to/local/bus-schema.json\" ) # ... or load all schemas in a directory bus . schema . load_local ( \"/path/to/local/schema/\" ) Your bus client will now use the specified schema to validate RPCs and events. Parameters and responses may also be directly validated so you need: Event (parameters): bus.my_api.my_event.validate_parameters(parameters) RPC (parameters): bus.my_api.my_rpc.validate_parameters(parameters) RPC (response): bus.my_api.my_rpc.validate_response(response) You can use these methods to manually validate parameters or response values against the locally loaded schema.","title":"Manual validation"},{"location":"reference/testing-and-mocking/","text":"Testing & mocking \u00b6 Lightbus provides utilities to make testing easier. These utilities allow you to: Ensure only specific events & RPCs were fired Access the sent messages Mock RPC responses Mocking with events \u00b6 from lightbus.utilities.testing import BusMocker from bus import bus def test_firing_event (): with BusMocker ( bus ) as bus_mock : # Setup the mocker to expect the auth.user_created was fired. # An error will be raised if the tested code fires any other events bus_mock . mock_event_firing ( \"auth.user_created\" ) # Run the code to be tested bus . auth . user_created . fire ( field = \"x\" ) # Check the event was fired once bus_mock . assert_events_fired ( \"auth.user_created\" , times = 1 ) # Get the fired event message message = bus_mock . get_event_messages ( \"auth.user_created\" )[ 0 ] assert message . kwargs == { \"username\" : \"sarahjane\" } Mocking with RPCs \u00b6 from lightbus.utilities.testing import BusMocker from bus import bus def test_calling_rpc (): with BusMocker ( bus ) as bus_mock : # Setup the mocker to expect the auth.user_created was fired. # An error will be raised if the tested code calls any other RPCs bus_mock . mock_rpc_call ( \"auth.check_password\" , result = True ) # Run the code to be tested bus . auth . check_password ( username = \"sarahjane\" , password = \"secret\" ) # Check the event was fired once bus_mock . assert_rpc_called ( \"auth.check_password\" , times = 1 ) # Get the fired RPC message = bus_mock . get_rpc_messages ( \"auth.check_password\" )[ 0 ] assert message . kwargs == { \"username\" : \"sarahjane\" , \"password\" : \"secret\" } Allowing arbitrary events and RPCs \u00b6 By default the mocker will raise an error if any event or RPC is fired or called which has not be setup using the mock_event_firing / mock_rpc_call methods. You can disable this and therefore allow any events or RPCs to the fired/called by using BusMocker(bus, require_mocking=False) . For example, the following will result in no errors even though we have not setup any mocks: from lightbus.utilities.testing import BusMocker from bus import bus def test_permissive_mocking (): with BusMocker ( bus , require_mocking = False ) as bus_mock : # Neither will cause an error despite the lack of mocking setup. # This is because we have set require_mocking=False bus . auth . user_created . fire ( field = \"x\" ) bus . auth . check_password ( username = \"sarahjane\" , password = \"secret\" ) The @bus_mocker decorator \u00b6 You can also access the bus mocker using the @bus_mocker decorator. We can rewrite our earlier event example (above) as follows: from lightbus.utilities.testing import bus_mocker from bus import bus @bus_mocker ( bus ) def test_firing_event ( bus_mock ): # Setup the mocker to expect the auth.user_created was fired. # An error will be raised if the tested code fires any other events bus_mock . mock_event_firing ( \"auth.user_created\" ) # Run the code to be tested bus . auth . user_created . fire ( field = \"x\" ) # Check the event was fired once bus_mock . assert_events_fired ( \"auth.user_created\" , times = 1 ) # Get the fired event message message = bus_mock . get_event_messages ( \"auth.user_created\" )[ 0 ] assert message . kwargs == { \"username\" : \"sarahjane\" } Testing in Django \u00b6 You can use the mocker in your Django tests just as shown above. For example, we can access the mocker using the context manager: from django.test import TestCase from lightbus.utilities.testing import BusMocker from bus import bus class ExampleTestCase ( TestCase ): def test_something ( self ): with BusMocker ( bus , require_mocking = False ) as bus_mock : ... Or we can use the @bus_mocker decorator: from django.test import TestCase from lightbus.utilities.testing import bus_mocker from bus import bus class ExampleTestCase ( TestCase ): @bus_mocker ( bus ) def test_something ( self , bus_mock ): ...","title":"Testing & Mocking"},{"location":"reference/testing-and-mocking/#testing-mocking","text":"Lightbus provides utilities to make testing easier. These utilities allow you to: Ensure only specific events & RPCs were fired Access the sent messages Mock RPC responses","title":"Testing & mocking"},{"location":"reference/testing-and-mocking/#mocking-with-events","text":"from lightbus.utilities.testing import BusMocker from bus import bus def test_firing_event (): with BusMocker ( bus ) as bus_mock : # Setup the mocker to expect the auth.user_created was fired. # An error will be raised if the tested code fires any other events bus_mock . mock_event_firing ( \"auth.user_created\" ) # Run the code to be tested bus . auth . user_created . fire ( field = \"x\" ) # Check the event was fired once bus_mock . assert_events_fired ( \"auth.user_created\" , times = 1 ) # Get the fired event message message = bus_mock . get_event_messages ( \"auth.user_created\" )[ 0 ] assert message . kwargs == { \"username\" : \"sarahjane\" }","title":"Mocking with events"},{"location":"reference/testing-and-mocking/#mocking-with-rpcs","text":"from lightbus.utilities.testing import BusMocker from bus import bus def test_calling_rpc (): with BusMocker ( bus ) as bus_mock : # Setup the mocker to expect the auth.user_created was fired. # An error will be raised if the tested code calls any other RPCs bus_mock . mock_rpc_call ( \"auth.check_password\" , result = True ) # Run the code to be tested bus . auth . check_password ( username = \"sarahjane\" , password = \"secret\" ) # Check the event was fired once bus_mock . assert_rpc_called ( \"auth.check_password\" , times = 1 ) # Get the fired RPC message = bus_mock . get_rpc_messages ( \"auth.check_password\" )[ 0 ] assert message . kwargs == { \"username\" : \"sarahjane\" , \"password\" : \"secret\" }","title":"Mocking with RPCs"},{"location":"reference/testing-and-mocking/#allowing-arbitrary-events-and-rpcs","text":"By default the mocker will raise an error if any event or RPC is fired or called which has not be setup using the mock_event_firing / mock_rpc_call methods. You can disable this and therefore allow any events or RPCs to the fired/called by using BusMocker(bus, require_mocking=False) . For example, the following will result in no errors even though we have not setup any mocks: from lightbus.utilities.testing import BusMocker from bus import bus def test_permissive_mocking (): with BusMocker ( bus , require_mocking = False ) as bus_mock : # Neither will cause an error despite the lack of mocking setup. # This is because we have set require_mocking=False bus . auth . user_created . fire ( field = \"x\" ) bus . auth . check_password ( username = \"sarahjane\" , password = \"secret\" )","title":"Allowing arbitrary events and RPCs"},{"location":"reference/testing-and-mocking/#the-bus_mocker-decorator","text":"You can also access the bus mocker using the @bus_mocker decorator. We can rewrite our earlier event example (above) as follows: from lightbus.utilities.testing import bus_mocker from bus import bus @bus_mocker ( bus ) def test_firing_event ( bus_mock ): # Setup the mocker to expect the auth.user_created was fired. # An error will be raised if the tested code fires any other events bus_mock . mock_event_firing ( \"auth.user_created\" ) # Run the code to be tested bus . auth . user_created . fire ( field = \"x\" ) # Check the event was fired once bus_mock . assert_events_fired ( \"auth.user_created\" , times = 1 ) # Get the fired event message message = bus_mock . get_event_messages ( \"auth.user_created\" )[ 0 ] assert message . kwargs == { \"username\" : \"sarahjane\" }","title":"The @bus_mocker decorator"},{"location":"reference/testing-and-mocking/#testing-in-django","text":"You can use the mocker in your Django tests just as shown above. For example, we can access the mocker using the context manager: from django.test import TestCase from lightbus.utilities.testing import BusMocker from bus import bus class ExampleTestCase ( TestCase ): def test_something ( self ): with BusMocker ( bus , require_mocking = False ) as bus_mock : ... Or we can use the @bus_mocker decorator: from django.test import TestCase from lightbus.utilities.testing import bus_mocker from bus import bus class ExampleTestCase ( TestCase ): @bus_mocker ( bus ) def test_something ( self , bus_mock ): ...","title":"Testing in Django"},{"location":"reference/transport-configuration/","text":"Transport configuration \u00b6 Lightbus ships with built-in support for Redis. This is provided by the following transports: An event transport \u2013 sends and consumes RPC calls An RPC transport \u2013 sends and receives RPC results A result transport \u2013 sends and consumes events A schema transport \u2013 stores and retrieves the bus schema Configuration auto-complete using JSON Schema Many code editors support using a JSON schema to provide auto-complete and validation when editing a JSON file. If you wish, you can write your configuration in JSON (rather than YAML), and load the following JSON schema into your editor: https://lightbus.org/static/default-config-schema.json This will provide you with autocomplete and validation for Lightbus' various configuration options. If you are using custom transports or plugins you should generate your own config schema . Complete configuration example \u00b6 ... apis : # Catch-all api config default : # ...API Config options here (see configuration reference)... # Per-transport configuration event_transport : redis : url : \"redis://redis_host:6379/0\" batch_size : 10 reclaim_batch_size : 100 serializer : \"lightbus.serializers.ByFieldMessageSerializer\" deserializer : \"lightbus.serializers.ByFieldMessageDeserializer\" acknowledgement_timeout : 60 max_stream_length : 100000 stream_use : \"per_api\" consumption_restart_delay : 5 consumer_ttl : 2592000 rpc_transport : redis : url : \"redis://redis_host:6379/0\" batch_size : 10 serializer : \"lightbus.serializers.BlobMessageSerializer\" deserializer : \"lightbus.serializers.BlobMessageDeserializer\" rpc_timeout : 5 rpc_retry_delay : 1 consumption_restart_delay : 5 result_transport : redis : url : \"redis://redis_host:6379/0\" serializer : \"lightbus.serializers.BlobMessageSerializer\" deserializer : \"lightbus.serializers.BlobMessageDeserializer\" rpc_timeout : 5 rpc_retry_delay : 1 consumption_restart_delay : 5 # Can respecify the above config for specific APIs # if customisation is needed. marketing.analytics : ... # Schema transport configuration is at the root level schema : transport : redis : url : \"redis://redis.svc.cluster.local:6379/0\" Redis Event Transport configuration \u00b6 url \u00b6 Type: str , default: redis://127.0.0.1:6379/0 The connection string for the redis server. Format is: `redis://host:port/db_number` batch_size \u00b6 Type: int , default: 10 The maximum number of messages to be fetched at one time. A higher value will reduce overhead for large volumes of messages. However, should a worker die then the processing of the fetched messages will be delayed by acknowledgement_timeout . In this case those messages will be processed out-of-order. reclaim_batch_size \u00b6 Type: int , default: reclaim_batch_size * 10 The maximum number of messages to be fetched at one time when reclaiming timed out messages . serializer \u00b6 Type: str , default: lightbus.serializers.ByFieldMessageSerializer The serializer to be used in converting the lightbus message into a bus-appropriate format. deserializer \u00b6 Type: str , default: lightbus.serializers.ByFieldMessageDeserializer The deserializer to be used in converting the lightbus message into a bus-appropriate format. acknowledgement_timeout \u00b6 Type: float , default: 60.0 , seconds Any message not processed acknowledgement_timeout seconds will assume to have failed and will therefore be reclaimed up by another Lightbus worker. This is typically caused by a Lightbus worker exiting ungracefully. Long running events You will need to modify this if you have event handlers which take a long time to execute. This value must exceed the length of time it takes any event to be processed reclaim_interval \u00b6 Type: float , default is the value of acknowledgement_timeout , in seconds How often should each Lightbus client attempt to reclaim events? When an error occurs in a Lightbus worker then it may die while still holding a claim over events. Other Lightbus works therefore periodically check for any events which have timed out (see acknowledgement_timeout ) and attempt to reclaim any events found. Reclaimed events are then processed and acknowledged as normal. max_stream_length \u00b6 Type: int , default: 100_000 Streams will be trimmed so they never exceed the given length. Set to null for no limit. stream_use \u00b6 Type: str , default: per_api How should Redis streams be created? There are two options: per_api \u2013 One stream for an entire API. per_event \u2013 One stream for each event on an API Setting this to per_api will ensure event listeners receive all events on an api in order. However, all events for the API will be received even if not needed (Lightbus will discard these unwanted events before passing them to your event handler). This will consume unnecessary resources if an API contains high-volume events which your listener does not care for. Setting this to per_event will ensure that Lightbus only receives the needed events, but messages will only be ordered for an individual event. consumption_restart_delay \u00b6 Type: int , default: 5 , seconds How long to wait before attempting to reconnect after loosing the connection to Redis. consumer_ttl \u00b6 Type: int , default: 2_592_000 , seconds How long to wait before cleaning up inactive consumers. Default is 30 days. Redis RPC Transport configuration \u00b6 url \u00b6 Type: str , default: redis://127.0.0.1:6379/0 The connection string for the redis server. Format is: `redis://host:port/db_number` batch_size \u00b6 Type: int , default: 10 The maximum number of messages to be fetched at one time. A higher value will reduce overhead for large volumes of messages. serializer \u00b6 Type: str , default: lightbus.serializers.BlobMessageSerializer The serializer to be used in converting the lightbus message into a bus-appropriate format. deserializer \u00b6 Type: str , default: lightbus.serializers.BlobMessageDeserializer The deserializer to be used in converting the lightbus message into a bus-appropriate format. rpc_timeout \u00b6 Type: int , default: 5 , seconds How long to wait before we give up waiting for an RPC to be processed. Note on rpc_timeout This configuration option is also repeated in the API config . For now this value needs to be specified in three places: The API config RPC transport config Result transport config (see below) rpc_retry_delay \u00b6 Type: int , default: 1 , seconds How long to wait before reattempting to call the remote RPC. For example, in cases where Redis errors (e.g. connection issues). Execution will be retried once only. consumption_restart_delay \u00b6 Type: int , default: 5 , seconds How long to wait before attempting to reconnect after loosing the connection to Redis. Redis Result Transport configuration \u00b6 url \u00b6 Type: str , default: redis://127.0.0.1:6379/0 The connection string for the redis server. Format is: `redis://host:port/db_number` serializer \u00b6 Type: str , default: lightbus.serializers.BlobMessageSerializer The serializer to be used in converting the lightbus message into a bus-appropriate format. deserializer \u00b6 Type: str , default: lightbus.serializers.BlobMessageDeserializer The deserializer to be used in converting the lightbus message into a bus-appropriate format. rpc_timeout \u00b6 Type: int , default: 5 , seconds How long to wait before we give up waiting for an RPC result to be received. Note on rpc_timeout This configuration option is also repeated in the API config . For now this value needs to be specified in three places: # The API config * RPC transport config (see above) * Result transport config rpc_retry_delay \u00b6 Type: int , default: 1 , seconds How long to wait before reattempting to call the remote RPC. For example, in cases where Redis errors (e.g. connection issues). Execution will be retried once only. consumption_restart_delay \u00b6 Type: int , default: 5 , seconds How long to wait before attempting to reconnect after loosing the connection to Redis. Redis Schema Transport configuration \u00b6 url \u00b6 Type: str , default: redis://127.0.0.1:6379/0 The connection string for the redis server. Format is: `redis://host:port/db_number`","title":"Transport configuration"},{"location":"reference/transport-configuration/#transport-configuration","text":"Lightbus ships with built-in support for Redis. This is provided by the following transports: An event transport \u2013 sends and consumes RPC calls An RPC transport \u2013 sends and receives RPC results A result transport \u2013 sends and consumes events A schema transport \u2013 stores and retrieves the bus schema Configuration auto-complete using JSON Schema Many code editors support using a JSON schema to provide auto-complete and validation when editing a JSON file. If you wish, you can write your configuration in JSON (rather than YAML), and load the following JSON schema into your editor: https://lightbus.org/static/default-config-schema.json This will provide you with autocomplete and validation for Lightbus' various configuration options. If you are using custom transports or plugins you should generate your own config schema .","title":"Transport configuration"},{"location":"reference/transport-configuration/#complete-configuration-example","text":"... apis : # Catch-all api config default : # ...API Config options here (see configuration reference)... # Per-transport configuration event_transport : redis : url : \"redis://redis_host:6379/0\" batch_size : 10 reclaim_batch_size : 100 serializer : \"lightbus.serializers.ByFieldMessageSerializer\" deserializer : \"lightbus.serializers.ByFieldMessageDeserializer\" acknowledgement_timeout : 60 max_stream_length : 100000 stream_use : \"per_api\" consumption_restart_delay : 5 consumer_ttl : 2592000 rpc_transport : redis : url : \"redis://redis_host:6379/0\" batch_size : 10 serializer : \"lightbus.serializers.BlobMessageSerializer\" deserializer : \"lightbus.serializers.BlobMessageDeserializer\" rpc_timeout : 5 rpc_retry_delay : 1 consumption_restart_delay : 5 result_transport : redis : url : \"redis://redis_host:6379/0\" serializer : \"lightbus.serializers.BlobMessageSerializer\" deserializer : \"lightbus.serializers.BlobMessageDeserializer\" rpc_timeout : 5 rpc_retry_delay : 1 consumption_restart_delay : 5 # Can respecify the above config for specific APIs # if customisation is needed. marketing.analytics : ... # Schema transport configuration is at the root level schema : transport : redis : url : \"redis://redis.svc.cluster.local:6379/0\"","title":"Complete configuration example"},{"location":"reference/transport-configuration/#redis-event-transport-configuration","text":"","title":"Redis Event Transport configuration"},{"location":"reference/transport-configuration/#url","text":"Type: str , default: redis://127.0.0.1:6379/0 The connection string for the redis server. Format is: `redis://host:port/db_number`","title":"url"},{"location":"reference/transport-configuration/#batch_size","text":"Type: int , default: 10 The maximum number of messages to be fetched at one time. A higher value will reduce overhead for large volumes of messages. However, should a worker die then the processing of the fetched messages will be delayed by acknowledgement_timeout . In this case those messages will be processed out-of-order.","title":"batch_size"},{"location":"reference/transport-configuration/#reclaim_batch_size","text":"Type: int , default: reclaim_batch_size * 10 The maximum number of messages to be fetched at one time when reclaiming timed out messages .","title":"reclaim_batch_size"},{"location":"reference/transport-configuration/#serializer","text":"Type: str , default: lightbus.serializers.ByFieldMessageSerializer The serializer to be used in converting the lightbus message into a bus-appropriate format.","title":"serializer"},{"location":"reference/transport-configuration/#deserializer","text":"Type: str , default: lightbus.serializers.ByFieldMessageDeserializer The deserializer to be used in converting the lightbus message into a bus-appropriate format.","title":"deserializer"},{"location":"reference/transport-configuration/#acknowledgement_timeout","text":"Type: float , default: 60.0 , seconds Any message not processed acknowledgement_timeout seconds will assume to have failed and will therefore be reclaimed up by another Lightbus worker. This is typically caused by a Lightbus worker exiting ungracefully. Long running events You will need to modify this if you have event handlers which take a long time to execute. This value must exceed the length of time it takes any event to be processed","title":"acknowledgement_timeout"},{"location":"reference/transport-configuration/#reclaim_interval","text":"Type: float , default is the value of acknowledgement_timeout , in seconds How often should each Lightbus client attempt to reclaim events? When an error occurs in a Lightbus worker then it may die while still holding a claim over events. Other Lightbus works therefore periodically check for any events which have timed out (see acknowledgement_timeout ) and attempt to reclaim any events found. Reclaimed events are then processed and acknowledged as normal.","title":"reclaim_interval"},{"location":"reference/transport-configuration/#max_stream_length","text":"Type: int , default: 100_000 Streams will be trimmed so they never exceed the given length. Set to null for no limit.","title":"max_stream_length"},{"location":"reference/transport-configuration/#stream_use","text":"Type: str , default: per_api How should Redis streams be created? There are two options: per_api \u2013 One stream for an entire API. per_event \u2013 One stream for each event on an API Setting this to per_api will ensure event listeners receive all events on an api in order. However, all events for the API will be received even if not needed (Lightbus will discard these unwanted events before passing them to your event handler). This will consume unnecessary resources if an API contains high-volume events which your listener does not care for. Setting this to per_event will ensure that Lightbus only receives the needed events, but messages will only be ordered for an individual event.","title":"stream_use"},{"location":"reference/transport-configuration/#consumption_restart_delay","text":"Type: int , default: 5 , seconds How long to wait before attempting to reconnect after loosing the connection to Redis.","title":"consumption_restart_delay"},{"location":"reference/transport-configuration/#consumer_ttl","text":"Type: int , default: 2_592_000 , seconds How long to wait before cleaning up inactive consumers. Default is 30 days.","title":"consumer_ttl"},{"location":"reference/transport-configuration/#redis-rpc-transport-configuration","text":"","title":"Redis RPC Transport configuration"},{"location":"reference/transport-configuration/#url_1","text":"Type: str , default: redis://127.0.0.1:6379/0 The connection string for the redis server. Format is: `redis://host:port/db_number`","title":"url"},{"location":"reference/transport-configuration/#batch_size_1","text":"Type: int , default: 10 The maximum number of messages to be fetched at one time. A higher value will reduce overhead for large volumes of messages.","title":"batch_size"},{"location":"reference/transport-configuration/#serializer_1","text":"Type: str , default: lightbus.serializers.BlobMessageSerializer The serializer to be used in converting the lightbus message into a bus-appropriate format.","title":"serializer"},{"location":"reference/transport-configuration/#deserializer_1","text":"Type: str , default: lightbus.serializers.BlobMessageDeserializer The deserializer to be used in converting the lightbus message into a bus-appropriate format.","title":"deserializer"},{"location":"reference/transport-configuration/#rpc_timeout","text":"Type: int , default: 5 , seconds How long to wait before we give up waiting for an RPC to be processed. Note on rpc_timeout This configuration option is also repeated in the API config . For now this value needs to be specified in three places: The API config RPC transport config Result transport config (see below)","title":"rpc_timeout"},{"location":"reference/transport-configuration/#rpc_retry_delay","text":"Type: int , default: 1 , seconds How long to wait before reattempting to call the remote RPC. For example, in cases where Redis errors (e.g. connection issues). Execution will be retried once only.","title":"rpc_retry_delay"},{"location":"reference/transport-configuration/#consumption_restart_delay_1","text":"Type: int , default: 5 , seconds How long to wait before attempting to reconnect after loosing the connection to Redis.","title":"consumption_restart_delay"},{"location":"reference/transport-configuration/#redis-result-transport-configuration","text":"","title":"Redis Result Transport configuration"},{"location":"reference/transport-configuration/#url_2","text":"Type: str , default: redis://127.0.0.1:6379/0 The connection string for the redis server. Format is: `redis://host:port/db_number`","title":"url"},{"location":"reference/transport-configuration/#serializer_2","text":"Type: str , default: lightbus.serializers.BlobMessageSerializer The serializer to be used in converting the lightbus message into a bus-appropriate format.","title":"serializer"},{"location":"reference/transport-configuration/#deserializer_2","text":"Type: str , default: lightbus.serializers.BlobMessageDeserializer The deserializer to be used in converting the lightbus message into a bus-appropriate format.","title":"deserializer"},{"location":"reference/transport-configuration/#rpc_timeout_1","text":"Type: int , default: 5 , seconds How long to wait before we give up waiting for an RPC result to be received. Note on rpc_timeout This configuration option is also repeated in the API config . For now this value needs to be specified in three places: # The API config * RPC transport config (see above) * Result transport config","title":"rpc_timeout"},{"location":"reference/transport-configuration/#rpc_retry_delay_1","text":"Type: int , default: 1 , seconds How long to wait before reattempting to call the remote RPC. For example, in cases where Redis errors (e.g. connection issues). Execution will be retried once only.","title":"rpc_retry_delay"},{"location":"reference/transport-configuration/#consumption_restart_delay_2","text":"Type: int , default: 5 , seconds How long to wait before attempting to reconnect after loosing the connection to Redis.","title":"consumption_restart_delay"},{"location":"reference/transport-configuration/#redis-schema-transport-configuration","text":"","title":"Redis Schema Transport configuration"},{"location":"reference/transport-configuration/#url_3","text":"Type: str , default: redis://127.0.0.1:6379/0 The connection string for the redis server. Format is: `redis://host:port/db_number`","title":"url"},{"location":"reference/typing/","text":"Specifying type hints allows Lightbus to validate data types in both incoming and outgoing messages. Type hints are used to create your bus' schema , which is shared across your entire bus. Typing syntax for RPCs \u00b6 You can provide typing information for Remote Procedure Calls using regular Python type hinting: class AuthApi ( lightbus . Api ): class Meta : name = 'auth' def check_password ( self , username : str , password : str ) -> bool : return username == 'admin' and password == 'secret' This will: Ensure the received username parameter is a string Ensure the received password parameter is a string Ensure the returned value is a boolean This behaviour can be configured via the validate configuration option . Typing syntax for events \u00b6 Typing information for events is different to that for RPCs. Firstly, events do not provide return values. Secondly, event parameters are specified differently: # auth_service/bus.py from lightbus import Api , Event , Parameter class AuthApi ( Api ): # WITHOUT types user_created = Event ( parameters = ( 'username' , 'email' , 'is_admin' )) # WITH types user_created = Event ( parameters = ( Parameter ( 'username' , str ), Parameter ( 'new_email' , str ), Parameter ( 'is_admin' , bool , default = False ), )) This will: Ensure the received username parameter is a string Ensure the received new_email parameter is a string Ensure the received is_admin parameter is a boolean. If omitted, False will be used. This behaviour can be configured via the validate configuration option . Data structures \u00b6 In additional to built in types, Lightbus can derive typing information from the following data structures: Named Tuples Dataclasses Any class defining the __to_bus__ and __from_bus__ methods For each of these data structures Lightbus will: Encode values as JSON objects (i.e. dictionaries) Use the structure's type hints in generating the JSON schema... ... and therefore validate incoming/outgoing objects against this schema NamedTuple example \u00b6 # bus.py from lightbus import Api from typing import NamedTuple class User ( NamedTuple ): username : str name : str email : str is_admin : bool = False class AuthApi ( Api ): class Meta : name = 'auth' def get_user ( self , username : str ) -> User : return ... Dataclass example \u00b6 Lightbus supports dataclasses in the same way as it supports named tuples . For example: # bus.py from lightbus import Api from dataclasses import dataclass @dataclass () class User (): username : str name : str email : str is_admin : bool = False class AuthApi ( Api ): class Meta : name = 'auth' def get_user ( self , username : str ) -> User : return ... Custom class example \u00b6 Lightbus can also work with classes of any type provided that: The class defines a __from_bus__(self, value) class method, which returns an instance of the class. If value is a dict, it will be best-effort cased to the class' type annotations before __from_bus__ is invoked. The class defines a __to_bus__(self) method which both annotates its return type, and returns a value suitable for serialising on the bus. from lightbus import Api class User : username : str name : str email : str is_admin : bool = False def do_something ( self ): pass @classmethod def __from_bus__ ( cls , value ): user = cls () user . username = value [ \"username\" ] user . name = value [ \"name\" ] user . email = value [ \"email\" ] user . is_admin = value . get ( \"is_admin\" , False ) return user def __to_bus__ ( self ) -> dict : return dict ( username = self . username , name = self . name , email = self . email , is_admin = self . is_admin , ) class AuthApi ( Api ): class Meta : name = 'auth' def get_user ( self , username : str ) -> User : return ...","title":"Typing"},{"location":"reference/typing/#typing-syntax-for-rpcs","text":"You can provide typing information for Remote Procedure Calls using regular Python type hinting: class AuthApi ( lightbus . Api ): class Meta : name = 'auth' def check_password ( self , username : str , password : str ) -> bool : return username == 'admin' and password == 'secret' This will: Ensure the received username parameter is a string Ensure the received password parameter is a string Ensure the returned value is a boolean This behaviour can be configured via the validate configuration option .","title":"Typing syntax for RPCs"},{"location":"reference/typing/#typing-syntax-for-events","text":"Typing information for events is different to that for RPCs. Firstly, events do not provide return values. Secondly, event parameters are specified differently: # auth_service/bus.py from lightbus import Api , Event , Parameter class AuthApi ( Api ): # WITHOUT types user_created = Event ( parameters = ( 'username' , 'email' , 'is_admin' )) # WITH types user_created = Event ( parameters = ( Parameter ( 'username' , str ), Parameter ( 'new_email' , str ), Parameter ( 'is_admin' , bool , default = False ), )) This will: Ensure the received username parameter is a string Ensure the received new_email parameter is a string Ensure the received is_admin parameter is a boolean. If omitted, False will be used. This behaviour can be configured via the validate configuration option .","title":"Typing syntax for events"},{"location":"reference/typing/#data-structures","text":"In additional to built in types, Lightbus can derive typing information from the following data structures: Named Tuples Dataclasses Any class defining the __to_bus__ and __from_bus__ methods For each of these data structures Lightbus will: Encode values as JSON objects (i.e. dictionaries) Use the structure's type hints in generating the JSON schema... ... and therefore validate incoming/outgoing objects against this schema","title":"Data structures"},{"location":"reference/typing/#namedtuple-example","text":"# bus.py from lightbus import Api from typing import NamedTuple class User ( NamedTuple ): username : str name : str email : str is_admin : bool = False class AuthApi ( Api ): class Meta : name = 'auth' def get_user ( self , username : str ) -> User : return ...","title":"NamedTuple example"},{"location":"reference/typing/#dataclass-example","text":"Lightbus supports dataclasses in the same way as it supports named tuples . For example: # bus.py from lightbus import Api from dataclasses import dataclass @dataclass () class User (): username : str name : str email : str is_admin : bool = False class AuthApi ( Api ): class Meta : name = 'auth' def get_user ( self , username : str ) -> User : return ...","title":"Dataclass example"},{"location":"reference/typing/#custom-class-example","text":"Lightbus can also work with classes of any type provided that: The class defines a __from_bus__(self, value) class method, which returns an instance of the class. If value is a dict, it will be best-effort cased to the class' type annotations before __from_bus__ is invoked. The class defines a __to_bus__(self) method which both annotates its return type, and returns a value suitable for serialising on the bus. from lightbus import Api class User : username : str name : str email : str is_admin : bool = False def do_something ( self ): pass @classmethod def __from_bus__ ( cls , value ): user = cls () user . username = value [ \"username\" ] user . name = value [ \"name\" ] user . email = value [ \"email\" ] user . is_admin = value . get ( \"is_admin\" , False ) return user def __to_bus__ ( self ) -> dict : return dict ( username = self . username , name = self . name , email = self . email , is_admin = self . is_admin , ) class AuthApi ( Api ): class Meta : name = 'auth' def get_user ( self , username : str ) -> User : return ...","title":"Custom class example"},{"location":"reference/command-line-use/dumpconfigschema/","text":"lightbus dumpconfigschema \u00b6 This command will output a JSON schema for the global bus configuration file. The global bus configuration file is typically written as YAML, but it can also be written as JSON. In which case, you can validate the structure against the JSON schema produced by this command. This schema can also be loaded into some editors to provide auto-completion when editing your bus' configuration file. Important Be careful not to confuse this command with dumpschema . The dumpschema command dumps your bus' schema, whereas this dumpconfigschema simply dumps the schema for your bus' configuration. Examples \u00b6 Dump the configuration file schema to standard out: lightbus dumpschema Dump the configuration file schema to a file: lightbus dumpschema --out my_schema.json Schema for 3 APIs saved to my_schema.json Option reference \u00b6 $ lightbus dumpconfigschema --help usage: lightbus dumpconfigschema [-h] [--out FILE] [--bus BUS_MODULE] [--service-name SERVICE_NAME] [--process-name PROCESS_NAME] [--config FILE] [--log-level LOG_LEVEL] optional arguments: -h, --help show this help message and exit Dump config schema command arguments: --out FILE, -o FILE File to write config schema to. If omitted the schema will be written to standard out. (default: None) Common arguments: --bus BUS_MODULE, -b BUS_MODULE The bus module to import. Example 'bus', 'my_project.bus'. Defaults to the value of the LIGHTBUS_MODULE environment variable, or 'bus' (default: None) --service-name SERVICE_NAME, -s SERVICE_NAME Name of service in which this process resides. YOU SHOULD LIKELY SET THIS IN PRODUCTION. Can also be set using the LIGHTBUS_SERVICE_NAME environment. Will default to a random string. (default: None) --process-name PROCESS_NAME, -p PROCESS_NAME A unique name of this process within the service. Can also be set using the LIGHTBUS_PROCESS_NAME environment. Will default to a random string. (default: None) --config FILE Config file to load, JSON or YAML (default: None) --log-level LOG_LEVEL Set the log level. Overrides any value set in config. One of debug, info, warning, critical, exception. (default: None)","title":"dumpconfigschema"},{"location":"reference/command-line-use/dumpconfigschema/#lightbus-dumpconfigschema","text":"This command will output a JSON schema for the global bus configuration file. The global bus configuration file is typically written as YAML, but it can also be written as JSON. In which case, you can validate the structure against the JSON schema produced by this command. This schema can also be loaded into some editors to provide auto-completion when editing your bus' configuration file. Important Be careful not to confuse this command with dumpschema . The dumpschema command dumps your bus' schema, whereas this dumpconfigschema simply dumps the schema for your bus' configuration.","title":"lightbus dumpconfigschema"},{"location":"reference/command-line-use/dumpconfigschema/#examples","text":"Dump the configuration file schema to standard out: lightbus dumpschema Dump the configuration file schema to a file: lightbus dumpschema --out my_schema.json Schema for 3 APIs saved to my_schema.json","title":"Examples"},{"location":"reference/command-line-use/dumpconfigschema/#option-reference","text":"$ lightbus dumpconfigschema --help usage: lightbus dumpconfigschema [-h] [--out FILE] [--bus BUS_MODULE] [--service-name SERVICE_NAME] [--process-name PROCESS_NAME] [--config FILE] [--log-level LOG_LEVEL] optional arguments: -h, --help show this help message and exit Dump config schema command arguments: --out FILE, -o FILE File to write config schema to. If omitted the schema will be written to standard out. (default: None) Common arguments: --bus BUS_MODULE, -b BUS_MODULE The bus module to import. Example 'bus', 'my_project.bus'. Defaults to the value of the LIGHTBUS_MODULE environment variable, or 'bus' (default: None) --service-name SERVICE_NAME, -s SERVICE_NAME Name of service in which this process resides. YOU SHOULD LIKELY SET THIS IN PRODUCTION. Can also be set using the LIGHTBUS_SERVICE_NAME environment. Will default to a random string. (default: None) --process-name PROCESS_NAME, -p PROCESS_NAME A unique name of this process within the service. Can also be set using the LIGHTBUS_PROCESS_NAME environment. Will default to a random string. (default: None) --config FILE Config file to load, JSON or YAML (default: None) --log-level LOG_LEVEL Set the log level. Overrides any value set in config. One of debug, info, warning, critical, exception. (default: None)","title":"Option reference"},{"location":"reference/command-line-use/dumpschema/","text":"lightbus dumpschema \u00b6 The lightbus dumpschema command will dump the bus' JSON schema to either a file or standard out. This schema file can then be manually provided to lightbus run using the --schema option. Why is this useful? \u00b6 The idea behind this command is to aid in testing and local development. You can take a dump of your production bus' schema and use it in your local development or testing environment. This will allow Lightbus to validate your locally emitted events and RPCs against the expectations of your production environment. See manual validation for more information. Examples \u00b6 Dump the schema to standard out: lightbus dumpschema Dump the schema to a file: lightbus dumpschema --out my_schema.json Schema for 3 APIs saved to my_schema.json Options reference \u00b6 $ lightbus dumpschema --help usage: lightbus dumpschema [-h] [--out FILE_OR_DIRECTORY] [--bus BUS_MODULE] [--service-name SERVICE_NAME] [--process-name PROCESS_NAME] [--config FILE] [--log-level LOG_LEVEL] optional arguments: -h, --help show this help message and exit Dump config schema command arguments: --out FILE_OR_DIRECTORY, -o FILE_OR_DIRECTORY File or directory to write schema to. If a directory is specified one schema file will be created for each API. If omitted the schema will be written to standard out. (default: None) Common arguments: --bus BUS_MODULE, -b BUS_MODULE The bus module to import. Example 'bus', 'my_project.bus'. Defaults to the value of the LIGHTBUS_MODULE environment variable, or 'bus' (default: None) --service-name SERVICE_NAME, -s SERVICE_NAME Name of service in which this process resides. YOU SHOULD LIKELY SET THIS IN PRODUCTION. Can also be set using the LIGHTBUS_SERVICE_NAME environment. Will default to a random string. (default: None) --process-name PROCESS_NAME, -p PROCESS_NAME A unique name of this process within the service. Can also be set using the LIGHTBUS_PROCESS_NAME environment. Will default to a random string. (default: None) --config FILE Config file to load, JSON or YAML (default: None) --log-level LOG_LEVEL Set the log level. Overrides any value set in config. One of debug, info, warning, critical, exception. (default: None)","title":"dumpschema"},{"location":"reference/command-line-use/dumpschema/#lightbus-dumpschema","text":"The lightbus dumpschema command will dump the bus' JSON schema to either a file or standard out. This schema file can then be manually provided to lightbus run using the --schema option.","title":"lightbus dumpschema"},{"location":"reference/command-line-use/dumpschema/#why-is-this-useful","text":"The idea behind this command is to aid in testing and local development. You can take a dump of your production bus' schema and use it in your local development or testing environment. This will allow Lightbus to validate your locally emitted events and RPCs against the expectations of your production environment. See manual validation for more information.","title":"Why is this useful?"},{"location":"reference/command-line-use/dumpschema/#examples","text":"Dump the schema to standard out: lightbus dumpschema Dump the schema to a file: lightbus dumpschema --out my_schema.json Schema for 3 APIs saved to my_schema.json","title":"Examples"},{"location":"reference/command-line-use/dumpschema/#options-reference","text":"$ lightbus dumpschema --help usage: lightbus dumpschema [-h] [--out FILE_OR_DIRECTORY] [--bus BUS_MODULE] [--service-name SERVICE_NAME] [--process-name PROCESS_NAME] [--config FILE] [--log-level LOG_LEVEL] optional arguments: -h, --help show this help message and exit Dump config schema command arguments: --out FILE_OR_DIRECTORY, -o FILE_OR_DIRECTORY File or directory to write schema to. If a directory is specified one schema file will be created for each API. If omitted the schema will be written to standard out. (default: None) Common arguments: --bus BUS_MODULE, -b BUS_MODULE The bus module to import. Example 'bus', 'my_project.bus'. Defaults to the value of the LIGHTBUS_MODULE environment variable, or 'bus' (default: None) --service-name SERVICE_NAME, -s SERVICE_NAME Name of service in which this process resides. YOU SHOULD LIKELY SET THIS IN PRODUCTION. Can also be set using the LIGHTBUS_SERVICE_NAME environment. Will default to a random string. (default: None) --process-name PROCESS_NAME, -p PROCESS_NAME A unique name of this process within the service. Can also be set using the LIGHTBUS_PROCESS_NAME environment. Will default to a random string. (default: None) --config FILE Config file to load, JSON or YAML (default: None) --log-level LOG_LEVEL Set the log level. Overrides any value set in config. One of debug, info, warning, critical, exception. (default: None)","title":"Options reference"},{"location":"reference/command-line-use/inspect/","text":"lightbus inspect \u00b6 The lightbus inspect command allows for inspecting activity on the bus. This is currently limited to debugging only events, not RPCs. The querying facilities provided here are not very performant. The lightbus inspect command will pull a large number of messages from the bus and apply filters locally. Important lightbus inspect will only return results for APIs which are currently being served by worker processes. If you have no workers running you will see no results. This is because lightbus inspect relies on the schema being present on the bus. Examples \u00b6 You can query the bus for a specific lightbus event ID : # Will produce JSON output lightbus inspect --id 1bff06d0-fa9b-11e9-b8ca-f218986bf8ce # Will produce human-readable output lightbus inspect --id 1bff06d0-fa9b-11e9-b8ca-f218986bf8ce --format human You can also query the bus using the message ID native to the underlying broker (in this case, this will be the Redis streams message ID): lightbus inspect --native-id 1572389392059-0 You can also query by API and/or event name : # Event/api filtering lightbus inspect --event user_registered lightbus inspect --api my_company.auth Wildcards are also supported in event & api filtering: # Wildcard filtering lightbus inspect --event user_* lightbus inspect --api my_company.* Unexpected results when using wildcards? You may need to quote wilcard strings in order to prevent your shell expanding them. For example: lightbus inspect --api \"my_company.*\" You can see a continually updating stream of all events for a single API: # Continually show events for the given API lightbus inspect --follow --api auth You can query based on the data sent in the event . For example, to query based on the value of the email parameter of a fired event: lightbus inspect --json-path email=joe@example.com You can also query on more complex JSON path expressions : lightbus inspect --json-path address.city=London Cache \u00b6 This command will maintain an cache of fetched messages in ~/.lightbus/ . This will speed up subsequent executions and reduce the load on the bus. Option reference \u00b6 $ lightbus inspect -h usage: lightbus inspect [-h] [--json-path JSON_SEARCH] [--id LIGHTBUS_EVENT_ID] [--native-id NATIVE_EVENT_ID] [--api API_NAME] [--event EVENT_NAME] [--version VERSION_NUMBER] [--format FORMAT] [--cache-only] [--follow] [--validate] [--show-casting] [--internal] [--bus BUS_MODULE] [--service-name SERVICE_NAME] [--process-name PROCESS_NAME] [--config FILE] [--log-level LOG_LEVEL] optional arguments: -h, --help show this help message and exit Inspect command arguments: --json-path JSON_SEARCH, -j JSON_SEARCH Search event body json for the givn value. Eg. address.city=London (default: None) --id LIGHTBUS_EVENT_ID, -i LIGHTBUS_EVENT_ID Find a single event with this Lightbus event ID (default: None) --native-id NATIVE_EVENT_ID, -n NATIVE_EVENT_ID Find a single event with this broker-native ID (default: None) --api API_NAME, -a API_NAME Find events for this API name. Supports the '*' wildcard. (default: None) --event EVENT_NAME, -e EVENT_NAME Find events for this event name. Supports the '*' wildcard. (default: None) --version VERSION_NUMBER, -v VERSION_NUMBER Find events with the specified version number. Can be prefixed by <, >, <=, >= (default: None) --format FORMAT, -F FORMAT Formatting style. One of json, pretty, or human. (default: json) --cache-only, -c Search the local cache only (default: False) --follow, -f Continually listen for new events matching the search criteria. May only be used on a single API (default: False) --validate, -d Validate displayed events against the bus schema. Only available when --format is set to 'human' (experimental) (default: False) --show-casting, -C Show how the message values will be casted for each event listener (experimental) (default: False) --internal, -I Include internal APIs (default: False) Common arguments: --bus BUS_MODULE, -b BUS_MODULE The bus module to import. Example 'bus', 'my_project.bus'. Defaults to the value of the LIGHTBUS_MODULE environment variable, or 'bus' (default: None) --service-name SERVICE_NAME, -s SERVICE_NAME Name of service in which this process resides. YOU SHOULD LIKELY SET THIS IN PRODUCTION. Can also be set using the LIGHTBUS_SERVICE_NAME environment. Will default to a random string. (default: None) --process-name PROCESS_NAME, -p PROCESS_NAME A unique name of this process within the service. Can also be set using the LIGHTBUS_PROCESS_NAME environment. Will default to a random string. (default: None) --config FILE Config file to load, JSON or YAML (default: None) --log-level LOG_LEVEL Set the log level. Overrides any value set in config. One of debug, info, warning, critical, exception. (default: None)","title":"inspect"},{"location":"reference/command-line-use/inspect/#lightbus-inspect","text":"The lightbus inspect command allows for inspecting activity on the bus. This is currently limited to debugging only events, not RPCs. The querying facilities provided here are not very performant. The lightbus inspect command will pull a large number of messages from the bus and apply filters locally. Important lightbus inspect will only return results for APIs which are currently being served by worker processes. If you have no workers running you will see no results. This is because lightbus inspect relies on the schema being present on the bus.","title":"lightbus inspect"},{"location":"reference/command-line-use/inspect/#examples","text":"You can query the bus for a specific lightbus event ID : # Will produce JSON output lightbus inspect --id 1bff06d0-fa9b-11e9-b8ca-f218986bf8ce # Will produce human-readable output lightbus inspect --id 1bff06d0-fa9b-11e9-b8ca-f218986bf8ce --format human You can also query the bus using the message ID native to the underlying broker (in this case, this will be the Redis streams message ID): lightbus inspect --native-id 1572389392059-0 You can also query by API and/or event name : # Event/api filtering lightbus inspect --event user_registered lightbus inspect --api my_company.auth Wildcards are also supported in event & api filtering: # Wildcard filtering lightbus inspect --event user_* lightbus inspect --api my_company.* Unexpected results when using wildcards? You may need to quote wilcard strings in order to prevent your shell expanding them. For example: lightbus inspect --api \"my_company.*\" You can see a continually updating stream of all events for a single API: # Continually show events for the given API lightbus inspect --follow --api auth You can query based on the data sent in the event . For example, to query based on the value of the email parameter of a fired event: lightbus inspect --json-path email=joe@example.com You can also query on more complex JSON path expressions : lightbus inspect --json-path address.city=London","title":"Examples"},{"location":"reference/command-line-use/inspect/#cache","text":"This command will maintain an cache of fetched messages in ~/.lightbus/ . This will speed up subsequent executions and reduce the load on the bus.","title":"Cache"},{"location":"reference/command-line-use/inspect/#option-reference","text":"$ lightbus inspect -h usage: lightbus inspect [-h] [--json-path JSON_SEARCH] [--id LIGHTBUS_EVENT_ID] [--native-id NATIVE_EVENT_ID] [--api API_NAME] [--event EVENT_NAME] [--version VERSION_NUMBER] [--format FORMAT] [--cache-only] [--follow] [--validate] [--show-casting] [--internal] [--bus BUS_MODULE] [--service-name SERVICE_NAME] [--process-name PROCESS_NAME] [--config FILE] [--log-level LOG_LEVEL] optional arguments: -h, --help show this help message and exit Inspect command arguments: --json-path JSON_SEARCH, -j JSON_SEARCH Search event body json for the givn value. Eg. address.city=London (default: None) --id LIGHTBUS_EVENT_ID, -i LIGHTBUS_EVENT_ID Find a single event with this Lightbus event ID (default: None) --native-id NATIVE_EVENT_ID, -n NATIVE_EVENT_ID Find a single event with this broker-native ID (default: None) --api API_NAME, -a API_NAME Find events for this API name. Supports the '*' wildcard. (default: None) --event EVENT_NAME, -e EVENT_NAME Find events for this event name. Supports the '*' wildcard. (default: None) --version VERSION_NUMBER, -v VERSION_NUMBER Find events with the specified version number. Can be prefixed by <, >, <=, >= (default: None) --format FORMAT, -F FORMAT Formatting style. One of json, pretty, or human. (default: json) --cache-only, -c Search the local cache only (default: False) --follow, -f Continually listen for new events matching the search criteria. May only be used on a single API (default: False) --validate, -d Validate displayed events against the bus schema. Only available when --format is set to 'human' (experimental) (default: False) --show-casting, -C Show how the message values will be casted for each event listener (experimental) (default: False) --internal, -I Include internal APIs (default: False) Common arguments: --bus BUS_MODULE, -b BUS_MODULE The bus module to import. Example 'bus', 'my_project.bus'. Defaults to the value of the LIGHTBUS_MODULE environment variable, or 'bus' (default: None) --service-name SERVICE_NAME, -s SERVICE_NAME Name of service in which this process resides. YOU SHOULD LIKELY SET THIS IN PRODUCTION. Can also be set using the LIGHTBUS_SERVICE_NAME environment. Will default to a random string. (default: None) --process-name PROCESS_NAME, -p PROCESS_NAME A unique name of this process within the service. Can also be set using the LIGHTBUS_PROCESS_NAME environment. Will default to a random string. (default: None) --config FILE Config file to load, JSON or YAML (default: None) --log-level LOG_LEVEL Set the log level. Overrides any value set in config. One of debug, info, warning, critical, exception. (default: None)","title":"Option reference"},{"location":"reference/command-line-use/run/","text":"lightbus run \u00b6 The lightbus run command is used to start the Lightbus worker process and will be used in any Lightbus deployment. See the anatomy lesson for further details. Examples \u00b6 In its basic form lightbus run will expect to be able to import a module called bus which will contain your bus client: lightbus run You can also enable/disable specific features . For example, you may wish to run a worker which responds soley to RPCs in order to ensure a timely response: # Handles RPCs lightbus run --only rpcs # Handles everything else (events and tasks) lightbus run --skip rpcs A production use may look something like this (for more information on service & process names see the events explanation ): lightbus run \\ --bus my_company.my_project.bus \\ --service-name video-encoder \\ --process-name lightbus-worker-1 \\ --config /etc/lightbus/global-bus.yaml \\ --log-level debug This could also be re-written as: export LIGHTBUS_MODULE = my_company.my_project.bus export LIGHTBUS_SERVICE_NAME = video-encoder export LIGHTBUS_PROCESS_NAME = lightbus-worker-1 lightbus run \\ --config /etc/lightbus/global-bus.yaml \\ --log-level debug Configuration can also be loaded over HTTP or HTTPS . This may be useful if you wish to pull the global bus config from an (internal-only) endpoint/service: # Can load JSON/YAML over HTTP/HTTPS lightbus run --config http://config.internal/lightbus/global-bus.yaml Option reference \u00b6 $ lightbus run --help usage: lightbus run [-h] [--only ONLY] [--skip SKIP] [--schema FILE_OR_DIRECTORY] [--bus BUS_MODULE] [--service-name SERVICE_NAME] [--process-name PROCESS_NAME] [--config FILE] [--log-level LOG_LEVEL] optional arguments: -h, --help show this help message and exit Run command arguments: --only ONLY, -o ONLY Only provide the specified features. Comma separated list. Possible values: rpcs, events, tasks (default: None) --skip SKIP, -k SKIP Provide all except the specified features. Comma separated list. Possible values: rpcs, events, tasks (default: None) --schema FILE_OR_DIRECTORY, -m FILE_OR_DIRECTORY Manually load the schema from the given file or directory. This will normally be provided by the schema transport, but manual loading may be useful during development or testing. (default: None) Common arguments: --bus BUS_MODULE, -b BUS_MODULE The bus module to import. Example 'bus', 'my_project.bus'. Defaults to the value of the LIGHTBUS_MODULE environment variable, or 'bus' (default: None) --service-name SERVICE_NAME, -s SERVICE_NAME Name of service in which this process resides. YOU SHOULD LIKELY SET THIS IN PRODUCTION. Can also be set using the LIGHTBUS_SERVICE_NAME environment. Will default to a random string. (default: None) --process-name PROCESS_NAME, -p PROCESS_NAME A unique name of this process within the service. Can also be set using the LIGHTBUS_PROCESS_NAME environment. Will default to a random string. (default: None) --config FILE Config file to load, JSON or YAML (default: None) --log-level LOG_LEVEL Set the log level. Overrides any value set in config. One of debug, info, warning, critical, exception. (default: None)","title":"run"},{"location":"reference/command-line-use/run/#lightbus-run","text":"The lightbus run command is used to start the Lightbus worker process and will be used in any Lightbus deployment. See the anatomy lesson for further details.","title":"lightbus run"},{"location":"reference/command-line-use/run/#examples","text":"In its basic form lightbus run will expect to be able to import a module called bus which will contain your bus client: lightbus run You can also enable/disable specific features . For example, you may wish to run a worker which responds soley to RPCs in order to ensure a timely response: # Handles RPCs lightbus run --only rpcs # Handles everything else (events and tasks) lightbus run --skip rpcs A production use may look something like this (for more information on service & process names see the events explanation ): lightbus run \\ --bus my_company.my_project.bus \\ --service-name video-encoder \\ --process-name lightbus-worker-1 \\ --config /etc/lightbus/global-bus.yaml \\ --log-level debug This could also be re-written as: export LIGHTBUS_MODULE = my_company.my_project.bus export LIGHTBUS_SERVICE_NAME = video-encoder export LIGHTBUS_PROCESS_NAME = lightbus-worker-1 lightbus run \\ --config /etc/lightbus/global-bus.yaml \\ --log-level debug Configuration can also be loaded over HTTP or HTTPS . This may be useful if you wish to pull the global bus config from an (internal-only) endpoint/service: # Can load JSON/YAML over HTTP/HTTPS lightbus run --config http://config.internal/lightbus/global-bus.yaml","title":"Examples"},{"location":"reference/command-line-use/run/#option-reference","text":"$ lightbus run --help usage: lightbus run [-h] [--only ONLY] [--skip SKIP] [--schema FILE_OR_DIRECTORY] [--bus BUS_MODULE] [--service-name SERVICE_NAME] [--process-name PROCESS_NAME] [--config FILE] [--log-level LOG_LEVEL] optional arguments: -h, --help show this help message and exit Run command arguments: --only ONLY, -o ONLY Only provide the specified features. Comma separated list. Possible values: rpcs, events, tasks (default: None) --skip SKIP, -k SKIP Provide all except the specified features. Comma separated list. Possible values: rpcs, events, tasks (default: None) --schema FILE_OR_DIRECTORY, -m FILE_OR_DIRECTORY Manually load the schema from the given file or directory. This will normally be provided by the schema transport, but manual loading may be useful during development or testing. (default: None) Common arguments: --bus BUS_MODULE, -b BUS_MODULE The bus module to import. Example 'bus', 'my_project.bus'. Defaults to the value of the LIGHTBUS_MODULE environment variable, or 'bus' (default: None) --service-name SERVICE_NAME, -s SERVICE_NAME Name of service in which this process resides. YOU SHOULD LIKELY SET THIS IN PRODUCTION. Can also be set using the LIGHTBUS_SERVICE_NAME environment. Will default to a random string. (default: None) --process-name PROCESS_NAME, -p PROCESS_NAME A unique name of this process within the service. Can also be set using the LIGHTBUS_PROCESS_NAME environment. Will default to a random string. (default: None) --config FILE Config file to load, JSON or YAML (default: None) --log-level LOG_LEVEL Set the log level. Overrides any value set in config. One of debug, info, warning, critical, exception. (default: None)","title":"Option reference"},{"location":"reference/command-line-use/shell/","text":"lightbus shell \u00b6 The lightbus shell command provides an interactive prompt through which you can interface with the bus. To use this command you must first install bpython : pip install bpython Examples \u00b6 You should see the following when starting up the shell. This is a fully functional Python shell, with your bus loaded in a ready to be used: $ lightbus shell >>> \u2588 Welcome to the Lightbus shell. Use `bus` to access your bus.\\ Upon typing the shell will begin to auto-complete based on the locally available APIs: $ lightbus shell >>> bus.au\u2588 \u250c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2510 \u2502 auth \u2502 \u2514\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2518 You can fire events and call RPCs as follows: # Fire an event >>> bus.auth.user_registered.fire(email=\"joe@example.com\", username=\"joe\") # Call an RPC >>> bus.auth.check_password(username=\"admin\", password=\"secret\") True Option reference \u00b6 $ lightbus shell --help usage: lightbus shell [-h] [--bus BUS_MODULE] [--service-name SERVICE_NAME] [--process-name PROCESS_NAME] [--config FILE] [--log-level LOG_LEVEL] optional arguments: -h, --help show this help message and exit Common arguments: --bus BUS_MODULE, -b BUS_MODULE The bus module to import. Example 'bus', 'my_project.bus'. Defaults to the value of the LIGHTBUS_MODULE environment variable, or 'bus' (default: None) --service-name SERVICE_NAME, -s SERVICE_NAME Name of service in which this process resides. YOU SHOULD LIKELY SET THIS IN PRODUCTION. Can also be set using the LIGHTBUS_SERVICE_NAME environment. Will default to a random string. (default: None) --process-name PROCESS_NAME, -p PROCESS_NAME A unique name of this process within the service. Can also be set using the LIGHTBUS_PROCESS_NAME environment. Will default to a random string. (default: None) --config FILE Config file to load, JSON or YAML (default: None) --log-level LOG_LEVEL Set the log level. Overrides any value set in config. One of debug, info, warning, critical, exception. (default: None)","title":"shell"},{"location":"reference/command-line-use/shell/#lightbus-shell","text":"The lightbus shell command provides an interactive prompt through which you can interface with the bus. To use this command you must first install bpython : pip install bpython","title":"lightbus shell"},{"location":"reference/command-line-use/shell/#examples","text":"You should see the following when starting up the shell. This is a fully functional Python shell, with your bus loaded in a ready to be used: $ lightbus shell >>> \u2588 Welcome to the Lightbus shell. Use `bus` to access your bus.\\ Upon typing the shell will begin to auto-complete based on the locally available APIs: $ lightbus shell >>> bus.au\u2588 \u250c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2510 \u2502 auth \u2502 \u2514\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2518 You can fire events and call RPCs as follows: # Fire an event >>> bus.auth.user_registered.fire(email=\"joe@example.com\", username=\"joe\") # Call an RPC >>> bus.auth.check_password(username=\"admin\", password=\"secret\") True","title":"Examples"},{"location":"reference/command-line-use/shell/#option-reference","text":"$ lightbus shell --help usage: lightbus shell [-h] [--bus BUS_MODULE] [--service-name SERVICE_NAME] [--process-name PROCESS_NAME] [--config FILE] [--log-level LOG_LEVEL] optional arguments: -h, --help show this help message and exit Common arguments: --bus BUS_MODULE, -b BUS_MODULE The bus module to import. Example 'bus', 'my_project.bus'. Defaults to the value of the LIGHTBUS_MODULE environment variable, or 'bus' (default: None) --service-name SERVICE_NAME, -s SERVICE_NAME Name of service in which this process resides. YOU SHOULD LIKELY SET THIS IN PRODUCTION. Can also be set using the LIGHTBUS_SERVICE_NAME environment. Will default to a random string. (default: None) --process-name PROCESS_NAME, -p PROCESS_NAME A unique name of this process within the service. Can also be set using the LIGHTBUS_PROCESS_NAME environment. Will default to a random string. (default: None) --config FILE Config file to load, JSON or YAML (default: None) --log-level LOG_LEVEL Set the log level. Overrides any value set in config. One of debug, info, warning, critical, exception. (default: None)","title":"Option reference"},{"location":"reference/protocols/","text":"Protocols \u00b6 Here we define the specific interactions between Lightbus and its underlying communication medium, Redis. The intention is to provide enough information to allow services written in other languages to interact with Lightbus. Event protocol (Redis) RPC & result protocol (Redis) Schema protocol (Redis)","title":"Overview"},{"location":"reference/protocols/#protocols","text":"Here we define the specific interactions between Lightbus and its underlying communication medium, Redis. The intention is to provide enough information to allow services written in other languages to interact with Lightbus. Event protocol (Redis) RPC & result protocol (Redis) Schema protocol (Redis)","title":"Protocols"},{"location":"reference/protocols/event/","text":"Event protocol (Redis) \u00b6 Here we document the specific interactions between Lightbus and Redis. The concrete implementation of this is provided by the RedisEventTransport class. Before reading you should be familiar with Lightbus' data marshalling . This documentation may be useful when debugging, developing third-party client libraries, or simply for general interest and review. You do not need to be aware of this protocol in order to use Lightbus . Sending events \u00b6 The following command will send an event: XADD {stream_name} [MAXLEN ~ {max_stream_length}] * {field_name_1} {field_value_1} {field_name_2} {field_value_2}... {stream_name} \u00b6 The {stream_name} value is composed in one of the following ways: One stream per event: {api_name}.{event_name} One stream per API: {api_name}.* MAXLEN \u00b6 The MAXLEN ~ {max_stream_length} component is optional, but will be used by Lightbus to limit the stream to the approximate configured length. Fields \u00b6 Field names are strings. Field values are JSON-encoded strings. The following metadata fields must be sent: :id - A unique message ID :api_name - The API name for this event :event_name - The name of this event :version \u2013 The version of this event ( 1 is a sensible default value) Note that metadata fields are prefixed by a colon. User-specified fields should not include this colon. Lightbus does not currently provide specific functionality around the version field, but the field is available to developers via the EventMessage class. Lightbus may implement event functionality around event versions in future (such as event migrations). Consuming events \u00b6 Consuming events involves an initial setup stage in which we check for any events which this process has consumed yet failed to process (for example, due to an error, hardware failure, network problem, etc). We perform this initial check for events as follows: XREAD_GROUP {group_name} {consumer_name} {stream_name} 0 The 0 above indicates we wish to receive un-acknowledged events for this consumer (i.e this Lightbus process). Once we have received and processed any of these events, we can retrieve further events as follows: XREAD_GROUP {group_name} {consumer_name} {stream} > {group_name} \u00b6 The {group_name} value is comprised of the service name and listener name as follows: {service_name}-{listener_name} . {consumer_name} \u00b6 The {consumer_name} is set to the process name of the Lightbus process. {stream_name} \u00b6 The stream name, as described above in sending events . Reclaiming timed-out events \u00b6 Events can be considered timed if another Lightbus process has held onto them for too long. Any client consuming events should check for these from time to time. Timed out events can be claimed as follows: # Get pending messages XPENDING {stream} {group_name} - + {batch_size} # For each message try to claim it XCLAIM {stream} {group_name} {timeout} # If successful, process the event. Otherwise ignore it {batch_size} \u00b6 How many pending messages to fetch in each batch {timeout} \u00b6 Timeout in milliseconds. This is the maximum time a Lightbus process will have to process an event before another processes assumes it has failed and takes over. Encoding & Customisation \u00b6 See also: data marshalling By default field values are serialised using JSON. This can be This encoding is customisable within the Lightbus configuration. You are welcome to use something custom here, but be aware that: A single API must have a single encoding for all events on that API All clients accessing the bus must be configured to use the same custom encoding Data validation \u00b6 Validation of outgoing and incoming events is optional. However, validation of outgoing events is recommended as sending of event messages which fail validation may result in the message being rejected by any consumer. This validation can be performed using the using the schema available through the schema protocol . Data deformation & casting \u00b6 The Lightbus client provides Python-specific functionality to streamline the process of moving between Python data structures and interoperable JSON data structures. The level of functionality required in this regard is not specified here, and is left up to individual library developers.","title":"Event Protocol (Redis)"},{"location":"reference/protocols/event/#event-protocol-redis","text":"Here we document the specific interactions between Lightbus and Redis. The concrete implementation of this is provided by the RedisEventTransport class. Before reading you should be familiar with Lightbus' data marshalling . This documentation may be useful when debugging, developing third-party client libraries, or simply for general interest and review. You do not need to be aware of this protocol in order to use Lightbus .","title":"Event protocol (Redis)"},{"location":"reference/protocols/event/#sending-events","text":"The following command will send an event: XADD {stream_name} [MAXLEN ~ {max_stream_length}] * {field_name_1} {field_value_1} {field_name_2} {field_value_2}...","title":"Sending events"},{"location":"reference/protocols/event/#stream_name","text":"The {stream_name} value is composed in one of the following ways: One stream per event: {api_name}.{event_name} One stream per API: {api_name}.*","title":"{stream_name}"},{"location":"reference/protocols/event/#maxlen","text":"The MAXLEN ~ {max_stream_length} component is optional, but will be used by Lightbus to limit the stream to the approximate configured length.","title":"MAXLEN"},{"location":"reference/protocols/event/#fields","text":"Field names are strings. Field values are JSON-encoded strings. The following metadata fields must be sent: :id - A unique message ID :api_name - The API name for this event :event_name - The name of this event :version \u2013 The version of this event ( 1 is a sensible default value) Note that metadata fields are prefixed by a colon. User-specified fields should not include this colon. Lightbus does not currently provide specific functionality around the version field, but the field is available to developers via the EventMessage class. Lightbus may implement event functionality around event versions in future (such as event migrations).","title":"Fields"},{"location":"reference/protocols/event/#consuming-events","text":"Consuming events involves an initial setup stage in which we check for any events which this process has consumed yet failed to process (for example, due to an error, hardware failure, network problem, etc). We perform this initial check for events as follows: XREAD_GROUP {group_name} {consumer_name} {stream_name} 0 The 0 above indicates we wish to receive un-acknowledged events for this consumer (i.e this Lightbus process). Once we have received and processed any of these events, we can retrieve further events as follows: XREAD_GROUP {group_name} {consumer_name} {stream} >","title":"Consuming events"},{"location":"reference/protocols/event/#group_name","text":"The {group_name} value is comprised of the service name and listener name as follows: {service_name}-{listener_name} .","title":"{group_name}"},{"location":"reference/protocols/event/#consumer_name","text":"The {consumer_name} is set to the process name of the Lightbus process.","title":"{consumer_name}"},{"location":"reference/protocols/event/#stream_name_1","text":"The stream name, as described above in sending events .","title":"{stream_name}"},{"location":"reference/protocols/event/#reclaiming-timed-out-events","text":"Events can be considered timed if another Lightbus process has held onto them for too long. Any client consuming events should check for these from time to time. Timed out events can be claimed as follows: # Get pending messages XPENDING {stream} {group_name} - + {batch_size} # For each message try to claim it XCLAIM {stream} {group_name} {timeout} # If successful, process the event. Otherwise ignore it","title":"Reclaiming timed-out events"},{"location":"reference/protocols/event/#batch_size","text":"How many pending messages to fetch in each batch","title":"{batch_size}"},{"location":"reference/protocols/event/#timeout","text":"Timeout in milliseconds. This is the maximum time a Lightbus process will have to process an event before another processes assumes it has failed and takes over.","title":"{timeout}"},{"location":"reference/protocols/event/#encoding-customisation","text":"See also: data marshalling By default field values are serialised using JSON. This can be This encoding is customisable within the Lightbus configuration. You are welcome to use something custom here, but be aware that: A single API must have a single encoding for all events on that API All clients accessing the bus must be configured to use the same custom encoding","title":"Encoding & Customisation"},{"location":"reference/protocols/event/#data-validation","text":"Validation of outgoing and incoming events is optional. However, validation of outgoing events is recommended as sending of event messages which fail validation may result in the message being rejected by any consumer. This validation can be performed using the using the schema available through the schema protocol .","title":"Data validation"},{"location":"reference/protocols/event/#data-deformation-casting","text":"The Lightbus client provides Python-specific functionality to streamline the process of moving between Python data structures and interoperable JSON data structures. The level of functionality required in this regard is not specified here, and is left up to individual library developers.","title":"Data deformation & casting"},{"location":"reference/protocols/rpc-and-result/","text":"RPC & result protocol (Redis) \u00b6 Here we document the specific interactions between Lightbus and Redis. The concrete implementation of this is provided by the RedisRpcTransport and RedisResultTransport classes. Before reading you should be familiar with Lightbus' data marshalling . This documentation may be useful when debugging, developing third-party client libraries, or simply for general interest and review. You do not need to be aware of this protocol in order to use Lightbus . Sending RPC Calls (client) \u00b6 The following Redis commands will send a remote procedure call: RPUSH \"{api_name}:rpc_queue\" \"{blob_serialized_message}\" SET \"rpc_expiry_key:{rpc_call_message_id}\" EXPIRE \"rpc_expiry_key:{rpc_call_message_id}\" {rpc_timeout_seconds:5} See message serialisation & encoding for the format of {blob_serialized_message} . The return path \u00b6 Each RPC message must specify a return_path in its metadata. This states how the client expects to receive the result of the RPC. Both the client and the server must be able to comprehend and act up the provided return path, otherwise results will fail to be communicated. Assuming you are using the built-in Redis RPC result transport, the return path should be in the following format: redis+key://{api_name}.{rpc_name}:result:{rpc_call_message_id} The server will place the RPC result into a Redis key. The value following redis+key:// will be used as the key name. Receiving RPC Results (client) \u00b6 The following Redis commands will block and await the result of an RPC call: BLPOP \"{api_name}.{rpc_name}:result:{rpc_call_message_id}\" This simply waits from a value to appear within the key specified by the return_path in the sent RPC message. See message serialisation & encoding for the format of the returned RPC result. Consuming incoming RPC calls (worker) \u00b6 RPCs are consumed as follows: # Blocks until a RPC message is received BLPOP \"{api_name}:rpc_queue\" DEL \"rpc_expiry_key:{rpc_call_message_id}\" # If DEL returns 1 (key deleted) then execute the RPC # If DEL returns 0 (key did not exist) then ignore the RPC # Parse blob-serialised RPC message, Execute RPC, get result LPUSH {redis_key_specific_in_return_path} {blob_serialized_result} EXPIRE {redis_key_specific_in_return_path} {result_ttl_seconds} Message serialisation & encoding \u00b6 Above we have often referred to {blob_serialized_message} or {blob_serialized_result} . These are JSON blobs of data in a specific format. Serialisation \u00b6 See also: data marshalling Within the Redis RPC transports all messages are serialised into a single value. This value is referred to as a 'blob'. This serialisation is performed by the BlobMessageSerializer and BlobMessageDeserializer classes. RPC Message \u00b6 This is an outgoing RPC message body. This RPC message represents a request that an RPC be executed and a response be returned. { \"metadata\" : { # Unique base64-encoded UUID \"id\" : \"KrXz5EUXEem2gazeSAARIg==\" , # Fully qualified API name \"api_name\" : \"my_company.auth\" , # The name of the remote procedure call \"procedure_name\" : \"check_password\" , # How and where should the result be sent \"return_path\" : \"redis+key://my_company.auth.check_password:result:KrXz5EUXEem2gazeSAARIg==\" , }, # Key/value arguments potentiually needed to execute the remote procedure call \"kwargs\" : { \"username\" : \"adam\" , \"password\" : \"secret\" , ... } } Result Message \u00b6 This is the response message indicating an RPC has been (successfully or unsuccessfully) executed. { \"metadata\" : { # The newly randomly generated ID of this RPC result message. # (base64-encoded UUID) \"id\" : \"L4iXaEUgEemnBazeSAARIg==\" , # The ID of the received RPC call message \"rpc_message_id\" : \"KrXz5EUXEem2gazeSAARIg==\" , # String representation of any error which occurred \"error\" : \"\" , # Error stack trace. Only used in event of error, may be omitted otherwise \"trace\" : \" {error_stack_trace} \" , }, # The result as returned by the executed RPC \"result\" : ... } Encoding & Customisation \u00b6 See also: data marshalling The message must be encoded as a string once it has been serialised to the above blob structure. The default implementation is to use JSON as this encoding. This encoding is customisable within the Lightbus configuration. You are welcome to use something custom here, but be aware that: A single API must have a single encoding for all RPCs on that API All clients accessing the bus must be configured to use the same custom encoding Data validation \u00b6 Validation of outgoing parameters and incoming results is optional. However, validation of outgoing parameters is recommended as sending of RPC messages which fail validation may result in the message being rejected by the Lightbus worker. This validation can be performed using the using the schema available through the schema protocol . Data deformation & casting \u00b6 The Lightbus client provides Python-specific functionality to streamline the process of moving between Python data structures and interoperable JSON data structures. The level of functionality required in this regard is not specified here, and is left up to individual library developers.","title":"RPC & Result Protocol (Redis)"},{"location":"reference/protocols/rpc-and-result/#rpc-result-protocol-redis","text":"Here we document the specific interactions between Lightbus and Redis. The concrete implementation of this is provided by the RedisRpcTransport and RedisResultTransport classes. Before reading you should be familiar with Lightbus' data marshalling . This documentation may be useful when debugging, developing third-party client libraries, or simply for general interest and review. You do not need to be aware of this protocol in order to use Lightbus .","title":"RPC & result protocol (Redis)"},{"location":"reference/protocols/rpc-and-result/#sending-rpc-calls-client","text":"The following Redis commands will send a remote procedure call: RPUSH \"{api_name}:rpc_queue\" \"{blob_serialized_message}\" SET \"rpc_expiry_key:{rpc_call_message_id}\" EXPIRE \"rpc_expiry_key:{rpc_call_message_id}\" {rpc_timeout_seconds:5} See message serialisation & encoding for the format of {blob_serialized_message} .","title":"Sending RPC Calls (client)"},{"location":"reference/protocols/rpc-and-result/#the-return-path","text":"Each RPC message must specify a return_path in its metadata. This states how the client expects to receive the result of the RPC. Both the client and the server must be able to comprehend and act up the provided return path, otherwise results will fail to be communicated. Assuming you are using the built-in Redis RPC result transport, the return path should be in the following format: redis+key://{api_name}.{rpc_name}:result:{rpc_call_message_id} The server will place the RPC result into a Redis key. The value following redis+key:// will be used as the key name.","title":"The return path"},{"location":"reference/protocols/rpc-and-result/#receiving-rpc-results-client","text":"The following Redis commands will block and await the result of an RPC call: BLPOP \"{api_name}.{rpc_name}:result:{rpc_call_message_id}\" This simply waits from a value to appear within the key specified by the return_path in the sent RPC message. See message serialisation & encoding for the format of the returned RPC result.","title":"Receiving RPC Results (client)"},{"location":"reference/protocols/rpc-and-result/#consuming-incoming-rpc-calls-worker","text":"RPCs are consumed as follows: # Blocks until a RPC message is received BLPOP \"{api_name}:rpc_queue\" DEL \"rpc_expiry_key:{rpc_call_message_id}\" # If DEL returns 1 (key deleted) then execute the RPC # If DEL returns 0 (key did not exist) then ignore the RPC # Parse blob-serialised RPC message, Execute RPC, get result LPUSH {redis_key_specific_in_return_path} {blob_serialized_result} EXPIRE {redis_key_specific_in_return_path} {result_ttl_seconds}","title":"Consuming incoming RPC calls (worker)"},{"location":"reference/protocols/rpc-and-result/#message-serialisation-encoding","text":"Above we have often referred to {blob_serialized_message} or {blob_serialized_result} . These are JSON blobs of data in a specific format.","title":"Message serialisation & encoding"},{"location":"reference/protocols/rpc-and-result/#serialisation","text":"See also: data marshalling Within the Redis RPC transports all messages are serialised into a single value. This value is referred to as a 'blob'. This serialisation is performed by the BlobMessageSerializer and BlobMessageDeserializer classes.","title":"Serialisation"},{"location":"reference/protocols/rpc-and-result/#rpc-message","text":"This is an outgoing RPC message body. This RPC message represents a request that an RPC be executed and a response be returned. { \"metadata\" : { # Unique base64-encoded UUID \"id\" : \"KrXz5EUXEem2gazeSAARIg==\" , # Fully qualified API name \"api_name\" : \"my_company.auth\" , # The name of the remote procedure call \"procedure_name\" : \"check_password\" , # How and where should the result be sent \"return_path\" : \"redis+key://my_company.auth.check_password:result:KrXz5EUXEem2gazeSAARIg==\" , }, # Key/value arguments potentiually needed to execute the remote procedure call \"kwargs\" : { \"username\" : \"adam\" , \"password\" : \"secret\" , ... } }","title":"RPC Message"},{"location":"reference/protocols/rpc-and-result/#result-message","text":"This is the response message indicating an RPC has been (successfully or unsuccessfully) executed. { \"metadata\" : { # The newly randomly generated ID of this RPC result message. # (base64-encoded UUID) \"id\" : \"L4iXaEUgEemnBazeSAARIg==\" , # The ID of the received RPC call message \"rpc_message_id\" : \"KrXz5EUXEem2gazeSAARIg==\" , # String representation of any error which occurred \"error\" : \"\" , # Error stack trace. Only used in event of error, may be omitted otherwise \"trace\" : \" {error_stack_trace} \" , }, # The result as returned by the executed RPC \"result\" : ... }","title":"Result Message"},{"location":"reference/protocols/rpc-and-result/#encoding-customisation","text":"See also: data marshalling The message must be encoded as a string once it has been serialised to the above blob structure. The default implementation is to use JSON as this encoding. This encoding is customisable within the Lightbus configuration. You are welcome to use something custom here, but be aware that: A single API must have a single encoding for all RPCs on that API All clients accessing the bus must be configured to use the same custom encoding","title":"Encoding & Customisation"},{"location":"reference/protocols/rpc-and-result/#data-validation","text":"Validation of outgoing parameters and incoming results is optional. However, validation of outgoing parameters is recommended as sending of RPC messages which fail validation may result in the message being rejected by the Lightbus worker. This validation can be performed using the using the schema available through the schema protocol .","title":"Data validation"},{"location":"reference/protocols/rpc-and-result/#data-deformation-casting","text":"The Lightbus client provides Python-specific functionality to streamline the process of moving between Python data structures and interoperable JSON data structures. The level of functionality required in this regard is not specified here, and is left up to individual library developers.","title":"Data deformation & casting"},{"location":"reference/protocols/schema/","text":"Schema protocol (Redis) \u00b6 Here we document the specific interactions between Lightbus and Redis. The concrete implementation of this is provided by the RedisSchemaTransport class. Before reading you should be familiar with the schema explanation . This documentation may be useful when debugging, developing third-party client libraries, or simply for general interest and review. You do not need to be aware of this protocol in order to use Lightbus . Example API \u00b6 We will use the following API within the examples in the remainder of this page: from lightbus import Api , Event , Parameter class AuthApi ( Api ): user_registered = Event ( parameters = ( Parameter ( 'username' , str ), Parameter ( 'email' , str ), Parameter ( 'is_admin' , bool , default = False ), )) class Meta : name = 'my_company.auth' def check_password ( self , username : str , password : str ) -> bool : return username == 'admin' and password == 'secret' Schema format \u00b6 The Lightbus schema format is a JSON structure containing multiple JSON schemas. Below is the schema for the example my_company.auth API shown above (generalised format follows): // Auto-generated schema for auth API { \"my_company.auth\" : { // Events specify only parameters \"events\" : { \"user_registered\" : { \"parameters\" : { \"$schema\" : \"http://json-schema.org/draft-07/schema#\" , \"title\" : \"Event my_company.auth.user_registered parameters\" , \"type\" : \"object\" , \"username\" : { \"type\" : \"string\" }, \"properties\" : { \"email\" : { \"type\" : \"string\" }, \"is_admin\" : { \"default\" : false , \"type\" : \"boolean\" } }, \"required\" : [ \"username\" , \"email\" ], \"additionalProperties\" : false } } }, // RPCs specify both parameters and response \"rpcs\" : { \"check_password\" : { \"parameters\" : { \"$schema\" : \"http://json-schema.org/draft-07/schema#\" , \"title\" : \"RPC my_company.auth.check_password() parameters\" , \"type\" : \"object\" , \"properties\" : { \"username\" : { \"type\" : \"string\" }, \"password\" : { \"type\" : \"string\" } }, \"required\" : [ \"username\" , \"password\" ], \"additionalProperties\" : false }, \"response\" : { \"$schema\" : \"http://json-schema.org/draft-07/schema#\" , \"title\" : \"RPC my_company.auth.check_password() response\" , \"type\" : \"boolean\" } } } } } The generalised format is as follows: // Generalised Lightbus schema format { \"\" : { \"events\" : { \"\" : { \"parameters\" : { /* json schema */ } } // additional events }, \"rpcs\" : { \"\" : { \"parameters\" : { /* json schema */ }, \"response\" : { /* json schema */ } } // additional procedures } } // additional APIs } Per-schema Redis key \u00b6 Each schema is stored in redis as a string-ified JSON blob using a key formed as follows: schema:{api_name} For example: schema:my_company.auth Storing schemas \u00b6 A schema can be stored on the bus as follows: SET \"schema:{api_name}\" \"{json_schema}\" SADD \"schemas\" \"{api_name}\" EXPIRE \"schema:{api_name}\" \"{json_schema}\" {ttl_seconds} Where: {api_name} is replaced with the fully qualified API name (e.g. my_company.auth ) {json_schema} is replaced with the string-ified JSON structure detailed above {ttl_seconds} is replaced with the maximum time the schema should persist before being expired. Schemas expire to ensure shutdown Lightbus processes no longer advertise their APIs. Lightbus has a default value of 60 seconds. This process must be repeated at least every ttl_seconds in order to keep the schema active on the bus. Loading schemas \u00b6 Available schemas are loaded as follows: schemas = SMEMBERS \"schemas\" for api_name in scheams: GET \"schema:{api_name}\" A schema is only available if the GET succeeds. The data returned by the GET should be JSON decoded, and will contain the structure detailed above.","title":"Schema Protocol (Redis)"},{"location":"reference/protocols/schema/#schema-protocol-redis","text":"Here we document the specific interactions between Lightbus and Redis. The concrete implementation of this is provided by the RedisSchemaTransport class. Before reading you should be familiar with the schema explanation . This documentation may be useful when debugging, developing third-party client libraries, or simply for general interest and review. You do not need to be aware of this protocol in order to use Lightbus .","title":"Schema protocol (Redis)"},{"location":"reference/protocols/schema/#example-api","text":"We will use the following API within the examples in the remainder of this page: from lightbus import Api , Event , Parameter class AuthApi ( Api ): user_registered = Event ( parameters = ( Parameter ( 'username' , str ), Parameter ( 'email' , str ), Parameter ( 'is_admin' , bool , default = False ), )) class Meta : name = 'my_company.auth' def check_password ( self , username : str , password : str ) -> bool : return username == 'admin' and password == 'secret'","title":"Example API"},{"location":"reference/protocols/schema/#schema-format","text":"The Lightbus schema format is a JSON structure containing multiple JSON schemas. Below is the schema for the example my_company.auth API shown above (generalised format follows): // Auto-generated schema for auth API { \"my_company.auth\" : { // Events specify only parameters \"events\" : { \"user_registered\" : { \"parameters\" : { \"$schema\" : \"http://json-schema.org/draft-07/schema#\" , \"title\" : \"Event my_company.auth.user_registered parameters\" , \"type\" : \"object\" , \"username\" : { \"type\" : \"string\" }, \"properties\" : { \"email\" : { \"type\" : \"string\" }, \"is_admin\" : { \"default\" : false , \"type\" : \"boolean\" } }, \"required\" : [ \"username\" , \"email\" ], \"additionalProperties\" : false } } }, // RPCs specify both parameters and response \"rpcs\" : { \"check_password\" : { \"parameters\" : { \"$schema\" : \"http://json-schema.org/draft-07/schema#\" , \"title\" : \"RPC my_company.auth.check_password() parameters\" , \"type\" : \"object\" , \"properties\" : { \"username\" : { \"type\" : \"string\" }, \"password\" : { \"type\" : \"string\" } }, \"required\" : [ \"username\" , \"password\" ], \"additionalProperties\" : false }, \"response\" : { \"$schema\" : \"http://json-schema.org/draft-07/schema#\" , \"title\" : \"RPC my_company.auth.check_password() response\" , \"type\" : \"boolean\" } } } } } The generalised format is as follows: // Generalised Lightbus schema format { \"\" : { \"events\" : { \"\" : { \"parameters\" : { /* json schema */ } } // additional events }, \"rpcs\" : { \"\" : { \"parameters\" : { /* json schema */ }, \"response\" : { /* json schema */ } } // additional procedures } } // additional APIs }","title":"Schema format"},{"location":"reference/protocols/schema/#per-schema-redis-key","text":"Each schema is stored in redis as a string-ified JSON blob using a key formed as follows: schema:{api_name} For example: schema:my_company.auth","title":"Per-schema Redis key"},{"location":"reference/protocols/schema/#storing-schemas","text":"A schema can be stored on the bus as follows: SET \"schema:{api_name}\" \"{json_schema}\" SADD \"schemas\" \"{api_name}\" EXPIRE \"schema:{api_name}\" \"{json_schema}\" {ttl_seconds} Where: {api_name} is replaced with the fully qualified API name (e.g. my_company.auth ) {json_schema} is replaced with the string-ified JSON structure detailed above {ttl_seconds} is replaced with the maximum time the schema should persist before being expired. Schemas expire to ensure shutdown Lightbus processes no longer advertise their APIs. Lightbus has a default value of 60 seconds. This process must be repeated at least every ttl_seconds in order to keep the schema active on the bus.","title":"Storing schemas"},{"location":"reference/protocols/schema/#loading-schemas","text":"Available schemas are loaded as follows: schemas = SMEMBERS \"schemas\" for api_name in scheams: GET \"schema:{api_name}\" A schema is only available if the GET succeeds. The data returned by the GET should be JSON decoded, and will contain the structure detailed above.","title":"Loading schemas"},{"location":"tutorial/","text":"Tutorial overview \u00b6 These tutorials will give you a practical concrete introduction to Lightbus . We will link to concepts as we go, but the aim here is to get you up and running quickly. Do you prefer to read the theory first? Feel free to start with the explanation section and come back here later. We recommend you approach the tutorials in the following order: Installation Quick start Worked example After completing these tutorials you should make sure you look over the explanation section.","title":"Overview"},{"location":"tutorial/#tutorial-overview","text":"These tutorials will give you a practical concrete introduction to Lightbus . We will link to concepts as we go, but the aim here is to get you up and running quickly. Do you prefer to read the theory first? Feel free to start with the explanation section and come back here later. We recommend you approach the tutorials in the following order: Installation Quick start Worked example After completing these tutorials you should make sure you look over the explanation section.","title":"Tutorial overview"},{"location":"tutorial/django-and-lightbus/","text":"If you get stuck... It is really useful to hear from people who have encountered a problem or got stuck. Hearing from you means we can improve our documentation and error messages. If you get stuck drop then please drop an email to adam@adamcharnock.com , visit the Lightbus discord server , or call me (Adam) on +442032896620. The more information you can include the better (problem description, screenshots, and code are all useful) Lightbus is designed to work smoothly with Django. A few of additions are needed to your bus.py file in order to setup django correctly. You should perform these at the very start of your @bus.client.on_start() function within your bus.py : Set your DJANGO_SETTINGS_MODULE environment variable 1 Call django.setup() Decorate RPCs and handlers which use the Django ORM with @uses_django_db . This will ensure database errors are cleaned up correctly. Simple example \u00b6 The following example demonstrates: Setting DJANGO_SETTINGS_MODULE Calling django.setup() # bus.py import lightbus import django import os # Do NOT import your django models here bus = lightbus . create () # ...Define and register APIs here... @bus . client . on_start () def on_start ( ** kwargs ): # DJANGO STEP 1: Set DJANGO_SETTINGS_MODULE\\ # Customise this value (if necessary) to point to your settings module. # This should be the same as the corresponding line in your manage.py file. os . environ . setdefault ( \"DJANGO_SETTINGS_MODULE\" , \"settings\" ) # DJANGO STEP 2: Call django.setup() # We must do this before we import any of our django models django . setup () # You can now safely import your django models here if needed, # or within any event handlers we setup (see next example) from my_app.models import MyModel Practical example \u00b6 Here we build upon the simple example (above). This example includes an API and an event handler, and is loosely based upon our work in the quick start tutorial . Here we demonstrate: Setting DJANGO_SETTINGS_MODULE Calling django.setup() Applying @uses_django_db to an RPC and event handler import lightbus from lightbus.utilities.django import uses_django_db import django import os bus = lightbus . create () class AuthApi ( lightbus . Api ): user_registered = lightbus . Event ( parameters = ( 'username' , 'email' )) class Meta : name = 'auth' # Decorating our RPC with @uses_django_db # ensures database connections are cleaned up @uses_django_db def check_password ( self , username , password ): # Import django models here, once django.setup() has been called from django.contrib.auth.models import User try : user = User . objects . get ( username = username ) return user . check_password ( password ) except User . DoesNotExist : return False bus . client . register_api ( AuthApi ()) # Decorating our event handler with @uses_django_db # ensures database connections are cleaned up @uses_django_db def handle_new_user ( event , username , email ): # For example, we may want to create an audit log entry from my_app.models import AuditLogEntry AuditLogEntry . objects . create ( message = f \"User { username } created\" ) @bus . client . on_start () def on_start ( ** kwargs ): os . environ . setdefault ( \"DJANGO_SETTINGS_MODULE\" , \"my_company.a_project.settings\" ) django . setup () # Setup a listener for the 'auth.user_registered' event, to be # handled by handle_new_user() (above) bus . auth . user_registered . listen ( handle_new_user , listener_name = \"create_audit_entry\" ) Further development Simplification to the Django setup process is tracked in GitHub . You will see that your Django project' manage.py file also performs this same step \u21a9","title":"4. Django & Lightbus"},{"location":"tutorial/django-and-lightbus/#simple-example","text":"The following example demonstrates: Setting DJANGO_SETTINGS_MODULE Calling django.setup() # bus.py import lightbus import django import os # Do NOT import your django models here bus = lightbus . create () # ...Define and register APIs here... @bus . client . on_start () def on_start ( ** kwargs ): # DJANGO STEP 1: Set DJANGO_SETTINGS_MODULE\\ # Customise this value (if necessary) to point to your settings module. # This should be the same as the corresponding line in your manage.py file. os . environ . setdefault ( \"DJANGO_SETTINGS_MODULE\" , \"settings\" ) # DJANGO STEP 2: Call django.setup() # We must do this before we import any of our django models django . setup () # You can now safely import your django models here if needed, # or within any event handlers we setup (see next example) from my_app.models import MyModel","title":"Simple example"},{"location":"tutorial/django-and-lightbus/#practical-example","text":"Here we build upon the simple example (above). This example includes an API and an event handler, and is loosely based upon our work in the quick start tutorial . Here we demonstrate: Setting DJANGO_SETTINGS_MODULE Calling django.setup() Applying @uses_django_db to an RPC and event handler import lightbus from lightbus.utilities.django import uses_django_db import django import os bus = lightbus . create () class AuthApi ( lightbus . Api ): user_registered = lightbus . Event ( parameters = ( 'username' , 'email' )) class Meta : name = 'auth' # Decorating our RPC with @uses_django_db # ensures database connections are cleaned up @uses_django_db def check_password ( self , username , password ): # Import django models here, once django.setup() has been called from django.contrib.auth.models import User try : user = User . objects . get ( username = username ) return user . check_password ( password ) except User . DoesNotExist : return False bus . client . register_api ( AuthApi ()) # Decorating our event handler with @uses_django_db # ensures database connections are cleaned up @uses_django_db def handle_new_user ( event , username , email ): # For example, we may want to create an audit log entry from my_app.models import AuditLogEntry AuditLogEntry . objects . create ( message = f \"User { username } created\" ) @bus . client . on_start () def on_start ( ** kwargs ): os . environ . setdefault ( \"DJANGO_SETTINGS_MODULE\" , \"my_company.a_project.settings\" ) django . setup () # Setup a listener for the 'auth.user_registered' event, to be # handled by handle_new_user() (above) bus . auth . user_registered . listen ( handle_new_user , listener_name = \"create_audit_entry\" ) Further development Simplification to the Django setup process is tracked in GitHub . You will see that your Django project' manage.py file also performs this same step \u21a9","title":"Practical example"},{"location":"tutorial/getting-involved/","text":"Getting involved (even if you don't code) \u00b6 Lightbus can only thrive as a project if it has new contributors getting involved, this means we need you! \"But I've never contributed to open source before,\" I hear you cry. Well, it turns out most people have never contributed to open source before either, so don't worry, you're not alone. The Lightbus team (which currently consists of me, Adam) is here to support you. Code of Conduct \u00b6 Make sure you read over the Code of Conduct . This will outline how you should engage with others, and how you should expect to be treated yourself. How to get started (non-coding) \u00b6 There are lots of ways you can get even without getting stuck into code. Even if you are a Python developer, this can still be a good place to start: Say hello in our community chat . We can have a chat about ways in which you can help. Raise an issue and describe something you find confusing about Lightbus. Something you think is not clear or missing from the documentation. Submit a pull request to improve the documentation (for example, I'm sure you can find some typos) Tell me how you are using (or would like to use) Lightbus. This kind of insight is invaluable. How to get started (coding) \u00b6 If you want to get stuck in to writing some code here are some good places to start: See how to modify Lightbus for a guide to getting your local development environment setup. If you have a specific idea or bug you would like to fix then come and discuss it , either in our community chat or in a GitHub issue . It would be great to meet you, and I may be able to advise you of any pitfalls to be aware of. Read over the code to get a feel for how things work. For a high-level view take a look at the BusClient class. For a low-level view of the messaging system take a look at the redis transports ( lightbus/transports/redis ). You'll likely find some typo's as you go, so feel free to submit a pull request ! Work on one of the project's issues . I recommend you talk your plan over me first, that way we can make sure you develop something that fits with the the project as a whole.","title":"Getting involved (even if you don't code)"},{"location":"tutorial/getting-involved/#getting-involved-even-if-you-dont-code","text":"Lightbus can only thrive as a project if it has new contributors getting involved, this means we need you! \"But I've never contributed to open source before,\" I hear you cry. Well, it turns out most people have never contributed to open source before either, so don't worry, you're not alone. The Lightbus team (which currently consists of me, Adam) is here to support you.","title":"Getting involved (even if you don't code)"},{"location":"tutorial/getting-involved/#code-of-conduct","text":"Make sure you read over the Code of Conduct . This will outline how you should engage with others, and how you should expect to be treated yourself.","title":"Code of Conduct"},{"location":"tutorial/getting-involved/#how-to-get-started-non-coding","text":"There are lots of ways you can get even without getting stuck into code. Even if you are a Python developer, this can still be a good place to start: Say hello in our community chat . We can have a chat about ways in which you can help. Raise an issue and describe something you find confusing about Lightbus. Something you think is not clear or missing from the documentation. Submit a pull request to improve the documentation (for example, I'm sure you can find some typos) Tell me how you are using (or would like to use) Lightbus. This kind of insight is invaluable.","title":"How to get started (non-coding)"},{"location":"tutorial/getting-involved/#how-to-get-started-coding","text":"If you want to get stuck in to writing some code here are some good places to start: See how to modify Lightbus for a guide to getting your local development environment setup. If you have a specific idea or bug you would like to fix then come and discuss it , either in our community chat or in a GitHub issue . It would be great to meet you, and I may be able to advise you of any pitfalls to be aware of. Read over the code to get a feel for how things work. For a high-level view take a look at the BusClient class. For a low-level view of the messaging system take a look at the redis transports ( lightbus/transports/redis ). You'll likely find some typo's as you go, so feel free to submit a pull request ! Work on one of the project's issues . I recommend you talk your plan over me first, that way we can make sure you develop something that fits with the the project as a whole.","title":"How to get started (coding)"},{"location":"tutorial/installation/","text":"If you get stuck... It is really useful to hear from people who have encountered a problem or got stuck. Hearing from you means we can improve our documentation and error messages. If you get stuck drop then please drop an email to adam@adamcharnock.com , visit the Lightbus discord server , or call me (Adam) on +442032896620. The more information you can include the better (problem description, screenshots, and code are all useful) 1.1. Preface: Installing Python 3.7 (or above) \u00b6 Lightbus requires Python 3.7 or newer. This is available for all major operating systems. Python 3.7 on macOS \u00b6 You can check your current version of Python as follows: $ python3 --version Python 3.7.2 You need version 3.7 or above to run Lightbus. If you are running an older version of Python you can install a newer version via one of the following methods: Install Python 3.7 using Homebrew \u2013 This is the easiest option, you will install that latest version of Python 3. Install Python 3.7 using Homebrew + pyenv \u2013 This option has some additional steps, but you will have complete control over the Python versions available to you. If you work on multiple Python projects this may be more suitable. Install Python 3.7 manually \u2013 Not recommended Python 3.7 on Linux \u00b6 Your Linux distribution may already come with Python 3.7 installed. You can check your Python version as follows: $ python3 --version Python 3.7.2 You need version 3.7 or above to run Lightbus. Digital Ocean has a beginner-suitable guide on installing Python 3 which you may find useful. If you require more granular control of your python versions you may find pyenv more suitable. Windows \u00b6 Lightbus is not currently tested for deployment on Windows, so your millage may vary. The Hitchhiker's Guide to Python covers installing Python 3 on Windows . 1.2. Installing Lightbus \u00b6 Installing using pip (recommended) \u00b6 $ pip3 install lightbus Installing using git \u00b6 This will clone the bleeding-edge version Lightbus and install it ready to use. This is useful if you need the latest (albeit unstable) changes, or if you wish to modify the Lightbus source. $ pip install https://github.com/adamcharnock/lightbus.git#egg=lightbus 1.3. Installing Redis \u00b6 You will need Redis 5.0 or above in order to use Lightbus. You can install Redis 5.0 on macOS by either: Using Homebrew ( brew install redis ), or Using docker ( docker run --rm -p 6379:6379 -d redis ) 1.4. Check it works \u00b6 You should now have: Python 3.7 or above installed Lightbus installed Redis installed and running You check check everything is setup correctly by starting up lightbus. First create a file name bus.py which contains: # Create a bus.py file with the following contents import lightbus bus = lightbus . create () Now run the command: $ lightbus run Lightbus should start without errors and wait for messages. You can exit using Ctrl + C . This Lightbus process isn't setup to do anything yet, but its a start! 1.5 Next \u00b6 Now you are up and running, take a look at the quick start tutorial .","title":"1. Installation"},{"location":"tutorial/installation/#11-preface-installing-python-37-or-above","text":"Lightbus requires Python 3.7 or newer. This is available for all major operating systems.","title":"1.1. Preface: Installing Python 3.7 (or above)"},{"location":"tutorial/installation/#python-37-on-macos","text":"You can check your current version of Python as follows: $ python3 --version Python 3.7.2 You need version 3.7 or above to run Lightbus. If you are running an older version of Python you can install a newer version via one of the following methods: Install Python 3.7 using Homebrew \u2013 This is the easiest option, you will install that latest version of Python 3. Install Python 3.7 using Homebrew + pyenv \u2013 This option has some additional steps, but you will have complete control over the Python versions available to you. If you work on multiple Python projects this may be more suitable. Install Python 3.7 manually \u2013 Not recommended","title":"Python 3.7 on macOS"},{"location":"tutorial/installation/#python-37-on-linux","text":"Your Linux distribution may already come with Python 3.7 installed. You can check your Python version as follows: $ python3 --version Python 3.7.2 You need version 3.7 or above to run Lightbus. Digital Ocean has a beginner-suitable guide on installing Python 3 which you may find useful. If you require more granular control of your python versions you may find pyenv more suitable.","title":"Python 3.7 on Linux"},{"location":"tutorial/installation/#windows","text":"Lightbus is not currently tested for deployment on Windows, so your millage may vary. The Hitchhiker's Guide to Python covers installing Python 3 on Windows .","title":"Windows"},{"location":"tutorial/installation/#12-installing-lightbus","text":"","title":"1.2. Installing Lightbus"},{"location":"tutorial/installation/#installing-using-pip-recommended","text":"$ pip3 install lightbus","title":"Installing using pip (recommended)"},{"location":"tutorial/installation/#installing-using-git","text":"This will clone the bleeding-edge version Lightbus and install it ready to use. This is useful if you need the latest (albeit unstable) changes, or if you wish to modify the Lightbus source. $ pip install https://github.com/adamcharnock/lightbus.git#egg=lightbus","title":"Installing using git"},{"location":"tutorial/installation/#13-installing-redis","text":"You will need Redis 5.0 or above in order to use Lightbus. You can install Redis 5.0 on macOS by either: Using Homebrew ( brew install redis ), or Using docker ( docker run --rm -p 6379:6379 -d redis )","title":"1.3. Installing Redis"},{"location":"tutorial/installation/#14-check-it-works","text":"You should now have: Python 3.7 or above installed Lightbus installed Redis installed and running You check check everything is setup correctly by starting up lightbus. First create a file name bus.py which contains: # Create a bus.py file with the following contents import lightbus bus = lightbus . create () Now run the command: $ lightbus run Lightbus should start without errors and wait for messages. You can exit using Ctrl + C . This Lightbus process isn't setup to do anything yet, but its a start!","title":"1.4. Check it works"},{"location":"tutorial/installation/#15-next","text":"Now you are up and running, take a look at the quick start tutorial .","title":"1.5 Next"},{"location":"tutorial/quick-start/","text":"If you get stuck... It is really useful to hear from people who have encountered a problem or got stuck. Hearing from you means we can improve our documentation and error messages. If you get stuck drop then please drop an email to adam@adamcharnock.com , visit the Lightbus discord server , or call me (Adam) on +442032896620. The more information you can include the better (problem description, screenshots, and code are all useful) 2.1 Requirements \u00b6 Before continuing, ensure you have completed the following steps detailed in the installation section : Installed Python 3.7 or above Installed Lightbus Running Redis 5 locally on the default port (6379) Optionally, you can read some additional explanation in the anatomy lesson section. 2.2 Define your API \u00b6 First we will define an API for Lightbus to serve. This will be an authentication API and will live in an authentication service. Create an auth_service directory and within there create the following in a bus.py file: # File: auth_service/bus.py import lightbus # Create your service's bus client. You can import this elsewere # in your service's codebase in order to access the bus bus = lightbus . create () class AuthApi ( lightbus . Api ): user_registered = lightbus . Event ( parameters = ( 'username' , 'email' )) class Meta : name = 'auth' def check_password ( self , username , password ): return username == 'admin' and password == 'secret' # Register this API with Lightbus. Lightbus will respond to # remote procedure calls for registered APIs, as well as allow you # as the developer to fire events on any registered APIs. bus . client . register_api ( AuthApi ()) You should now be able to startup Lightbus as follows: cd ./auth_service/ lightbus run Lightbus will output some logging data which will include a list of registered APIs, including your new auth API: Congratulations, you have now created an authentication service which contains a single API named auth . Leave Lightbus running and open a new terminal window for the next stage. 2.3 Remote procedure calls \u00b6 With Lightbus still running in another terminal window, create a new directory alongside auth_service called another_service . This will be an example service which will interact with the auth service. You should now have the following structure: ./auth_service/bus.py , created above ./another_service/ , which we will create files within now. We always define a service's bus client within a bus.py file. Therefore, create ./another_service/bus.py containing the following: # File: ./another_service/bus.py import lightbus bus = lightbus . create () Now create ./another_service/check_password.py . We will use this to experiment with making RPC calls: # File: ./another_service/check_password.py # Import our service's bus client from .bus import bus # Call the check_password() procedure on our auth API valid = bus . auth . check_password ( username = 'admin' , password = 'secret' ) # Show the result if valid : print ( 'Password valid!' ) else : print ( 'Oops, bad username or password' ) Running this script should show you the following: cd ./another_service/ python3 ./check_password.py Password valid! Looking at the other terminal window you have open you should see that Lightbus has also reported that it is handled a remote procedure call. 2.4 Events \u00b6 Events allow services to broadcast a message to any other services which care to listen. Your service can fire events on any API which has been registered. We have already created and registered our AuthApi in ./auth_service/bus.py which provides the user_registered event. Firing events \u00b6 Let's write a simple script to manually register users and fire events. Note we can only fire events for APIs we have registered. We have registered the auth API in auth_service , so it is within this service that this new script must reside. We could not put the script within another_service as this service does not include the AuthApi class and can therefore not register it. # ./auth_service/manually_register_user.py # Import the service's bus client from bus.py from bus import bus print ( \"New user creation\" ) new_username = input ( \"Enter a username: \" ) . strip () new_email = input ( \"Enter the user's email address: \" ) . strip () # You would normally store the new user in your database # at this point. We don't show this here for simplicity. # Let the bus know a user has been registered by firing the event bus . auth . user_registered . fire ( username = new_username , email = new_email ) print ( \"Done\" ) Run this script using: cd ./auth_service/ python3 manually_register_user.py You should be prompted for a new username & email address, at which point an event will be fired onto the bus. We will make use of this event in the next section. Listening for events \u00b6 Perhaps another_service needs to be notified when a user is created. To achieve this we can setup an event listener by modifying ./another_service/bus.py : # File: ./another_service/bus.py import lightbus bus = lightbus . create () def handle_new_user ( event , username , email ): print ( f \"A new user was created in the authentication service:\" ) print ( f \" Username: { username } \" ) print ( f \" Email: { email } \" ) @bus . client . on_start () def bus_start ( client ): bus . auth . user_registered . listen ( handle_new_user , listener_name = \"print_on_new_registration\" ) Listening for events requires your service to sit waiting for something to happen. Sitting around and waiting for something to happen is precisely what the lightbus run console command is for. Therefore, in one terminal window startup the lightbus run command for another_service : # In the first terminal window: cd ./another_service/ lightbus run Your service is now waiting for events. In the second terminal window let's cause an event to be fired using the script we wrote in the previous section: cd ./auth_service/ python3 manually_register_user.py You should see that the event gets sent by the manually_register_user.py script within the auth_service , and received by by the lightbus run process within the another_service . Listener names \u00b6 You may have noticed we specified the listener_name parameter in the above example, and we set it to an arbitrary value of \"print_on_new_registration\" . This is important, because Lightbus tracks which listeners have received which events. It is this listener name which is used to uniquely identify each listener. You don't need to worry about this often, just follow the rule: Ensure you specify a unique listener_name every time you setup a new listener . The listener name should be unique within the service. Two listeners with the same name will only receive a partial set of events each. 2.5 Next \u00b6 This was a simple example to get you started. The worked example considers a more realistic scenario involving multiple services.","title":"2. Quick start"},{"location":"tutorial/quick-start/#21-requirements","text":"Before continuing, ensure you have completed the following steps detailed in the installation section : Installed Python 3.7 or above Installed Lightbus Running Redis 5 locally on the default port (6379) Optionally, you can read some additional explanation in the anatomy lesson section.","title":"2.1 Requirements"},{"location":"tutorial/quick-start/#22-define-your-api","text":"First we will define an API for Lightbus to serve. This will be an authentication API and will live in an authentication service. Create an auth_service directory and within there create the following in a bus.py file: # File: auth_service/bus.py import lightbus # Create your service's bus client. You can import this elsewere # in your service's codebase in order to access the bus bus = lightbus . create () class AuthApi ( lightbus . Api ): user_registered = lightbus . Event ( parameters = ( 'username' , 'email' )) class Meta : name = 'auth' def check_password ( self , username , password ): return username == 'admin' and password == 'secret' # Register this API with Lightbus. Lightbus will respond to # remote procedure calls for registered APIs, as well as allow you # as the developer to fire events on any registered APIs. bus . client . register_api ( AuthApi ()) You should now be able to startup Lightbus as follows: cd ./auth_service/ lightbus run Lightbus will output some logging data which will include a list of registered APIs, including your new auth API: Congratulations, you have now created an authentication service which contains a single API named auth . Leave Lightbus running and open a new terminal window for the next stage.","title":"2.2 Define your API"},{"location":"tutorial/quick-start/#23-remote-procedure-calls","text":"With Lightbus still running in another terminal window, create a new directory alongside auth_service called another_service . This will be an example service which will interact with the auth service. You should now have the following structure: ./auth_service/bus.py , created above ./another_service/ , which we will create files within now. We always define a service's bus client within a bus.py file. Therefore, create ./another_service/bus.py containing the following: # File: ./another_service/bus.py import lightbus bus = lightbus . create () Now create ./another_service/check_password.py . We will use this to experiment with making RPC calls: # File: ./another_service/check_password.py # Import our service's bus client from .bus import bus # Call the check_password() procedure on our auth API valid = bus . auth . check_password ( username = 'admin' , password = 'secret' ) # Show the result if valid : print ( 'Password valid!' ) else : print ( 'Oops, bad username or password' ) Running this script should show you the following: cd ./another_service/ python3 ./check_password.py Password valid! Looking at the other terminal window you have open you should see that Lightbus has also reported that it is handled a remote procedure call.","title":"2.3 Remote procedure calls"},{"location":"tutorial/quick-start/#24-events","text":"Events allow services to broadcast a message to any other services which care to listen. Your service can fire events on any API which has been registered. We have already created and registered our AuthApi in ./auth_service/bus.py which provides the user_registered event.","title":"2.4 Events"},{"location":"tutorial/quick-start/#firing-events","text":"Let's write a simple script to manually register users and fire events. Note we can only fire events for APIs we have registered. We have registered the auth API in auth_service , so it is within this service that this new script must reside. We could not put the script within another_service as this service does not include the AuthApi class and can therefore not register it. # ./auth_service/manually_register_user.py # Import the service's bus client from bus.py from bus import bus print ( \"New user creation\" ) new_username = input ( \"Enter a username: \" ) . strip () new_email = input ( \"Enter the user's email address: \" ) . strip () # You would normally store the new user in your database # at this point. We don't show this here for simplicity. # Let the bus know a user has been registered by firing the event bus . auth . user_registered . fire ( username = new_username , email = new_email ) print ( \"Done\" ) Run this script using: cd ./auth_service/ python3 manually_register_user.py You should be prompted for a new username & email address, at which point an event will be fired onto the bus. We will make use of this event in the next section.","title":"Firing events"},{"location":"tutorial/quick-start/#listening-for-events","text":"Perhaps another_service needs to be notified when a user is created. To achieve this we can setup an event listener by modifying ./another_service/bus.py : # File: ./another_service/bus.py import lightbus bus = lightbus . create () def handle_new_user ( event , username , email ): print ( f \"A new user was created in the authentication service:\" ) print ( f \" Username: { username } \" ) print ( f \" Email: { email } \" ) @bus . client . on_start () def bus_start ( client ): bus . auth . user_registered . listen ( handle_new_user , listener_name = \"print_on_new_registration\" ) Listening for events requires your service to sit waiting for something to happen. Sitting around and waiting for something to happen is precisely what the lightbus run console command is for. Therefore, in one terminal window startup the lightbus run command for another_service : # In the first terminal window: cd ./another_service/ lightbus run Your service is now waiting for events. In the second terminal window let's cause an event to be fired using the script we wrote in the previous section: cd ./auth_service/ python3 manually_register_user.py You should see that the event gets sent by the manually_register_user.py script within the auth_service , and received by by the lightbus run process within the another_service .","title":"Listening for events"},{"location":"tutorial/quick-start/#listener-names","text":"You may have noticed we specified the listener_name parameter in the above example, and we set it to an arbitrary value of \"print_on_new_registration\" . This is important, because Lightbus tracks which listeners have received which events. It is this listener name which is used to uniquely identify each listener. You don't need to worry about this often, just follow the rule: Ensure you specify a unique listener_name every time you setup a new listener . The listener name should be unique within the service. Two listeners with the same name will only receive a partial set of events each.","title":"Listener names"},{"location":"tutorial/quick-start/#25-next","text":"This was a simple example to get you started. The worked example considers a more realistic scenario involving multiple services.","title":"2.5 Next"},{"location":"tutorial/worked-example/","text":"If you get stuck... It is really useful to hear from people who have encountered a problem or got stuck. Hearing from you means we can improve our documentation and error messages. If you get stuck drop then please drop an email to adam@adamcharnock.com , visit the Lightbus discord server , or call me (Adam) on +442032896620. The more information you can include the better (problem description, screenshots, and code are all useful) In the following worked example we will create three services: An image resizing service An online store A stats dashboard This will involve a combination of web interfaces (using Flask ), and Lightbus APIs. The goal is to show how Lightbus can allow multiple services to interact. 3.1. Getting started \u00b6 The code created here can be found in Lightbus example ex03_worked_example , although the code will be repeated below. There is a directory for each service we will create \u2013 store/ , dashboard/ , and image/ . Before continuing ensure you have installed flask and honcho : pip3 install flask honcho A passing familiarity with Flask may be useful, but is not required. Honcho will assist us in running the various processes required for our services. We will assume you have already read and completed the installation and quick start tutorials. 3.2. Image resizing service \u00b6 The image resizing service will be a simple Lightbus API, the purpose of which is to allow our store to resize images prior to display: # File: ./image/bus.py import lightbus class ImageApi ( lightbus . Api ): class Meta : name = 'image' def resize ( self , url , width , height ): # This is a demo, so just return an animal picture of the correct size return f 'https://placeimg.com/ { width } / { height } /animals?_= { url } ' bus = lightbus . create () bus . client . register_api ( ImageApi ) There is no web interface for this service, so this is all we need. 3.3. Store service \u00b6 Our store will have both a Lightbus API and a web interface. We'll start with the API first: # File: ./store/bus.py import lightbus class StoreApi ( lightbus . Api ): page_viewed = lightbus . Event ( parameters = ( 'url' , )) class Meta : name = 'store' bus = lightbus . create () bus . client . register_api ( StoreApi ) This API has a single event called page_viewed . The store web interface will fire this event whenever a page is viewed. Our store web interface uses Flask and is a little longer: # File: ./store/web.py import lightbus from flask import Flask # Import our bus client from .bus import bus # Setup flask app = Flask ( __name__ ) # A dummy list of pets our store will sell PETS = ( 'http://store.company.com/image1.jpg' , 'http://store.company.com/image2.jpg' , 'http://store.company.com/image3.jpg' , ) @app . route ( '/' ) def home (): # A view to list all available pets html = '

Online pet store


' for pet_num , image_url in enumerate ( PETS ): # Get an image of the appropriate size resized_url = bus . image . resize ( url = image_url , width = 200 , height = 200 ) html += ( f '' f '' f ' ' ) # Fire the page view bus . store . page_viewed . fire ( url = '/' ) return html @app . route ( '/pet/' ) def pet ( pet_num ): # Show an individual pet resized_url = bus . image . resize ( url = PETS [ pet_num ], width = 200 , height = 200 ) # Fire the page view bus . store . page_viewed . fire ( url = f '/pet/ { pet_num } ' ) html = f '

Pet { pet_num }

' html = f '
' return html 3.4. Interlude: give it a go \u00b6 We're not quite done yet, but you can now startup the necessary processes and see the store. You will need to run each of these in a separate terminal window: $ ls image/ store/ # Start our image resizing service $ lightbus run --bus = image.bus # Start our store's web interface $ FLASK_APP = store/web.py flask run --port = 5001 Now open 127.0.0.1:5001 in your browser and you should see three animal pictures awaiting you. The URL for each image was fetched from the image resizing service. The flask web interface should also have some logging output akin to the following: Here you can see: image.resize was called three times, once for each image The store.page_viewed event was fired Next we will create the dashboard which will make use of the store.page_viewed event. 3.5. Dashboard service \u00b6 The dashboard service will provide internal reporting in the form of page view statistics for the online store. There dashboard will need to both receive events and provide a web interface. It will therefore need both a lightbus process and a web process. Fist we will start with the bus.py file: # File: ./dashboard/bus.py import json import lightbus bus = lightbus . client () page_views = {} def handle_page_view ( event_message , url ): # Record the page view in our page_views dictionary page_views . setdefault ( url , 0 ) page_views [ url ] += 1 # Store the dictionary on disk # (In reality you would probably use an external database) with open ( '/tmp/.dashboard.db.json' , 'w' ) as f : json . dump ( page_views , f ) @bus . client . on_start () def on_start ( ** kwargs ): # Called when lightbus starts up bus . store . page_viewed . listen ( handle_page_view ) This is a simple listener for the bus.store.page_viewed event. This event is fired by the store's web interface we created above. Note we do not define any APIs, instead we setup our event listener once the bus client has started up. Listening for this event is all the dashboard's Lightbus process will do, it will not provide any APIs. The handle_page_view() handler persists each view to the Dashboard services' local database. In a real service this would likely be a DBMS of some form (Postgres, MySQL, Redis, Mongo etc). For simplicity we just store JSON to a file. Now we'll define our dashboard's web interface: # File: ./dashboard/web.py import json from flask import Flask app = Flask ( __name__ ) @app . route ( '/' ) def home (): html = '

Dashboard

\\n ' html = '

Total store views

\\n ' with open ( '/tmp/.dashboard.db.json' , 'r' ) as f : page_views = json . load ( f ) html += '
    ' for url , total_views in page_views . items (): html += f '
  • URL { url } : { total_views }
    • ' html += '
' return html This reads the JSON data that was written by the event listener in dashboard/bus.py above, then render it to HTML. 3.7. Run it! \u00b6 You should now have the following python files: ./image/bus.py ./store/bus.py ./store/web.py ./dashboard/bus.py ./dashboard/web.py This translates into the following processes: Service Process type Purpose Image reszier Lightbus Will resize images and return a new URL Store Web Render the store UI. Use bus to resize image and fire events Store Lightbus While the store does have a bus.py , it does not have any RPCs to serve or events to listen for. We therefore do not need to run a lightbus service. Dashboard Web Render the dashboard web UI, read data from database Dashboard Lightbus Listen for page view events and store stats to database You can run each of these as follows: $ ls dashboard/ image/ store/ # Image resizer (Lightbus) $ lightbus run --bus = image.bus # Store (Web) $ FLASK_APP = store/web.py flask run --port = 5001 # Dashboard (Web + Lightbus) $ lightbus run --bus = dashboard.bus $ FLASK_APP = dashboard/web.py flask run --port = 5000 However, you may find it easier to startup these processes with the honcho tool we installed earlier. First, create a file called Procfile : # Procfile image_resizer_bus: lightbus run --bus = image.bus store_web: FLASK_APP = store/web.py flask run --port = 5001 dashboard_bus: lightbus run --bus = dashboard.bus dashboard_web: FLASK_APP = dashboard/web.py flask run --port = 5000 And now use honcho to startup all the processes together: $ ls Procfile dashboard/ image/ store/ $ honcho start If you see an error stating command not found , ensure you installed honcho as detailed above ( pip3 install honcho ). Once started, see the output for any errors. Each log line will state the process it came from. If all is well, you should see something like this: You should now be able to access the store's web interface at 127.0.0.1:5001 as you did previously. Upon viewing the page, the following will happen: The store web process will resize each image using the image resizing service The store web service will fire the store.page_viewed event. The dashboard will receive the store.page_viewed event and create the database for the first time. The logging output should reflect this: At this point you can view the dashboard at 127.0.0.1:5000 . Note that opening the dashboard before this point would have resulted in an error as the database would not have been created. The dashboard should show a simple list of URLs plus the total number of page views for each. Go back to the store and view a few pages. Now refresh the dashboard and note the new data. 3.7. Wrapping up \u00b6 While the services we have have created here are very crude, hopefully they have helped show how Lightbus can be used as a effective communications infrastructure. If you want to continue with practical learning, you should take a look at the How to or Reference sections. To learn more about the underlying concepts you should explore the Explanation section.","title":"3. Worked example"},{"location":"tutorial/worked-example/#31-getting-started","text":"The code created here can be found in Lightbus example ex03_worked_example , although the code will be repeated below. There is a directory for each service we will create \u2013 store/ , dashboard/ , and image/ . Before continuing ensure you have installed flask and honcho : pip3 install flask honcho A passing familiarity with Flask may be useful, but is not required. Honcho will assist us in running the various processes required for our services. We will assume you have already read and completed the installation and quick start tutorials.","title":"3.1. Getting started"},{"location":"tutorial/worked-example/#32-image-resizing-service","text":"The image resizing service will be a simple Lightbus API, the purpose of which is to allow our store to resize images prior to display: # File: ./image/bus.py import lightbus class ImageApi ( lightbus . Api ): class Meta : name = 'image' def resize ( self , url , width , height ): # This is a demo, so just return an animal picture of the correct size return f 'https://placeimg.com/ { width } / { height } /animals?_= { url } ' bus = lightbus . create () bus . client . register_api ( ImageApi ) There is no web interface for this service, so this is all we need.","title":"3.2. Image resizing service"},{"location":"tutorial/worked-example/#33-store-service","text":"Our store will have both a Lightbus API and a web interface. We'll start with the API first: # File: ./store/bus.py import lightbus class StoreApi ( lightbus . Api ): page_viewed = lightbus . Event ( parameters = ( 'url' , )) class Meta : name = 'store' bus = lightbus . create () bus . client . register_api ( StoreApi ) This API has a single event called page_viewed . The store web interface will fire this event whenever a page is viewed. Our store web interface uses Flask and is a little longer: # File: ./store/web.py import lightbus from flask import Flask # Import our bus client from .bus import bus # Setup flask app = Flask ( __name__ ) # A dummy list of pets our store will sell PETS = ( 'http://store.company.com/image1.jpg' , 'http://store.company.com/image2.jpg' , 'http://store.company.com/image3.jpg' , ) @app . route ( '/' ) def home (): # A view to list all available pets html = '

Online pet store


' for pet_num , image_url in enumerate ( PETS ): # Get an image of the appropriate size resized_url = bus . image . resize ( url = image_url , width = 200 , height = 200 ) html += ( f '' f '' f ' ' ) # Fire the page view bus . store . page_viewed . fire ( url = '/' ) return html @app . route ( '/pet/' ) def pet ( pet_num ): # Show an individual pet resized_url = bus . image . resize ( url = PETS [ pet_num ], width = 200 , height = 200 ) # Fire the page view bus . store . page_viewed . fire ( url = f '/pet/ { pet_num } ' ) html = f '

Pet { pet_num }

' html = f '
' return html","title":"3.3. Store service"},{"location":"tutorial/worked-example/#34-interlude-give-it-a-go","text":"We're not quite done yet, but you can now startup the necessary processes and see the store. You will need to run each of these in a separate terminal window: $ ls image/ store/ # Start our image resizing service $ lightbus run --bus = image.bus # Start our store's web interface $ FLASK_APP = store/web.py flask run --port = 5001 Now open 127.0.0.1:5001 in your browser and you should see three animal pictures awaiting you. The URL for each image was fetched from the image resizing service. The flask web interface should also have some logging output akin to the following: Here you can see: image.resize was called three times, once for each image The store.page_viewed event was fired Next we will create the dashboard which will make use of the store.page_viewed event.","title":"3.4. Interlude: give it a go"},{"location":"tutorial/worked-example/#35-dashboard-service","text":"The dashboard service will provide internal reporting in the form of page view statistics for the online store. There dashboard will need to both receive events and provide a web interface. It will therefore need both a lightbus process and a web process. Fist we will start with the bus.py file: # File: ./dashboard/bus.py import json import lightbus bus = lightbus . client () page_views = {} def handle_page_view ( event_message , url ): # Record the page view in our page_views dictionary page_views . setdefault ( url , 0 ) page_views [ url ] += 1 # Store the dictionary on disk # (In reality you would probably use an external database) with open ( '/tmp/.dashboard.db.json' , 'w' ) as f : json . dump ( page_views , f ) @bus . client . on_start () def on_start ( ** kwargs ): # Called when lightbus starts up bus . store . page_viewed . listen ( handle_page_view ) This is a simple listener for the bus.store.page_viewed event. This event is fired by the store's web interface we created above. Note we do not define any APIs, instead we setup our event listener once the bus client has started up. Listening for this event is all the dashboard's Lightbus process will do, it will not provide any APIs. The handle_page_view() handler persists each view to the Dashboard services' local database. In a real service this would likely be a DBMS of some form (Postgres, MySQL, Redis, Mongo etc). For simplicity we just store JSON to a file. Now we'll define our dashboard's web interface: # File: ./dashboard/web.py import json from flask import Flask app = Flask ( __name__ ) @app . route ( '/' ) def home (): html = '

Dashboard

\\n ' html = '

Total store views

\\n ' with open ( '/tmp/.dashboard.db.json' , 'r' ) as f : page_views = json . load ( f ) html += '
    ' for url , total_views in page_views . items (): html += f '
  • URL { url } : { total_views }
    • ' html += '
' return html This reads the JSON data that was written by the event listener in dashboard/bus.py above, then render it to HTML.","title":"3.5. Dashboard service"},{"location":"tutorial/worked-example/#37-run-it","text":"You should now have the following python files: ./image/bus.py ./store/bus.py ./store/web.py ./dashboard/bus.py ./dashboard/web.py This translates into the following processes: Service Process type Purpose Image reszier Lightbus Will resize images and return a new URL Store Web Render the store UI. Use bus to resize image and fire events Store Lightbus While the store does have a bus.py , it does not have any RPCs to serve or events to listen for. We therefore do not need to run a lightbus service. Dashboard Web Render the dashboard web UI, read data from database Dashboard Lightbus Listen for page view events and store stats to database You can run each of these as follows: $ ls dashboard/ image/ store/ # Image resizer (Lightbus) $ lightbus run --bus = image.bus # Store (Web) $ FLASK_APP = store/web.py flask run --port = 5001 # Dashboard (Web + Lightbus) $ lightbus run --bus = dashboard.bus $ FLASK_APP = dashboard/web.py flask run --port = 5000 However, you may find it easier to startup these processes with the honcho tool we installed earlier. First, create a file called Procfile : # Procfile image_resizer_bus: lightbus run --bus = image.bus store_web: FLASK_APP = store/web.py flask run --port = 5001 dashboard_bus: lightbus run --bus = dashboard.bus dashboard_web: FLASK_APP = dashboard/web.py flask run --port = 5000 And now use honcho to startup all the processes together: $ ls Procfile dashboard/ image/ store/ $ honcho start If you see an error stating command not found , ensure you installed honcho as detailed above ( pip3 install honcho ). Once started, see the output for any errors. Each log line will state the process it came from. If all is well, you should see something like this: You should now be able to access the store's web interface at 127.0.0.1:5001 as you did previously. Upon viewing the page, the following will happen: The store web process will resize each image using the image resizing service The store web service will fire the store.page_viewed event. The dashboard will receive the store.page_viewed event and create the database for the first time. The logging output should reflect this: At this point you can view the dashboard at 127.0.0.1:5000 . Note that opening the dashboard before this point would have resulted in an error as the database would not have been created. The dashboard should show a simple list of URLs plus the total number of page views for each. Go back to the store and view a few pages. Now refresh the dashboard and note the new data.","title":"3.7. Run it!"},{"location":"tutorial/worked-example/#37-wrapping-up","text":"While the services we have have created here are very crude, hopefully they have helped show how Lightbus can be used as a effective communications infrastructure. If you want to continue with practical learning, you should take a look at the How to or Reference sections. To learn more about the underlying concepts you should explore the Explanation section.","title":"3.7. Wrapping up"}]} \ No newline at end of file diff --git a/dev/sitemap.xml b/dev/sitemap.xml index 85785d33..c4f57fe3 100644 --- a/dev/sitemap.xml +++ b/dev/sitemap.xml @@ -2,287 +2,287 @@ http://lightbus.org/dev/ - 2023-08-17 + 2023-08-23 daily http://lightbus.org/dev/CODE_OF_CONDUCT/ - 2023-08-17 + 2023-08-23 daily http://lightbus.org/dev/explanation/ - 2023-08-17 + 2023-08-23 daily http://lightbus.org/dev/explanation/anatomy-lesson/ - 2023-08-17 + 2023-08-23 daily http://lightbus.org/dev/explanation/apis/ - 2023-08-17 + 2023-08-23 daily http://lightbus.org/dev/explanation/architecture-tips/ - 2023-08-17 + 2023-08-23 daily http://lightbus.org/dev/explanation/bus/ - 2023-08-17 + 2023-08-23 daily http://lightbus.org/dev/explanation/configuration/ - 2023-08-17 + 2023-08-23 daily http://lightbus.org/dev/explanation/events/ - 2023-08-17 + 2023-08-23 daily http://lightbus.org/dev/explanation/internal-architecture/ - 2023-08-17 + 2023-08-23 daily http://lightbus.org/dev/explanation/lightbus-vs-celery/ - 2023-08-17 + 2023-08-23 daily http://lightbus.org/dev/explanation/marshalling/ - 2023-08-17 + 2023-08-23 daily http://lightbus.org/dev/explanation/performance/ - 2023-08-17 + 2023-08-23 daily http://lightbus.org/dev/explanation/rpcs/ - 2023-08-17 + 2023-08-23 daily http://lightbus.org/dev/explanation/schema/ - 2023-08-17 + 2023-08-23 daily http://lightbus.org/dev/explanation/services/ - 2023-08-17 + 2023-08-23 daily http://lightbus.org/dev/explanation/transports/ - 2023-08-17 + 2023-08-23 daily http://lightbus.org/dev/howto/ - 2023-08-17 + 2023-08-23 daily http://lightbus.org/dev/howto/access-your-bus-client/ - 2023-08-17 + 2023-08-23 daily http://lightbus.org/dev/howto/event-sourcing/ - 2023-08-17 + 2023-08-23 daily http://lightbus.org/dev/howto/metrics/ - 2023-08-17 + 2023-08-23 daily http://lightbus.org/dev/howto/migrate-from-celery-to-lightbus/ - 2023-08-17 + 2023-08-23 daily http://lightbus.org/dev/howto/modify-lightbus/ - 2023-08-17 + 2023-08-23 daily http://lightbus.org/dev/howto/run-background-tasks/ - 2023-08-17 + 2023-08-23 daily http://lightbus.org/dev/howto/schedule-recurring-tasks/ - 2023-08-17 + 2023-08-23 daily http://lightbus.org/dev/howto/write-idempotent-event-handlers/ - 2023-08-17 + 2023-08-23 daily http://lightbus.org/dev/includes/if-you-get-stuck/ - 2023-08-17 + 2023-08-23 daily http://lightbus.org/dev/includes/note-configuration-auto-complete/ - 2023-08-17 + 2023-08-23 daily http://lightbus.org/dev/reference/ - 2023-08-17 + 2023-08-23 daily http://lightbus.org/dev/reference/apis/ - 2023-08-17 + 2023-08-23 daily http://lightbus.org/dev/reference/authors/ - 2023-08-17 + 2023-08-23 daily http://lightbus.org/dev/reference/code-of-conduct/ - 2023-08-17 + 2023-08-23 daily http://lightbus.org/dev/reference/configuration/ - 2023-08-17 + 2023-08-23 daily http://lightbus.org/dev/reference/events/ - 2023-08-17 + 2023-08-23 daily http://lightbus.org/dev/reference/plugins-development/ - 2023-08-17 + 2023-08-23 daily http://lightbus.org/dev/reference/plugins/ - 2023-08-17 + 2023-08-23 daily http://lightbus.org/dev/reference/release-process/ - 2023-08-17 + 2023-08-23 daily http://lightbus.org/dev/reference/rpcs/ - 2023-08-17 + 2023-08-23 daily http://lightbus.org/dev/reference/schema/ - 2023-08-17 + 2023-08-23 daily http://lightbus.org/dev/reference/testing-and-mocking/ - 2023-08-17 + 2023-08-23 daily http://lightbus.org/dev/reference/transport-configuration/ - 2023-08-17 + 2023-08-23 daily http://lightbus.org/dev/reference/typing/ - 2023-08-17 + 2023-08-23 daily http://lightbus.org/dev/reference/command-line-use/dumpconfigschema/ - 2023-08-17 + 2023-08-23 daily http://lightbus.org/dev/reference/command-line-use/dumpschema/ - 2023-08-17 + 2023-08-23 daily http://lightbus.org/dev/reference/command-line-use/inspect/ - 2023-08-17 + 2023-08-23 daily http://lightbus.org/dev/reference/command-line-use/run/ - 2023-08-17 + 2023-08-23 daily http://lightbus.org/dev/reference/command-line-use/shell/ - 2023-08-17 + 2023-08-23 daily http://lightbus.org/dev/reference/protocols/ - 2023-08-17 + 2023-08-23 daily http://lightbus.org/dev/reference/protocols/event/ - 2023-08-17 + 2023-08-23 daily http://lightbus.org/dev/reference/protocols/rpc-and-result/ - 2023-08-17 + 2023-08-23 daily http://lightbus.org/dev/reference/protocols/schema/ - 2023-08-17 + 2023-08-23 daily http://lightbus.org/dev/tutorial/ - 2023-08-17 + 2023-08-23 daily http://lightbus.org/dev/tutorial/django-and-lightbus/ - 2023-08-17 + 2023-08-23 daily http://lightbus.org/dev/tutorial/getting-involved/ - 2023-08-17 + 2023-08-23 daily http://lightbus.org/dev/tutorial/installation/ - 2023-08-17 + 2023-08-23 daily http://lightbus.org/dev/tutorial/quick-start/ - 2023-08-17 + 2023-08-23 daily http://lightbus.org/dev/tutorial/worked-example/ - 2023-08-17 + 2023-08-23 daily \ No newline at end of file diff --git a/dev/sitemap.xml.gz b/dev/sitemap.xml.gz index b8d7c1c3f7747b16a1b4019eaa5cdc0dae1b525b..ab02192cb3c6489c28e548e8c3d5d56d68323353 100644 GIT binary patch delta 644 zcmV-~0(<@51>FUIABzYGt<>dY0{?SqbY*Q}a4vXlYyj1mJ9FDG49E9=3ZHjcZnC63 z#~G4lY?frsF(f3h<|`m*$KSq?<;3n|>Cih!izJ9g{f@-r53X+Bf}d>xi%ao({&}&S z&mfv)UF@#sPmlNNYJRi+w5sR}(2NJiay{>f^4F%>?RJZQ;s}DyWs$IJis|{KG)H5z zTzr|YKh5|WCK#U%*UBfeKF;D@*V9WKXRxqsR?YDBe6TJQl3iacFD}*cN?ly?kfZHv znCW%wphb9HTkZUQ)tp&Iisl33Ux02O?!G-g+&|wwe7}3TeUt|XZ`tdpiT6-8+;C4} zuRN3@o;ABNBAgIS&yi-RA}DfMI4;{|1DX_D*HxTAAi)+Q313bZ2{3A}&izz2^7}`% zEy{og+{=?N0Uac_&VcZhB-F9wgc1{JpLUd_Vrj#BThu;PR718j-f=mSN_(i2aRC=A zSQ-^2^*&lGHg~e^UOXn&s9v<$beL+iigekCWgY5|tsWGDsfZj>vUzutvjHMIHiA(j zFRl8ee^R<~elRIy4{848Ys=7WfU#l}4#i&Sv eDR&7sU=_US;q2`HgC1@LX#Ep2C{QI0CIA45J2*xF delta 645 zcmV;00($-31>OaJABzYGE9~850{?SqbY*Q}a4vXlYyj1mJ9FDG49E9;3ZHjcPC7L0 z+0KwOW3wc4jv*n5HD3WiJO1{CEGKp!ONZXUS|mX{>USg_e{ga07W`xbSX_$B>E&!b zoj^3ny4WqJPmlNNdU~_ESXA@{Xu^YISx$SR{IzMe+wH7>ID(*anI-I+VtRfl&B55r zXJ4kPiwR%D1mn~GTKHsEhgrPqdU~nj3>G%cq8YxP4%UT2va754)h9K-R+pc7$ia3p z%=9{T&?3C9tag64Xih96Me~92FF>~sci)~L?w@ZTzTZ9FKFR}xx9oM)#Cs?kZn&qg zQyxkY&zfB_5l)Dv=SVYD5fr&B9GC5~0Zodn>nff>Ai)MA313bZ2{3A}PW@Ci^7}`% zDawEc+{u$L0Uaba&VcZhB-F9wgc1{JpSF~wVrj#BThuO9R718j-f=mSO1rOos9v;LcbICliga0vWgY5|tsWGDsfZj>vU#_YvjHMI)`C$Z zFRl8ee^R<~elRIy4{Ymt4^3GRC%ug)fg(k=^MRF-aG*njULs9K=0 zHRN%9Xmx;zl1aXt2jpHIo(C!`g?)`fDAk_{ zv(zj`-0!2)UND*c!Sf|GM!(b(!4KLFXfMKC+$6s+0V^1l%M@wvg{rO(%-8D7S}`h; fV9IU6HCP32dN@1#|DgL@0b2b8ATdFs4JH5pbDK0J