-
Notifications
You must be signed in to change notification settings - Fork 6
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
create structure and first documentation pages
- Loading branch information
1 parent
b9cd2d0
commit 192ccb2
Showing
22 changed files
with
101 additions
and
1 deletion.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,29 @@ | ||
# Overview | ||
|
||
``kedro-boot`` tries to solve 3 differents issues of kedro (as of ``kedro==0.19.X``) which occur when deploying kedro projects. This is described in details in this [design document](https://github.com/kedro-org/kedro/issues/3540). | ||
|
||
## The standalone mode - Programmatically call the Kedro Session | ||
|
||
### The key feature : running the session multiple times | ||
|
||
We want to use the ``KedroSession`` object programmatically *outside a kedro project* to control its behaviour in another application. | ||
|
||
The key feature is to make the ``KedroSession`` runnable multiple times, so that we do not have to rebuilt the entire objects (which is slow and complicated) for each execution. This needs a couple of optimization for speed and flexibility: | ||
|
||
- runtime data injection: we should be able to pass data in memory to the session to override a dataset, and not force the dataset to read it from a persisted source | ||
- runtime params injection: we should be able to modify parameters between each execution to modify the behaviour | ||
- artifacts preloading: we should be able to preload and cache once for all session exeuxtions some input data with a high I/O cost (e.g. a machine learning model should not be hot reloaded on eeach predict, but only once ) | ||
|
||
Basically, we would like to be able to execute the following code: | ||
|
||
```python | ||
# PSEUDO CODE - Non runnable in plain kedro, this is a fictitious API | ||
session=KedroSession.create(project_path) | ||
session.preload(artifacts=["my_model", "my_encoder"]) # :sparkles: FEATURE 1: preload artifacts, cache them and not not release them between runs for speed | ||
session.run(pipeline="my_pipeline", runtime_data={"data": data1}) # :sparkles: FEATURE 2: inject data at runtime from the session | ||
session.run(pipeline="my_pipeline", runtime_params= {"customer_id": id}) # :sparkles: FEATURE 3: run the same session mulitple times + :sparkles: FEATURE 4: inject runtime params at... runtime (as the name says!) instead of instantation time | ||
``` | ||
|
||
```tip | ||
Running the session programatically is very convenient when you want to call kedro in a programm that owns the entrypoint (e.g. you launch the app with a command different from ``kedro run`` like ``streamlit run``, ``uvicorn run``...) | ||
``` |
File renamed without changes.
File renamed without changes.
File renamed without changes.
5 changes: 5 additions & 0 deletions
5
docs/source/02_getting_started/02_which_mode_should_i_choose.md
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,5 @@ | ||
```{tip} | ||
- If you want to change the behaviour of ``kedro run`` (e.g. to loop over the session multiple times), see the [embedded mode]() | ||
- If you want to call a kedro pipeline programatically in a third party application launched with its own command (eg. ``streamlit run``, ``uvicorn run``, datarobot...), see the [standalone mode]() | ||
- if you want to serve a kedro pipeline as an API, see the [KedroBootFastapiApp]() | ||
``` |
File renamed without changes.
File renamed without changes.
Empty file.
File renamed without changes.
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,6 @@ | ||
# Kedro Boot objects | ||
|
||
## KedroBootSession | ||
|
||
## CompilationSpec | ||
|
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,3 @@ | ||
# Preload artifacts | ||
|
||
# Passing itertime params |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,6 @@ | ||
# Kedro Boot objects | ||
|
||
## KedroBootSession | ||
|
||
## CompilationSpec | ||
|
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,3 @@ | ||
# Preload artifacts | ||
|
||
# Passing itertime params |
6 changes: 6 additions & 0 deletions
6
docs/source/06_fastapi_pipeline_serving/01_kedro_fast_api_hello_world.md
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,6 @@ | ||
# Kedro Boot objects | ||
|
||
## KedroBootSession | ||
|
||
## CompilationSpec | ||
|
6 changes: 6 additions & 0 deletions
6
docs/source/06_fastapi_pipeline_serving/02_kedro_fastapi_mapping.md
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,6 @@ | ||
# Kedro Boot objects | ||
|
||
## KedroBootSession | ||
|
||
## CompilationSpec | ||
|
6 changes: 6 additions & 0 deletions
6
docs/source/06_fastapi_pipeline_serving/03_kedro_fastapi_advanced.md
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,6 @@ | ||
# Kedro Boot objects | ||
|
||
## KedroBootSession | ||
|
||
## CompilationSpec | ||
|
6 changes: 6 additions & 0 deletions
6
docs/source/06_fastapi_pipeline_serving/04_fastapi_as_kedro_boot_app.md
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,6 @@ | ||
# Kedro Boot objects | ||
|
||
## KedroBootSession | ||
|
||
## CompilationSpec | ||
|
6 changes: 6 additions & 0 deletions
6
docs/source/07_example_gallery/embedded_mode/01_hyperparameter_tuning.md
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,6 @@ | ||
# Kedro Boot objects | ||
|
||
## KedroBootSession | ||
|
||
## CompilationSpec | ||
|
6 changes: 6 additions & 0 deletions
6
docs/source/07_example_gallery/embedded_mode/02_monte_carlo.md
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,6 @@ | ||
# Kedro Boot objects | ||
|
||
## KedroBootSession | ||
|
||
## CompilationSpec | ||
|
6 changes: 6 additions & 0 deletions
6
docs/source/07_example_gallery/standalone_mode/01_streamlit.md
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,6 @@ | ||
# Kedro Boot objects | ||
|
||
## KedroBootSession | ||
|
||
## CompilationSpec | ||
|
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,6 @@ | ||
# Kedro Boot objects | ||
|
||
## KedroBootSession | ||
|
||
## CompilationSpec | ||
|