diff --git a/docs/architecture/mono-repo.md b/docs/architecture/mono-repo.md new file mode 100644 index 0000000..cfe7847 --- /dev/null +++ b/docs/architecture/mono-repo.md @@ -0,0 +1,181 @@ +# Clojure Mono Repo example : server + 2 clients + +## πŸ”Έ Context + +Our app [skydread1/flybot.sg](https://github.com/skydread1/flybot.sg) is a full-stack Clojure **web** and **mobile** app. + +We opted for a mono-repo to host: +- the `server`: Clojure app +- the `web` client: Reagent (React) app using Re-Frame +- the `mobile` client: Reagent Native (React Native) app using Re-Frame + +Note that the web app does not use NPM at all. However, the React Native mobile app does use NPM and the `node_modules` need to be generated. + +By using only one `deps.edn`, we can easily starts the different parts of the app. + +## πŸ”Έ Goal + +The goal of this document is to highlight the mono-repo structure and how to run the different parts (dev, test, build etc). + +## πŸ”Έ Repo structure + +``` +β”œβ”€β”€ client +β”‚Β Β  β”œβ”€β”€ common +β”‚Β Β  β”‚Β Β  β”œβ”€β”€ src +β”‚Β Β  β”‚Β Β  β”‚Β Β  └── flybot.client.common +β”‚Β Β  β”‚Β Β  └── test +β”‚Β Β  β”‚Β Β  └── flybot.client.common +β”‚Β Β  β”œβ”€β”€ mobile +β”‚Β Β  β”‚Β Β  β”œβ”€β”€ src +β”‚Β Β  β”‚Β Β  β”‚Β Β  └── flybot.client.mobile +β”‚Β Β  β”‚Β Β  └── test +β”‚Β Β  β”‚Β Β  └── flybot.client.mobile +β”‚Β Β  └── web +β”‚Β Β  β”œβ”€β”€ src +β”‚Β Β  β”‚Β Β  └── flybot.client.web +β”‚Β Β  └── test +β”‚Β Β  └── flybot.client.web +β”œβ”€β”€ common +β”‚Β Β  β”œβ”€β”€ src +β”‚Β Β  β”‚Β Β  └── flybot.common +β”‚Β Β  └── test +β”‚Β Β  └── flybot.common +β”œβ”€β”€ server +β”‚Β Β  β”œβ”€β”€ src +β”‚Β Β  β”‚Β Β  └── flybot.server +β”‚Β Β  └── test +β”‚Β Β  └── flybot.server +``` + +- `server` dir contains then `.clj` files +- `common` dir the `.cljc` files +- `clients` dir the `.cljs` files. + +## πŸ”Έ Deps Management + +You can have a look at the [deps.edn](https://github.com/skydread1/flybot.sg/blob/master/deps.edn). + +We can use namespaced aliases in `deps.edn` to make the process clearer. + +I will go through the different aliases and explain their purposes and how to I used them to develop the app. + +## πŸ”Έ Common libraries + +### clj and cljc deps + +First, the root deps of the deps.edn, inherited by all aliases: + +#### Both frontend and backend +- org.clojure/clojure +- metosin/malli +- metosin/reitit +- metosin/muuntaja +- sg.flybot/lasagna-pull + +#### Backend +- ring/ring-defaults +- aleph/aleph +- robertluo/fun-map +- datalevin/datalevin +- skydread1/reitit-oauth2 + +The deps above are used in both `server/src` and `common/src` (clj and cljc files). + +So every time you start a `deps` REPL or a `deps+figwheel` REPL, these deps will be loaded. + +### Sample data + +In the [common/test/flybot/common/test_sample_data.cljc](https://github.com/skydread1/flybot.sg/blob/master/common/test/flybot/common/test_sample_data.cljc) namespace, we have sample data that can be loaded in both backend dev system of frontend dev systems. + +This is made possible by reader conditionals clj/cljs. + +### IDE integration + +I use the `calva` extension in VSCode to jack-in deps and figwheel REPLs but you can use Emacs if you prefer for instance. + +What is important to remember is that, when you work on the backend only, you just need a `deps` REPL. There is no need for figwheel since we do not modify the cljs content. +So in this scenario, the frontend is fixed (the main.js is generated and not being reloaded) but the backend changes (the `clj` files and `cljc` files). + +However, when you work on the frontend, you need to load the backend deps to have your server running but you also need to recompile the js when a cljs file is saved. Therefore your need both `deps+figwheel` REPL. So in this scenario, the backend is fixed and running but the frontend changes (the `cljs` files and `cljc` files) + +You can see that the **common** `cljc` files are being watched in both scenarios which makes sense since they "become" clj or cljs code depending on what REPL type you are currently working in. + +## πŸ”Έ Server aliases + +Following are the aliases used for the server: + +- `:jvm-base`: JVM options to make datalevin work with java version > java8 +- `:server/dev`: clj paths for the backend systems and tests +- `:server/test`: Run clj tests + +## πŸ”Έ Client common aliases + +Following is the alias used for both web and mobile clients: + +- `:client`: deps for frontend libraries common to web and react native. + +The extra-paths contains the `cljs` files. + +We can note the `client/common/src` path that contains most of the `re-frame` logic because most subscriptions and events work on both web and react native right away! + +The main differences between the re-frame logic for Reagent and Reagent Native are have to do with how to deal with Navigation and oauth2 redirection. That is the reason we have most of the logic in a **common** dir in `client`. + +## πŸ”Έ Mobile Client + +Following are the aliases used for the **mobile** client: + +- `:mobile/rn`: contains the cljs deps only used for react native. They are added on top of the client deps. +- `:mobile/ios`: starts the figwheel REPL to work on iOS. + +## πŸ”Έ Web Client + +Following are the aliases used for the **web** client: + +- `:web/dev`: starts the dev REPL +- `:web/prod`: generates the optimized js bundle main.js +- `:web/test`: runs the cljs tests +- `:web/test-headless`: runs the headless cljs tests (fot GitHub CI) + +## πŸ”Έ CI/CD aliases + +### build.clj + +Following is the alias used to build the js bundle or a uberjar: + +- `:build`: [clojure/tools.build](https://github.com/clojure/tools.build) is used to build the main.js and also an uber jar for local testing, we use . + +The build.clj contains the different build functions: + +- Build frontend js bundle: `clj -T:build js-bundle` +- Build backend uberjar: `clj -T:build uber` +- Build both js and jar: `clj -T:build uber+js` + +### Jibbit + +Following is the alias used to build an image and push it to local docker or AWS ECR: + +- `:jib`: build image and push to image repo + +## πŸ”Έ Antq + +Following is the alias used to points out outdated dependencies + +- `:outdated`: prints the outdated deps and their last available version + + +## πŸ”Έ Notes on Mobile CD + +We have not released the mobile app yet, that is why there is no aliases related to CD for react native yet. + +## πŸ”Έ Conclusion + +This is one solution to handle server and clients in the same repo. + +It is important to have a clear directory structure to only load required namespaces and avoid errors. + +Using `:extra-paths` and `:extra-deps` in deps.edn is important because it prevent deploying unnecessary namespaces and libraries on the server and client. + +Adding namespace to the aliases make the distinction between backend, common and client (web and mobile) clearer. + +Using `deps` jack-in for server only work and `deps+figwheel` for frontend work is made easy using `calva` in VSCode (work in other editors as well). diff --git a/docs/development/how-to-run.md b/docs/development/how-to-run.md new file mode 100644 index 0000000..6ca48a0 --- /dev/null +++ b/docs/development/how-to-run.md @@ -0,0 +1,219 @@ +# How to run the different systems + +Since the app is a full-stack app with 2 clients, you might want to work only on the **backend**, only on the **web** client or only on the **mobile** client. + +Ideally, you want to only work on either backend or frontend at a time. + +## πŸ”Έ Config files + +In the `config` directory, you can see a `sys.edn` file. It gathers systems, oauth2 and owner configurations. + +1) Systems + +Depending on the system you use, the config varies, such as: +- datalevin db uri +- server port +- oauth2 config + +We have 4 different system types: +- `figwheel-system`: provides a ring-handler to figwheel which starts its own server on port 9500. This system is designed for frontend dev with dummy data as website initial content. +- `dev-system`: starts an aleph server on port 8123. Uses the same dummy data as figwheel. This system is meant to be used for backend dev. +- `test-system`: starts aleph server on port 8100. Uses some test data that covers many scenario for good testing. +- `prod-system`: starts an aleph server on port 8123. It uses existing data and do not clear any data on system halt! + +You can read more about how the systems work in [here](../lasagna-stack/fun-map-applied-to-flybot.md) + +_Note_: +- the 3 systems `figwheel`, `dev` and `test` clear the data on system `halt!` +- the `prod` system does not clear the data on system `halt!` + +2) OAuth2.0 + +We use google as tier service to authenticate our users. + +The OAuth2.0 credentials allow our application to access google services. + +You need to provide the google id and secret that allow the oauth client to communicate with the oauth server. + +You can read more about it [here](https://github.com/skydread1/reitit-oauth2#readme) + +All the systems (except `test-system`) need the `oauth2` credentials and developers need a company account to be able to test OAuth2.0 locally as well. + +If you want to test the admin/owner features in the UI and you don't want to setup google OAuth2.0, you might need to implement your own login system or mock the OAuth2.0 logic to locally access all the website sections. + +_Note: not providing the creds will prevent you from login/logout during dev in the UI by default._ + +_Note 2: the **redirect-uri** as `:oauth2-callback` is specified in the :systems config and not :oauth2 config because it depends on the environment._ + +3) Owner + +The `:owner` user is loaded to the DB when the system is `touch` (except for `prod`). +You need to use a google account that is allowed by your `Location` (i.e. company account). +The account provided in `:owner` is granted all roles so it has access to all the website sections in dev. + +_Note: your google account needs to belong to the google application linked to the app._ + +_Note 2: not providing your acc id will prevent you from doing admin/owner tasks in the UI._ + +4) Figwheel + +You can notice that there is a flag `figwheel?` in `sys.edn`. It allows figwheel to use the system handler when you start the REPLs. + +The `figwheel-system` is `touch` when systems.clj is loaded. So if you do not want to work on the frontend (or in production), set the flag to false. + +## πŸ”Έ Frontend : WEB + +### DEV + +You can perform ClojureScript jack-in to open the webpage in a browser on port `9500`, alongside an interactive REPL in your IDE (VS Code or Emacs). + +You can then edit and save source files to trigger hot reloading in the browser. + +#### Prerequisites + +- Delete any `main.js` in the resources folder +- Delete `node_modules` at the root (not needed for the web) +- Go to `resources/public/index.html` and check if `cljs-out/dev-main.js` is the script source in `index.html` (near the end of the file) +- Open a source file in either VS Code or Emacs + +#### VS Code + +If you use VS Code, the jack-in is done in 2 steps to be able to start the REPL in VS Code instead of terminal: + +1. Choose the aliases for the deps and press enter +2. Choose the ClojureScript REPL you want to launch and press enter + +Jack-in `deps+figwheel`: + +- Deps: `:jvm-base`, `:client` +- REPL: `:web/dev` + +#### Emacs + +If you use Emacs (or Doom Emacs, or Spacemacs) with CIDER, the CIDER jack-in is done in 3 steps: + +1. `C-u M-x cider-jack-in-clj&cljs` or `C-u M-x cider-jack-in-cljs` +2. By default, emacs use the `cider/nrepl` alias such as in `-M:cider/nrepl`. You need to keep this alias at the end such as `-M:jvm-base:client:web/dev:cider/nrepl` +3. Select ClojureScript REPL type: `figwheel-main` +4. Select figwheel-main build: `dev` + +### TEST in terminal + +``` +clj -A:jvm-base:client:web/test +``` + +### Regression tests on save + +Regression tests are run on every save and the results are displayed at http://localhost:9500/figwheel-extra-main/auto-testing + +These frontend cljs tests ensure that the states (in our re-frame DB) is as expected after user actions (navigation, theme, post interaction etc). + +## πŸ”Έ Frontend : MOBILE + +### DEV + +Prerequisites: +- [prepare your environment](https://reactnative.dev/docs/next/environment-setup) +- if no `node_modules`, run `npm install` at the root +- for ios, run `pod install` in the `ios` directory +- be sure to update `:client-root-path` in config/system.edn + +_Note_: only tested with Xcode simulator + +Features: +- Server will be launched on port 9500 +- Just save a file to trigger hot reloading on your Xcode simulator + +Jack-in `deps+figwheel`: +- DEPS: `:jvm-base`, `:client`, `:mobile/rn` +- REPL: `:mobile/ios` +- Simulator: run `npm run ios` in an external terminal - once done it will star the cljs repl in VSCode + +## πŸ”Έ Backend + +### DEV + +Prerequisites: +- if you want to have a UI, you can generate the `main.js` bundle via `clj T:build js-bundle` + +Features: +- the `system` namespace provides a dev-system to start an aleph server on port 8123 with sample data for db + +Jack-in `deps`: +- DEPS: `:jvm-base`, `:server/dev` + +### TEST in terminal + +``` +clj -A:jvm-base:server/test +``` + +### Package to uberjar + +Prerequisites: +- delete `node_modules` at the root because no need for the web +- Check if `main.js` is the script source in `index.html` + +Features: +- build js bundle +- build uberjar + +Build: +- `clj T:build js-bundle` +- `clj T:build uber` +- `clj T:build uber+js` + +To run the uberjar +``` +SYSTEM="{...}" \ +java -jar \ +--add-opens=java.base/java.nio=ALL-UNNAMED \ +--add-opens=java.base/sun.nio.ch=ALL-UNNAMED \ +target/flybot.sg-{version}-standalone.jar +``` + +## πŸ”Έ CD + +### Create a container image and push it to ECR + +You need to have aws cli installed (v2 or v1) and you need an env variable `$ECR_REPO` setup with the ECR repo string. + +You have several [possibilities](https://github.com/atomisthq/jibbit/blob/main/src/jibbit/aws_ecr.clj) to provide credentials to login to your AWS ECR, notably +- For authorizer type `:profile`: AWS credentials profile in ~/.aws/credentials and `:profile-name` key to jibbit authorizer +- For authorizer type `:environment`: Env variables AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY + +To create the image and push it to the ECR account +- `clj -T:jib build` + +### Create a container image to try locally with docker + +There is a `jib-dev.edn` provided with the config to create the image locally. + +### Start container with image + +``` +docker run \ +--rm \ +-it \ +-p 8123:8123 \ +-v db-volume:/datalevin/prod/flybotdb \ +-e SYSTEM="{...}" \ +some-image-uri:latest +``` + +### Example of what the SYSTEM env variable could look like for prod: + +```clojure +{:systems {:prod {:http-port 8123 + :db-uri "/datalevin/prod/flybotdb" + :oauth2-callback "https://www.flybot.sg/oauth/google/callback"}} + :oauth2 {:google-creds {:client-id "secret" + :client-secret "secret"}} + :owner #:user{:id "google-personal-acc-id" + :email "bob@company.com" + :name "Bob Smith"}} +``` +Note that we removed the other systems (dev, test and figwheel) configs because they are not needed in prod. + +Also, the `:figwheel?` flag has been removed to prevent `figwheel-system` from starting when systems.clj loads in the prod container. diff --git a/docs/lasagna-stack/fun-map-applied-to-flybot.md b/docs/lasagna-stack/fun-map-applied-to-flybot.md new file mode 100644 index 0000000..2cfcb04 --- /dev/null +++ b/docs/lasagna-stack/fun-map-applied-to-flybot.md @@ -0,0 +1,250 @@ +# Fun-Map applied to flybot.sg + +## πŸ”Έ Prerequisites + +If you are not familiar with [fun-map](https://github.com/robertluo/fun-map), please refer to the doc [Fun Map Rational](./fun-map.md) + +## πŸ”Έ Goal + +In this document, I will show you how we leverage `fun-map` to create different systems: `prod-system`, `dev-system`, `test-system` and `figwheel-system`. + +## πŸ”Έ Prod System + +In our backend, we use `life-cycle-map` to manage the life cycle of all our stateful components. + +### Describe the system + +Here is the system we currently have for production: + +```clojure +(defn system + [{:keys [http-port db-uri google-creds oauth2-callback client-root-path] + :or {client-root-path "/"}}] + (life-cycle-map + {:db-uri db-uri + :db-conn (fnk [db-uri] + (let [conn (d/get-conn db-uri db/initial-datalevin-schema)] + (load-initial-data conn data/init-data) + (closeable + {:conn conn} + #(d/close conn)))) + :oauth2-config (let [{:keys [client-id client-secret]} google-creds] + (-> config/oauth2-default-config + (assoc-in [:google :client-id] client-id) + (assoc-in [:google :client-secret] client-secret) + (assoc-in [:google :redirect-uri] oauth2-callback) + (assoc-in [:google :client-root-path] client-root-path))) + :session-store (memory-store) + :injectors (fnk [db-conn] + [(fn [] {:db (d/db (:conn db-conn))})]) + :executors (fnk [db-conn] + [(handler/mk-executors (:conn db-conn))]) + :saturn-handler handler/saturn-handler + :ring-handler (fnk [injectors saturn-handler executors] + (handler/mk-ring-handler injectors saturn-handler executors)) + :reitit-router (fnk [ring-handler oauth2-config session-store] + (handler/app-routes ring-handler oauth2-config session-store)) + :http-server (fnk [http-port reitit-router] + (let [svr (http/start-server + reitit-router + {:port http-port})] + (closeable + svr + #(.close svr))))})) + +(def prod-system + "The prod system starts a server on port 8123. + It does not load any init-data on touch and it does not delete any data on halt!. + You can use it in your local environment as well." + (let [prod-cfg (config/system-config :prod)] + (system prod-cfg))) +``` + +At a glance, we can easily understand the dependency injections flow of the app. + +If we were to represent these deps as a simple graph, we could have: + +```bash +life-cycle-map +β”œβ”€β”€ :db-conn (closeable) +β”œβ”€β”€ :oauth2-config +β”œβ”€β”€ :session-store +β”œβ”€β”€ :injectors +β”‚ └── :db-conn +β”œβ”€β”€ :executors +β”‚ └── :db-conn +β”œβ”€β”€ :saturn-handler +β”œβ”€β”€ :ring-handler +β”‚ β”œβ”€β”€ :injectors +β”‚ β”œβ”€β”€ :executors +β”‚ β”œβ”€β”€ :saturn-handler +β”œβ”€β”€ :reitit-router +β”‚ β”œβ”€β”€ :ring-handler +β”‚ β”œβ”€β”€ :oauth2-config +β”‚ └── :session-store +└── :http-server (closeable) + β”œβ”€β”€ :http-port + β”œβ”€β”€ :reitit-router +``` + +The function `prod-system` just fetches some env variables with the necessary configs to start the system. + +### Run the system + +We can then easily start the system via the fun-map function `touch` : + +```clojure +cljκž‰clj.flybot.coreκž‰>Β  +(touch prod-system) +{:ring-handler #function[clj.flybot.handler/mk-ring-handler/fn--37646], + :executors [#function[clj.flybot.handler/mk-executors/fn--37616]], + :injectors [#function[clj.flybot.core/system/fn--38015/fn--38016]], + :http-server + #object[aleph.netty$start_server$reify__11448 0x389add75 "AlephServer[channel:[id: 0xd98ed2db, L:/0.0.0.0:8123], transport::nio]"], + :reitit-router #function[clojure.lang.AFunction/1], + :http-port 8123, + :db-uri "datalevin/prod/flybotdb", + :oauth2-config + {:google + {:scopes ["https://www.googleapis.com/auth/userinfo.email" "https://www.googleapis.com/auth/userinfo.profile"], + :redirect-uri "https://v2.fybot.sg/oauth/google/callback", + :client-id "client-id", + :access-token-uri "https://oauth2.googleapis.com/token", + :authorize-uri "https://accounts.google.com/o/oauth2/auth", + :launch-uri "/oauth/google/login", + :client-secret "client-secret", + :project-id "flybot-website", + :landing-uri "/oauth/google/success"}}, + :session-store + #object[ring.middleware.session.memory.MemoryStore 0x1afb7eac "ring.middleware.session.memory.MemoryStore@1afb7eac"], + :saturn-handler #function[clj.flybot.handler/saturn-handler], + :db-conn + {:conn + #}} +``` + +## πŸ”Έ Dev System + +The `system` described above can easily be adapted to be used for development purposes. + +Actually, the only differences between the prod and dev systems are the following: + +- The configs (db uri, oauth2 callback) +- How to shutdown the db system (`dev` clears the db, `prod` retains db data) + +Thus, we just have to assoc a new db component to the `system` and read some dev configs instead of getting prod env variables: + +```clojure +(defn db-conn-system + "On touch: empty the db and get conn. + On halt!: close conn and empty the db." + [init-data] + (fnk [db-uri] + (let [conn (d/get-conn db-uri) + _ (d/clear conn) + conn (d/get-conn db-uri db/initial-datalevin-schema)] + (load-initial-data conn init-data) + (closeable + {:conn conn} + #(d/clear conn))))) + +(def dev-system + "The dev system starts a server on port 8123. + It loads some real data sample. The data is deleted when the system halt!. + It is convenient if you want to see your backend changes in action in the UI." + (-> (system (config/system-config :dev)) + (assoc :db-conn (db-conn-system data/init-data)))) +``` + +The important thing to remember is that all the modifications to the system must be done before starting the system (via `touch` ). If some modifications need to be made to the running system: + +1. Shutdown the system (via `halt!`) +2. Update the system logic +3. Start the newly modified system (via `touch`) + +## πŸ”Έ Test system + +Naturally, the fun-map system also plays well with testing. + +Same process as for dev and prod, we just need to adapt the system a bit to run our tests. + +The tests requirement are: + +- Dedicated db uri and specific data sample to work with +- Ignore Oauth2.0. + +So same as for dev, we just read dedicated test configs and assoc a test db system to the default system: + +```clojure +(defn test-system + [] + (-> (config/system-config :test) + sys/system + (dissoc :oauth2-config) + (assoc :db-conn (sys/db-conn-system test-data)))) +``` + +This works well with the clojure.test fixtures: + +```clojure +;; atom required to re-evalualte (test-system) because of fixture `:each` +(def a-test-system (atom nil)) + +(defn system-fixture [f] + (reset! a-test-system (test-system)) + (touch @a-test-system) + (f) + (halt! @a-test-system)) + +(use-fixtures :each system-fixture) +``` + +## πŸ”Έ Figwheel system + +It is possible to [provide a ring-handler](https://figwheel.org/docs/ring-handler.html) to figwheel configs which will be passed to a server figwheel starts for us. + +We just need to specify a ring-handler in `figwheel-main.edn` like so: + +```clojure +{:ring-handler flybot.server.systems/figwheel-handler + :auto-testing true} +``` + +Our system does have a ring-handler we can supply to figwheel, it is called `reitit-router` in our system (it returns a ring-handler). + +Since figwheel starts the server, we do not need the aleph server dependency in our system anymore, se we can dissoc it from the system. + +So here is the `figwheel-system` : + +```clojure +(def figwheel-system + "Figwheel automatically touches the system via the figwheel-main.edn on port 9500. + Figwheel just needs a handler and starts its own server hence we dissoc the http-server. + If some changes are made in one of the backend component (such as handler for instance), + you can halt!, reload ns and touch again the system." + (-> (config/system-config :figwheel) + system + (assoc :db-conn (db-conn-system data/init-data)) + (dissoc :http-port :http-server))) + +(def figwheel-handler + "Provided to figwheel-main.edn. + Figwheel uses this handler to starts a server on port 9500. + Since the system is touched on namespace load, you need to have + the flag :figwheel? set to true in the config." + (when (:figwheel? CONFIG) + (-> figwheel-system + touch + :reitit-router))) +``` + +The `figheel-handler` is the value of the key `:reitit-router` of our running system. + +So the system is started first via `touch` and its handler is provided to the servers figwheel starts that will be running while we work on our frontend. diff --git a/docs/lasagna-stack/fun-map.md b/docs/lasagna-stack/fun-map.md new file mode 100644 index 0000000..d66b5ab --- /dev/null +++ b/docs/lasagna-stack/fun-map.md @@ -0,0 +1,238 @@ +# Fun-Map Rational + +This report aims at introducing the Lasagna stack library [fun-map](https://github.com/robertluo/fun-map). Fun-Map blurs the line between identity, state and function. As a results, it is a very convenient tool to define `system` in your applications by providing an elegant way to perform associative dependency injections. + +## πŸ”Έ Goal + +In this document, I will show you the benefit of `fun-map`, and especially the `life-cycle-map` as dependency injection system. + +## πŸ”Έ Rational + +### Managing state + +In any kind of programs, we need to manage the state. In Clojure, we want to keep the mutation parts of our code as isolated and minimum as possible. The different components of our application such as the db connections, queues or servers for instance are mutating the world and sometimes need each other to do so. The talk [Components Just Enough Structure](https://www.youtube.com/watch?v=13cmHf_kt-Q) by Stuart Sierra explains this dependency injection problem very well and provides a Clojure solution to this problem with the library [component](https://github.com/stuartsierra/component). + +Our library to do so is [fun-map](https://github.com/robertluo/fun-map). In order to understand why fun-map is very convenient, it is interesting to look at other existing solutions first. + +### Component + +Let’s first have a look at existing solution to deal with life cycle management of components in Clojure, especially the Component library which is a very good library to provide a way to define systems. + +#### stuartsierra/component + +In the Clojure word, we have stateful components (atom, channel etc) and we don’t want it to be scattered in our code without any clear way to link them and also know the order of which to start these external resources. + +The `component` of the library [component](https://github.com/stuartsierra/component) is just a record that implements a `Lifecycle` protocol to properly start and stop the component. As a developer, you just implement the `start` and `stop` methods of the protocol for each of your components (DB, server or even domain model). + +A DB component could look like this for instance + +```clojure +(defrecord Database [host port connection] + component/Lifecycle + (start [component] + (let [conn (connect-to-database host port)] + (assoc component :connection conn))) + (stop [component] + (.close connection) + (assoc component :connection nil))) +``` + +All these components are then combined together in a `system` map that just bound a keyword to each component. A system is a component that has its own start/stop implementation that is responsible to start all components in dependency order and shut them down in reverse order. + +If a component has dependencies on other components, they are then associated to the system and started first. Since all components return another state of the system; after all components are started, their return values are assoc back to the system. + +Here is an example of a system with 3 components. The `app` components depends on the `db` and `scheduler` components so they will be started first: + +```clojure +(defn system [config-options] + (let [{:keys [host port]} config-options] + (component/system-map + :db (new-database host port) + :scheduler (new-scheduler) + :app (component/using + (example-component config-options) + {:database :db + :scheduler :scheduler})))) +``` + +So, in the above example, `db` and `scheduler` have been injected to `app`. Stuart Sierra mentioned that contrary to `constructor` injections and `setter` injections OOP often use, we could refer this component injections (immutable map) as `associative` injections. + +This is very convenient way to adapt a system to other different situations such as testing for instance. You could just assoc to an in-memory DB and a simplistic schedular in a test-system to run some tests: + +```clojure +(defn test-system + [...] + (assoc (system some-config) + :db (test-db) + :scheduler (test-scheduler))) + +;; then we can call (start test-system) to start all components in deps order. +``` + +Thus, you can isolate what you want to test and even run tests in parallel. So, it is more powerful than `with-redefs` and `binding` because it is not limited by time. Your tests could replace a big portion of your logic quite easily instead of individual vars allowing us to decouple the tests from the rest of the code. + +Finally, we do not want to pass the whole system to every function in all namespaces. Instead, the components library allows you to specify just the component. + +#### Limitations + +However, there are some limitations to this design, the main ones being: + +- `stuartsierra/component` is a whole app buy-in. Your entire app needs to follow this design to get all the benefits from it. +- It is not easy to visually inspect the whole system in the REPL +- cannot start just a part of the system + +#### Other approaches + +Other libraries were created as replacement of component such as [mount](https://github.com/tolitius/mount) and [integrant](https://github.com/weavejester/integrant). + +- Mount highlights their differences with Component in [here](https://github.com/tolitius/mount/blob/master/doc/differences-from-component.md#differences-from-component). +- Integrant highlights their differences with Component in [here](https://github.com/weavejester/integrant/blob/master/README.md#rationale). + +## πŸ”Έ Fun-map + +[fun-map](https://github.com/robertluo/fun-map) is yet another replacement of [component](https://github.com/stuartsierra/component), but it does more than just providing state management. + +The very first goal of `fun-map` is to blur the line between identity, state and function, but in a good way. `fun-map` combines the idea of [lazy-map](https://github.com/originrose/lazy-map) and [plumbing](https://github.com/plumatic/plumbing) to allow lazy access to map values regardless of the types or when these values are accessed. + +### Wrappers + +In order to make the map’s values accessible on demand regardless of the type (delay, future, atom etc), map’s value arguments are wrapped to encapsulate the way the underlying values are accessed and return the values as if they were just data in the first place. + +For instance: + +```clojure +(def m (fun-map {:numbers (delay [3 4])})) + +m +;=> {:numbers [3 4]} + +(apply * (:numbers m)) +;=> 12 + +;; the delay will be evaluated just once +``` + +You can see that the user of the map is not impacted by the `delay` and only see the deref value as if it were just a vector in the first place. + +#### Associative dependency injections + +Similar to what we discussed regarding how the [component](https://github.com/stuartsierra/component) library assoc dependencies in order, fun-map as a wrapper macro `fk` to use other `:keys` as arguments of their function. + +Let’s have a look at an example of `fun-map`: + +```clojure +(def m (fun-map {:numbers [3 4] + :cnt (fw {:keys [numbers]} + (count numbers)) + :average (fw {:keys [numbers cnt]} + (/ (reduce + 0 numbers) cnt))})) +``` + +In the fun-map above, you can see that the key `:cnt` takes for argument the value of the key `:numbers`. The key `:average` takes for arguments the values of the key `:numbers` and `:cnt`. + +Calling the `:average` key will first call the keys it depends on, meaning `:cnt` and `:number` then call the `:average` and returns the results: + +```clojure +(:average m) +;=> 7/2 +``` + +We recognized the same dependency injections process highlighted in the Component section. + +Furthermore, fun-map provides a convenient wrapper `fnk` macro to destructure directly the keys we want to focus on: + +```clojure +(def m (fun-map {:numbers [3 4] + :cnt (fnk [numbers] + (count numbers)) + :average (fnk [numbers cnt] + (/ (reduce + 0 numbers) cnt))})) +``` + +As explained above, we could add some more diverse values, it wouldn’t be perceived by the user of the map: + +```clojure + (def m (fun-map {:numbers (delay [3 4]) + :cnt (fnk [numbers] + (count numbers)) + :multiply (fnk [numbers] + (atom (apply * numbers))) + :average (fnk [numbers cnt] + (/ (reduce + 0 numbers) cnt))})) + +(:multiply m) +;=> 12 + +m +;=> {:numbers [3 4] :cnt 2 :multiply 12 :average 7/2} + +``` + +### System + +#### Life Cycle Map + +Wrappers take care of getting other keys’s values (with eventual options we did not talk about so far). However, to get the life cycle we describe in the Component library section, we still need a way to + +- start each underlying values (components) in dependency order (other keys) +- close each underlying values in reverse order of their dependencies + +fun-map provides a `life-cycle-map` that allows us to specify the action to perform when the component is getting started/closed via the `closeable`. + +- `touch` start the system, meaning it injects all the dependencies in order. the first argument of `closeable` (eventually deref in case it is a delay or atom etc) is returned as value of the key. +- `halt!` close the system, meaning it executes the second argument of `closeable` which is a function taking no param. It does so in reverse order of the dependencies + +Here is an example: + +```clojure +(def system + (life-cycle-map ;; to support the closeable feature + {:a (fnk [] + (closeable + 100 ;; 1) returned at touch + #(println "a closed") ;; 4) evaluated at halt! + )) + :b (fnk [a] + (closeable + (inc a) ;; 2) returned at touch + #(println "b closed") ;; 3) evaluated at halt! + ))})) + +(touch system1) +;=> {:a 100, :b 101} + +(halt! system1) +;=> b closed +; a closed +; nil +``` + +`closeable` takes 2 params: +- value returned when we call the key of the fun-map. +- a no-arg function evaluated in reverse order of dependencies. + +#### Testing + +Same as for Component, you can easily dissoc/assoc/merge keys in your system for testing purposes. You need to be sure to build your system before `touch`. + +```clojure +(def test-system + (assoc system :a (fnk [] + (closeable + 200 + #(println "a closed v2"))))) + +(touch test-system) +;=> {:a 200, :b 201} + +(halt! test-system) +;=> b closed +; a closed v2 +; nil +``` + +fun-map also support other features such as function call tracing, value caching or lookup for instance. More info in the readme. + +## πŸ”Έ Fun-Map applied to flybot.sg + +To see Fun Map in action, refer to the doc [Fun-Map applied to flybot.sg](./fun-map-applied-to-flybot.md). diff --git a/docs/lasagna-stack/lasagna-pull-applied-to-flybot.md b/docs/lasagna-stack/lasagna-pull-applied-to-flybot.md new file mode 100644 index 0000000..ea9410a --- /dev/null +++ b/docs/lasagna-stack/lasagna-pull-applied-to-flybot.md @@ -0,0 +1,402 @@ +# Lasagna-pull applied to flybot.sg + +## πŸ”Έ Prerequisites + +If you are not familiar with [lasagna-pull](https://github.com/flybot-sg/lasagna-pull), please refer to the doc [Lasagna Pull Rational](./lasagna-pull.md) + +## πŸ”Έ Goal + +In this document, I will show you how we leverage `lasagna-pull` in our app to define a pure data API. + +## πŸ”Έ Defines API as pure data + +A good use case of the pattern is as parameter in a post request. + +In our backend, we have a structure representing all our endpoints: + +```clojure +;; BACKEND data structure +(defn pullable-data + "Path to be pulled with the pull-pattern. + The pull-pattern `:with` option will provide the params to execute the function + before pulling it." + [db session] + {:posts {:all (fn [] (get-all-posts db)) + :post (fn [post-id] (get-post db post-id)) + :new-post (with-role session :editor + (fn [post] (add-post db post))) + :removed-post (with-role session :editor + (fn [post-id user-id] (delete-post db post-id user-id)))} + :users {:all (with-role session :owner + (fn [] (get-all-users db))) + :user (fn [id] (get-user db id)) + :removed-user (with-role session :owner + (fn [id] (delete-user db id))) + :auth {:registered (fn [id email name picture] (register-user db id email name picture)) + :logged (fn [] (login-user db (:user-id session)))} + :new-role {:admin (with-role session :owner + (fn [email] (grant-admin-role db email))) + :owner (with-role session :owner + (fn [email] (grant-owner-role db email)))} + :revoked-role {:admin (with-role session :owner + (fn [email] (revoke-admin-role db email)))}}}) +``` + +This resembles a REST API structure. + +Since the API β€œroute” information is contained within the pattern keys themselves, all the http requests with a pattern as params can hit the same backend URI. + +So we have a single route for all pattern http request: + +```clojure +(into (auth/auth-routes oauth2-config) + [["/pattern" {:post ring-handler}] ;; all requests with pull pattern go here + ["/users/logout" {:get (auth/logout-handler client-root-path)}] + ["/oauth/google/success" {:get ring-handler :middleware [[auth/authentification-middleware client-root-path]]}] + ["/*" {:get {:handler index-handler}}]]) +``` + +Therefore the pull pattern: + +- Describes the API routes +- Provides the data expected by the server in its `:with` option for the concerned endpoints +- Describes what is asked by the client to only return relevant data +- Can easily perform authorization + +## πŸ”Έ Example: pull a post + +For instance, getting a specific post, meaning with the β€œroute”: `:posts :post`, can be done this way: + +```clojure +((pull/qfn + {:posts + {(list :post :with [s/post-1-id]) ;; provide required params to pullable-data :post function + {:post/id '? + :post/page '? + :post/css-class '? + :post/creation-date '? + :post/last-edit-date '? + :post/author {:user/id '? + :user/email '? + :user/name '? + :user/picture '? + :user/roles [{:role/name '? + :role/date-granted '?}]} + :post/last-editor {:user/id '? + :user/email '? + :user/name '? + :user/picture '? + :user/roles [{:role/name '? + :role/date-granted '?}]} + :post/md-content '? + :post/image-beside {:image/src '? + :image/src-dark '? + :image/alt '?} + :post/default-order '?}}} + '&? ;; bind the whole data + )) +; => +{:posts + {:post + #:post{:id #uuid "64cda032-b4e4-431e-bd85-0dbe34a8feeb" ;; s/post-1-id + :page :home + :css-class "post-1" + :creation-date #inst "2023-01-04T00:00:00.000-00:00" + :last-edit-date #inst "2023-01-05T00:00:00.000-00:00" + :author #:user{:id "alice-id" + :email "alice@basecity.com" + :name "Alice" + :picture "alice-pic" + :roles [#:role{:name :editor + :date-granted + #inst "2023-01-02T00:00:00.000-00:00"}]} + :last-editor #:user{:id "bob-id" + :email "bob@basecity.com" + :name "Bob" + :picture "bob-pic" + :roles [#:role{:name :editor + :date-granted + #inst "2023-01-01T00:00:00.000-00:00"} + #:role{:name :admin + :date-granted + #inst "2023-01-01T00:00:00.000-00:00"}]} + :md-content "#Some content 1" + :image-beside #:image{:src "https://some-image.svg" + :src-dark "https://some-image-dark-mode.svg" + :alt "something"} + :default-order 0}}} +``` + +It is important to understand that the param `s/post-1-id` in `(list :post :with [#uuid s/post-1-id])` was passed to `(fn [post-id] (get-post db post-id))` in `pullable-data`. + +The function returned the post fetched from the db. + +We decided to fetch all the information of the post in our pattern but we could have just fetch some of the keys only: + +```clojure +((pull/qfn + {:posts + {(list :post :with [s/post-1-id]) ;; only fetch id and page even though all the other keys have been returned here + {:post/id '? + :post/page '?}}} + '&?)) +=> {:posts + {:post + {:post/id #uuid "64cda032-b4e4-431e-bd85-0dbe34a8feeb" + :post/page :home}}} +``` + +The function `(fn [post-id] (get-post db post-id))` returned **all** the post keys but we only select the `post/id` and `post/page`. + +So we provided required param `s/post-1-id` to the endpoint `:post` and we also specified what information we want (`:post/id` and `:post/page`). + +You can start to see how convenient that is as a frontend request to the backend. our post request body can just be a `pull-pattern`! (more on this further down in the doc). + +## πŸ”Έ Post data validation + +It is common to use [malli](https://github.com/metosin/malli) schema to validate data. + +Here is the malli schema for the post data structure we used above: + +```clojure +(def post-schema + [:map {:closed true} + [:post/id :uuid] + [:post/page :keyword] + [:post/css-class {:optional true} [:string {:min 3}]] + [:post/creation-date inst?] + [:post/last-edit-date {:optional true} inst?] + [:post/author user-schema] + [:post/last-editor {:optional true} user-schema] + [:post/md-content [:and + [:string {:min 10}] + [:fn + {:error/message "Level 1 Heading `#` missing in markdown."} + md/has-valid-h1-title?]]] + [:post/image-beside + {:optional true} + [:map + [:image/src [:string {:min 10}]] + [:image/src-dark [:string {:min 10}]] + [:image/alt [:string {:min 5}]]]] + [:post/default-order {:optional true} nat-int?]]) +``` + +## πŸ”Έ Pattern data validation + +`lasagna-pull` also allows us to provide schema alongside the pattern to validate 2 things: + +- the pattern format is correct +- the pattern content respects a malli schema + +This is very good because we can have a malli schema for the entire `pullable-data` structure like so: + +```clojure +(def api-schema + "All keys are optional because it is just a data query schema. + maps with a property :preserve-required set to true have their keys remaining unchanged." + (all-keys-optional + [:map + {:closed true} + [:posts + [:map + [:post [:=> [:cat :uuid] post-schema]] ;; route from our get post example + [:all [:=> [:cat] [:vector post-schema]]] + [:new-post [:=> [:cat post-schema-create] post-schema]] + [:removed-post [:=> [:cat :uuid :string] post-schema]]]] + [:users + [:map + [:user [:=> [:cat :string] user-schema]] + [:all [:=> [:cat] [:vector user-schema]]] + [:removed-user [:=> [:cat :string] user-schema]] + [:auth [:map + [:registered [:=> [:cat :string user-email-schema :string :string] user-schema]] + [:logged [:=> [:cat] user-schema]]]] + [:new-role [:map + [:admin [:=> [:cat user-email-schema] user-schema]] + [:owner [:=> [:cat user-email-schema] user-schema]]]] + [:revoked-role [:map + [:admin [:=> [:cat user-email-schema] user-schema]]]]]]])) +``` + +If we go back to the scenario where we want to fetch a specific post from the DB, we can see that we are indeed having a function as params of the key `:post` that expects one param: a uuid: + +```clojure +[:post [:=> [:cat :uuid] post-schema]] +``` + +It corresponds to the pattern part: + +```clojure +(list :post :with [s/post-1-id]) +``` + +And `lasagna-pull` provides validation of the function’s params which is very good to be sure the proper data is sent to the server! + +Plus, in case the params given to one of the routes are not valid, the function won’t even be executed. + +So now we have a way to do post request to our backend providing a pull-pattern as the request body and our server can validate this pattern format and content as the data is being pulled. + +## πŸ”Έ Pattern query context + +### How it works + +Earlier, I asked you to assume that the function from `pullable-data` was returning a post data structure. + +In reality, it is a bit more complex than this because what is returned by the different functions (endpoints) in `pullable-data` is a map. For instance: + +```clojure +;; returned by get-post +{:response (db/get-post db post-id)} ;; note the response key here + +;; returned by register-user +{:response user + :effects {:db {:payload [user]}} ;; the db transaction description to be made + :session {:user-id user-id} ;; the user info to be added to the session +} +``` + +This is actually a problem because our pattern for a post is: + +```clojure +{:posts + {(list :post :with [s/post-1-id]) + {:post/id '?}}} +``` + +and with what is returned by `(fn [post-id] (get-post db post-id))`, we should have: + +```clojure +{:posts + {(list :post :with [s/post-1-id]) + {:response ;; note the response here + {:post/id '?}}}} +``` + +Also, in case of a user registration for instance, you saw that we have other useful information such as +- effects: the db transaction to add the user to the db +- session: some user info to add to the session. + +However we do not want to pull the `effects` and `session`. We just want a way to accumulate them somewhere. + +We could perform the transaction directly and return the post, but we don't want that. + +We prefer to accumulate side effects descriptions and execute them all at once in a dedicated `executor`. + +The `response` needs to be added to the pulled data, but the `effects` and `session` need to be stored elsewhere and executed later on. + +This is possible via a `modifier` and a `finalizer` context in the `pull/query` API. + +In our case, we have a `mk-query` function that uses a `modifier` and `finalizer` to achieve what I described above: + +```clojure +(defn mk-query + "Given the pattern, make an advance query using a context: + modifier: gather all the effects description in a coll + finalizer: assoc all effects descriptions in the second value of pattern." + [pattern] + (let [effects-acc (transient []) + session-map (transient {})] + (pull/query + pattern + (pull/context-of + (fn [_ [k {:keys [response effects session error] :as v}]] + (when error + (throw (ex-info "executor-error" error))) + (when session ;; assoc session to the map session + (reduce + (fn [res [k v]] (assoc! res k v)) + session-map + session)) + (when effects ;; conj the db transaction description to effects vector + (conj! effects-acc effects)) + (if response + [k response] + [k v])) + #(assoc % ;; returned the whole pulled data and assoc the effects and session to it + :context/effects (persistent! effects-acc) + :context/sessions (persistent! session-map)))))) +``` + +### Example of post creation + +Let’s have a look at an example: + +We want to add a new post. When we make a request for a new post, if everything works fine, the pullable-data function at the route `:new-post` returns a map such as: + +```clojure +{:response full-post ;; the pullable data to return to the client + :effects {:db {:payload posts}} ;; the new posts to be added to the db +} +``` + +The pull pattern for such request can be like this: + +```clojure +{:posts + {(list :new-post :with [post-in]) ;; post-in is a full post to be added with all required keys + {:post/id '? + :post/page '? + :post/default-order '?}}} +``` + +The `post-in` is provided to the pullable-data function of the key `:new-post`. + +The function of `add-post` actually determine all the new `:post/default-order` of the posts given the new post. That is why we see in the side effects that several `posts` are returned because we need to have their order updated in db. + +Running this pattern with the pattern **context** above returns: + +```clojure +{&? {:posts {:new-post {:post/id #uuid "64cda032-3dae-4845-b7b2-e4a6f9009cbd" + :post/page :home + :post/creation-date #inst "2023-01-07T00:00:00.000-00:00" + :post/default-order 2}}} + :context/effects [{:db {:payload [{:post/id #uuid "64cda032-3dae-4845-b7b2-e4a6f9009cbd" + :post/page :home + :post/md-content "#Some content 3" + :post/creation-date #inst "2023-01-07T00:00:00.000-00:00" + :post/author {:user/id "bob-id"} + :post/default-order 2}]}}] + :context/sessions {}} +``` + +- the response has been returned from the :with function to the pattern in the β€˜&? key +- the effects have been accumulated and assoc in `:context/effects` +- there was no data to be added to the session + +Then, in the ring response, we can just return the value of `&?` + +Also, the effects can be executed in a dedicated executor functions all at once. + +This allows us to deal with pure data until the very last moment when we run all the side effects (db transaction and session) in one place only we call `executor`. + +## πŸ”Έ Saturn handler + +You might have noticed a component in our system called the `saturn-handler`. The `ring-handler` depends on it. + +In order to isolate the side effects as much as we can, our endpoints from our `pullable-data`, highlighted previously, do not perform side effects but return **descriptions** in pure data of the side effects to be done. These side effects are the ones we gather in `:context/effects` and `:context/sessions` using the pull-pattern's query context. + +The saturn-handler returns a map with the `response` (data pulled and requested in the client pattern) to be sent to the client, the `effect-desc` to be perform (in our case, just db transactions) and the `session` update to be done: + +```clojure +(defn saturn-handler + "A saturn handler takes a ring request enhanced with additional keys form the injectors. + The saturn handler is purely functional. + The description of the side effects to be performed are returned and they will be executed later on in the executors." + [{:keys [params body-params session db]}] + (let [pattern (if (seq params) params body-params) + data (op/pullable-data db session) + {:context/keys [effects sessions] :as resp} + (pull/with-data-schema v/api-schema ((mk-query pattern) data))] + {:response ('&? resp) + :effects-desc effects + :session (merge session sessions)})) +``` + +You can also notice that the data is being validated via `pull/with-data-schema`. In case of validation error, since we do not have any side effects done during the pulling, an error will be thrown and no mutations will be done. + +Having no side-effects at all makes it way easier to tests and debug and it is more predictable. + +Finally, the `ring-handler` will be the component responsible to **execute** all the side effects at once. + +So the `saturn-handler` purpose was to be sure the data is being pulled properly, validated using malli, and that the side effects descriptions are gathered in one place to be executed later on. diff --git a/docs/lasagna-stack/lasagna-pull.md b/docs/lasagna-stack/lasagna-pull.md new file mode 100644 index 0000000..c3d9d46 --- /dev/null +++ b/docs/lasagna-stack/lasagna-pull.md @@ -0,0 +1,192 @@ +# Lasagna Pull Rational + +[flybot-sg/lasagna-pull](https://github.com/flybot-sg/lasagna-pull) by @robertluo aims at precisely select from deep data structure in Clojure. + +## πŸ”Έ Goal + +In this document, I will show you the benefit of `pull-pattern` in pulling nested data. + +## πŸ”Έ Rational + +In Clojure, it is very common to have to precisely select data in nested maps. the Clojure core `select-keys` and `get-in` functions do not allow to easily select in deeper levels of the maps with custom filters or parameters. + +One of the libraries of our `lasagna-stack` is [flybot-sg/lasagna-pull](https://github.com/flybot-sg/lasagna-pull). It takes inspiration from the [datomic pull API](https://docs.datomic.com/on-prem/query/pull.html) and the library [redplanetlabs/specter](https://github.com/redplanetlabs/specter). + +`lasagna-pull` aims at providing a clearer pattern that the datomic pull API. + +It also allows the user to add options on the selected keys (filtering, providing params to values which are functions etc). It supports less features than the `specter` library but the syntax is more intuitive and covers all major use cases you might need to select the data you want. + +Finally, a [metosin/malli](https://github.com/metosin/malli) schema can be provided to perform data validation directly using the provided pattern. This allows the client to prevent unnecessary pulling if the pattern does not match the expected shape (such as not providing the right params to a function, querying the wrong type etc). + +## πŸ”Έ A query language to select deep nested structure + +Selecting data in nested structure is made intuitive via a pattern that describes the data to be pulled following the shape of the data. + +### Simple query cases + +Here are some simple cases to showcase the syntax: + +- query a map + +```clojure +(require '[sg.flybot.pullable :as pull]) + +((pull/query '{:a ? :b {:b1 ?}}) + {:a 1 :b {:b1 2 :b2 3}}) +;=> {&? {:a 1, :b {:b1 2}}} +``` + +- query a sequence of maps + +```clojure +((pull/query '[{:a ? :b {:b1 ?}}]) + [{:a 1 :b {:b1 2 :b2 3}} + {:a 2 :b {:b1 2 :b2 4}}]) +;=> {&? [{:a 1, :b {:b1 2}} {:a 2, :b {:b1 2}}]} +``` + +- query nested sequences and maps + +```clojure +((pull/query '[{:a ? + :b [{:c ?}]}]) + [{:a 1 :b [{:c 2}]} + {:a 11 :b [{:c 22}]}]) +;=> {&? [{:a 1, :b [{:c 2}]} {:a 11, :b [{:c 22}]}]} +``` + +Let’s compare datomic pull and lasagna pull query with a simple example: + +- datomic pull + +```clojure +(def sample-data + [{:a 1 :b {:b1 2 :b2 3}} + {:a 2 :b {:b1 2 :b2 4}}]) + +(pull ?db + [:a {:b [:b1]}] + sample-data) +``` + +- Lasagna pull +```clojure +((pull/query '[{:a ? :b {:b1 ?}}]) + sample-data) +;=> {&? [{:a 1, :b {:b1 2}} {:a 2, :b {:b1 2}}]} +``` + +A few things to note + +- lasagna-pull uses a map to query a map and surround it with a vector to query a sequence which is very intuitive to use. +- `?` is just a placeholder on where the value will be after the pull. +- lasagna-pull returns a map with your pulled data in a key `&?`. + +### Query specific keys + +You might not want to fetch the whole path down to a leaf key, you might want to query that key and store it in a dedicated var. It is possible to do this providing a var name after the placeholder `?` such as `?a` for instance. The key `?a` will then be added to the result map along side the `&?` that contains the whole data structure. + +Let’s have a look at an example. + +Let’s say we want to fetch specific keys in addition to the whole data structure: + +```clojure +((pull/query '{:a ?a + :b {:b1 ?b1 :b2 ?}}) + {:a 1 :b {:b1 2 :b2 3}}) +; => {?& {:a 1 :b {:b1 2 :b2 3}} ;; all nested data structure +; ?a 1 ;; var a +; ?b1 2 ;; var b1 + } +``` + +The results now contain the logical variable we selected via `?a` and `?b1`. Note that the `:b2` key has just a `?` placeholder so it does not appear in the results map keys. + +It works also for sequences: + +```clojure +;; logical variable for a sequence +((pull/query '{:a [{:b1 ?} ?b1]}) + {:a [{:b1 1 :b2 2} {:b1 2} {}]}) +;=> {?b1 [{:b1 1} {:b1 2} {}] +; &? {:a [{:b1 1} {:b1 2} {}]}} +``` + +Note that `'{:a [{:b1 ?b1}]}` does not work because the logical value cannot be the same for all the `b1` keys: + +```clojure +((pull/query '{:a [{:b1 ?b1}]}) + {:a [{:b1 1 :b2 2} {:b1 2} {}]}) +;=> {&? {:a [{:b1 1} nil nil]}} ;; not your expected result +``` + +## πŸ”Έ A query language to select structure with params and filters + +Most of the time, just selecting nested keys is not enough. We might want to select the key if certain conditions are met, or even pass a parameter if the value of the key is a function so we can run the function and get the value. + +With library like [redplanetlabs/specter](https://github.com/redplanetlabs/specter), you have different possible transformations using diverse [macros](https://github.com/redplanetlabs/specter/wiki/List-of-Macros) which is an efficient way to select/transform data. The downside is that it introduces yet another syntax to get familiar with. + +`lasagna-pull` supports most of the features at a key level. + +Instead of just providing just the key you want to pull in the pattern, you can provide a list with the key as first argument and the options as the rest of the list. + +The transformation is done at the same time as the selection, the pattern can be enhanced with options: + +- not found + +```clojure +((pull/query '{(:a :not-found ::not-found) ?}) {:b 5}) +;=> {&? {:a :user/not-found}} +``` + +- when + +```clojure +((pull/query {(:a :when even?) '?}) {:a 5}) +;=> {&? {}} ;; empty because the value of :a is not even +``` + +- with + +If the value of a query is a function, using `:with` option can invoke it and returns the result instead: + +```clojure +((pull/query '{(:a :with [5]) ?}) {:a #(* % 2)}) +;=> {&? {:a 10}} ;; the arg 5 was given to #(* % 2) and the result returned +``` + +- batch + +Batched version of :with option: + +```clojure +((pull/query '{(:a :batch [[5] [7]]) ?}) {:a #(* % 2)}) +;=> {&? {:a (10 14)}} +``` + +- seq + +Apply to sequence value of a query, useful for pagination: + +```clojure +((pull/query '[{:a ? :b ?} ? :seq [2 3]]) [{:a 0} {:a 1} {:a 2} {:a 3} {:a 4}]) +;=> {&? ({:a 2} {:a 3} {:a 4})} +``` + +As you can see with the different options above, the transformations are specified within the selected keys. Unlike specter however, we do not have a way to apply transformation to all the keys for instance. + +## πŸ”Έ Pattern validation with Malli schema + +We can optionally provide a [metosin/malli](https://github.com/metosin/malli) schema to specify the shape of the data to be pulled. + +The client malli schema provided is actually internally "merged" to a internal schema that checks the pattern shape so both the pattern syntax and the pattern shape are validated. + +## πŸ”Έ Context + +You can provide a context to the query. You can provide a `modifier` and a `finalizer`. + +This context can help you gathering information from the query and apply a function on the results. + +## πŸ”Έ Lasagna Pull applied to flybot.sg + +To see Lasagna Pull in action, refer to the doc [Lasagna Pull applied to flybot.sg](./lasagna-pull-applied-to-flybot.md).