Documentation/binderhub

doc/RANCHER_PROXMOX.md

@@ -0,0 +1,101 @@
+Deploy FADI with rancher and proxmox
+=============
+
+* [1. Upload ISO on Proxmox Node](#1-Upload-IS0-on-Proxmox-Node)
+* [2. Install Rancher](#2-Install-Rancher)
+* [3. Add docker-machine driver](#1-Add-docker-machine-driver)
+* [4. Create Cluster With Rancher](#2-Create-Cluster-With-Rancher)
+     * [Create Node Template](#Create-Node-Template)
+     * [Create Cluster](#Create-Cluster)
+     * [Create The Nodes](#Create-The-Nodes)
+* [5.Manage the provisioning of the persistent volumes](#5-Manage-the-provisioning-of-the-persistent-volumes)
+* [5. Control Cluster from Local PC](#3-Control-Cluster-from-Local-PC)
+
+
+This page provides information on how to create a kubernetes cluster and deploy FADI using rancher and proxmox.
+
+## 1. Upload ISO on Proxmox Node
+
+<a href="https://www.proxmox.com/" alt="OpenLDAP"> <img src="images/logos/Proxmox.png" width="150px" /></a>
+
+> "Proxmox VE is a complete open-source platform for enterprise virtualization. With the built-in web interface you can easily manage VMs and containers, software-defined storage and networking, high-availability clustering, and multiple out-of-the-box tools on a single solution."
+
+First you need to download the iso rancheros-proxmoxve-autoformat.iso, you can download by clicking [here](https://github.com/rancher/os/releases/download/v1.5.5/rancheros-proxmoxve-autoformat.iso).
+
+Once downloaded you need to upload on your proxmox node.
+## 2. Install Rancher
+
+We consider you have already deploy Rancher. However we give you the instructions that we have followed to deploy our Rancher server : [https://rancher.com/docs/rancher/v2.x/en/installation/other-installation-methods/single-node-docker/advanced/](https://rancher.com/docs/rancher/v2.x/en/installation/other-installation-methods/single-node-docker/advanced/)
+## 3. Add docker-machine driver
+
+You need to allow Rancher to control the Proxmox. We have contributed to upgrade a existing docker-machine driver to make it compatible with Rancher.
+To add this driver in your Rancher, Follow this steps :
+
+![Proxmox driver](images/installation/proxmoxdriver.gif)
+
+Driver Url:
+```
+https://github.com/lnxbil/docker-machine-driver-proxmox-ve/releases/download/v3/docker-machine-driver-proxmoxve.linux-amd64
+```
+
+## 4. Create Cluster With Rancher
+
+<a href="https://www.proxmox.com/" alt="OpenLDAP"> <img src="images/logos/rancher.png" width="150px" /></a>
+
+> "Rancher is open source software that combines everything an organization needs to adopt and run containers in production. Built on Kubernetes, Rancher makes it easy for DevOps teams to test, deploy and manage their applications."
+
+After connecting to rancher, you can follow the following steps
+
+### Create Node Template
+This is where you define the templates you wanna use for your nodes ( both masters and workers ), to do so you can go to: `profile (top right corner)`  > `Node templates` > `Add Template` :
+
+Choose `Proxmoxve`
+![Proxmoxve](images/installation/Proxmoxve.png)
+and then fill the rest of the fields like the IP of the proxmox `i.e. proxmoxHost`, the username/password `i.e. proxmoxUserName, proxmoxUserPassword `, storage of the image file `vmImageFile ` which is in our case `local:iso/rancheros-proxmoxve-autoformat.isot` and coming down to the resources you want to allocate for your node `i.e. nodevmCpuCores, vmMemory, vmStorageSize `.
+
+### Create Cluster
+
+To create your cluster:
+
+ `Cluster`  > `Add Cluster` > `Proxmoxve`
+
+you'll need to give your cluster a name, then precise the nodes in the cluster, at first start with **one master node**, you give it a name, choose of the templates created earlier for that node and then tick all 3 boxes for `etcd`, `Control Plane` and `Worker`, then choose the kubernetes version and click `create`.
+
+> "you'll have to wait the `VM creation`, `the RancherOS install` and `the IP address retrieving` and that might take a while "
+
+ Once the master node gets it's IP address, go to `Cluster`  > `Edit Cluster` and addd an other worker node, untick the worker box from the master node and tick it in the new worker node, it should look something like this:
+ ![Proxmoxve](images/installation/workernode.png)
+
+if a second ( or more ) node ( master or worker ) is needed you can either add an other with a different template the same way we just did or you can add as much nodes as you want using the same template by simply going to  `YourCluster (not global)`  > `nodes` > `+` and it will add an other node of the same kind:
+
+ ![Proxmoxve](images/installation/addnode.png)
+
+
+## 5. Manage the provisioning of the persistent volumes.
+#### StorageOS
+Once all your nodes are up and running, it's time to deploy your services, but before you do you need to set your default PVC for the persistent volumes, to do so we first need to deploy the volume plugin `StorageOS`, go to `YourCluster (not global)`  > `system` > `apps` > `launch` and search for `StorageOS`. make sure all the fields are filled correctly like the following screenshot:
+
+![Proxmoxve](images/installation/StorageOS.png)
+
+and now, launch it 🚀 .
+
+> "launching apps usually takes several minutes, you're going to need to wait a few minutes till the "
+
+Be Careful this service give the posibility to allocate maximum 50Gi with the basic License.
+
+![Proxmoxve](images/installation/StorageOS_limits.png)
+#### Manualy
+
+
+TBT
+
+
+### Deploy FADI
+![defaultpvc](images/installation/defaultpvc.png)
+
+ selector set as default the StorageClass
+#### Longhorn
+7. edit the values yaml of fadi
+8. run fadi
+
+## 4. Control Cluster from Local PC

doc/README.md

@@ -6,5 +6,8 @@ FADI Documentation
 * [Reverse proxy](REVERSEPROXY.md) - Traefik reverse proxy configuration
 * [Security](SECURITY.md) - SSL setup
 * [TSimulus](TSIMULUS.md) - how to simulate sensors and generate realistic data with [TSimulus](https://github.com/cetic/TSimulus)
-
-For tutorials and examples, see the [examples section](../examples/README.md)
+* [Sample self-hosted infrastructure](RANCHER_PROXMOX.md) - How to install FADI on a self hosted infrastructure using 
+    * [Proxmox](https://www.proxmox.com/en/) as a self-hosted private cloud (IaaS) provider, it provides virtual machines for the various Kubernetes nodes.
+    * [Rancher](https://rancher.com/what-is-rancher/what-rancher-adds-to-kubernetes/) to manage (install, provision, maintain, upgrade, ...) several Kubernetes clusters, e.g. when needing several environments or several well separated tenant installations
+
+For tutorials and examples, see the [examples section](../examples/README.md)

examples/binderhub/README.md

@@ -0,0 +1,73 @@
+<a href="https://binderhub.readthedocs.io/en/latest/" alt="BinderHub"><img src="images/binderhub.png" width="200px"/></a>
+# What's Binderhub
+
+*The primary goal of BinderHub is creating custom computing environments that can be used by many remote users. BinderHub enables an end user to easily specify a desired computing environment from a Git repo. BinderHub then serves the custom computing environment at a URL which users can access remotely.*
+
+*BinderHub will build Docker images out of Git repositories, and then push them to a Docker registry so that JupyterHub can launch user servers based on these images*
+
+# Add Binderhub around FADI
+
+We assume that you have already a cluster deployed. If not, you can refer and follow our [Installation guide]() until the point 1.2.2
+
+you will also need to have a valid Docker account.
+
+
+Follow this step to have FADI with Binderhub installed into your cluster:
+1. Clone this repository and go to the binderhub example folder:
+```bash
+git clone https://github.com/cetic/fadi.git fadi
+cd $pwd/fadi/examples/binderhub
+```
+
+2. open the **config.yaml** file to set your docker credentials and the the name of your project to the following inputs:
+```yaml
+config:
+    BinderHub:
+      use_registry: true
+      image_prefix: <DOCKER_ID>/<PROJECT_NAME>-
+registry:
+    username: <DOCKER_ID>
+    password: >DOCKER_PASSWORD>
+```
+
+3. Launch the Helm script, this will deploy all the FADI services and Binderhub on the cluster (and may take some time).
+```bash
+./deploy.sh
+# see deploy.log for connection information to the various services
+```
+
+# Basic example of binderhub workflow
+This example is used to test the deployment of binderhub with a project using a requirement.txt file.
+
+## Input
+the first step is to connect to the BinderHub page.
+
+If you work with **minikube** the command `minikube service list` will allow you to recover the address to copy and paste in your browser. On the other hand in the case of a bare-metal cluster it is the command: `kubectl get svc -n binderhub` which will allow you to recover the service port.
+As this service is of NodePort type, you can use the IP address of any node to reach the BinderHub home page.
+
+Once on the binderhub page, simply fill in the fields with the following (as the figure below):
+- **github repository name or url:** `https://github.com/binder-examples/requirements`
+- **branch:** `master`
+
+
+To finish click on `launch` button
+
+![images/1_input.png](images/1_input.png)
+## Building
+From now, everything is automatic. BinderHub will create a container image based on the github project. depending on the power of your computer this step can take longer or shorter.
+
+![images/2_building.png](images/2_building.png)
+## Pushing
+The image is now built, it will be saved in your docker image repo ([hub.docker.com](hub.docker.com)). It will no longer be necessary to go back through the build stages to access this project. The time of this step will be dependent on your internet connection, indeed the image size is almost 600mb
+
+![images/3_pushing.png](images/3_pushing.png)
+## Server
+The project will automatically be launched in JupyterHub once all these steps are completed.
+
+![images/4_jupyter.png](images/4_jupyter.png)
+## Launch
+You can now enjoy your work environment!
+
+![images/5_notebook.png](images/5_notebook.png)
+# References
+ - [https://binderhub.readthedocs.io/en/latest/](https://binderhub.readthedocs.io/en/latest/)

examples/binderhub/config.yaml

@@ -0,0 +1,29 @@
+config:
+  BinderHub:
+    use_registry: true
+    image_prefix: <DOCKER_ID>/<PROJECT_NAME>-
+registry:
+  username: <DOCKER_ID>
+  password: >DOCKER_PASSWORD>
+
+
+dind:
+  enabled: true
+  daemonset:
+    image:
+      name: docker
+      tag: 18.09.2-dind
+
+jupyterhub:
+  hub:
+    services:
+      binder:
+        apiToken: 8675d9b1ff09ff2502886dfd4f0f36fd45c916372536aa09613cc9c5563d8d1d
+  proxy:
+    secretToken: 613e0ace7628f92bab45478873451f00e65977ca6a61d2f9255667b7bbd71d0e
+    db:
+      type: sqlite-memory
+    service:
+      type: NodePort
+      nodePorts:
+        http: 30902

examples/binderhub/deploy.sh

@@ -0,0 +1,53 @@
+#!/usr/bin/env bash
+# this script will deploy the various FADI services on a Kubernetes cluster using Helm and kubectl
+# usage: ./deploy.sh [namespace]
+set -o errexit
+
+LOG_FILE="deploy.log"
+[ -e ${LOG_FILE} ] && rm ${LOG_FILE}
+exec > >(tee -a ${LOG_FILE} )
+exec 2> >(tee -a ${LOG_FILE} >&2)
+
+# default namespace is fadi
+NAMESPACE=${1:-fadi}
+
+printf "\n\nCreating namespaces...\n"
+
+kubectl get namespace ${NAMESPACE}  2> /dev/null || kubectl create namespace ${NAMESPACE}
+
+
+printf "\n\nHelm all the things!...\n"
+# add stable repo
+helm repo add stable https://kubernetes-charts.storage.googleapis.com
+
+
+# add cetic helm repo
+helm repo add cetic https://cetic.github.io/helm-charts/
+helm repo update
+
+# create clusterrole for traefik
+kubectl get clusterrole traefik-ingress-controller 2> /dev/null || kubectl create -f ./traefik/rbac-config.yaml
+
+# install/upgrade traefik
+helm upgrade --install traefik stable/traefik -f ./traefik/values.yaml --namespace kube-system
+# install/upgrade FADI
+helm upgrade --install ${NAMESPACE} cetic/fadi -f ./values.yaml --namespace ${NAMESPACE}
+
+printf "\n\nFadi is deployed...Now helm will Install Binderhub Around FADI...\n"
+# install/binderhub around FADI
+kubectl get namespace binderhub 2> /dev/null || kubectl create namespace binderhub
+helm repo add jupyterhub https://jupyterhub.github.io/helm-chart/
+helm repo update
+helm upgrade --install binderhub jupyterhub/binderhub --version=0.2.0-n132.h1a8ce62  -f ./config.yaml --namespace binderhub
+
+sleep 5s
+# Get the node IP where jupyterhub is deployed
+nodeIP=$(kubectl get po -n binderhub -o wide |sed -n '/proxy/p' |awk '{ print $7 }')
+if [ $nodeIP = "minikube" ];then
+  nodeIP=$(minikube ip)
+fi
+
+#Upgrade the binderhub release with hub_url
+printf "\n\n Find jupyterhub deployed at $nodeIP\n"
+helm upgrade binderhub jupyterhub/binderhub --version=0.2.0-n132.h1a8ce62 -f ./config.yaml --set config.BinderHub.hub_url=http://$nodeIP:30902 --namespace binderhub
+printf "\n\nInstallation successful!\n"

examples/binderhub/traefik/rbac-config.yaml

@@ -0,0 +1,38 @@
+---
+kind: ClusterRole
+apiVersion: rbac.authorization.k8s.io/v1beta1
+metadata:
+  name: traefik-ingress-controller
+rules:
+  - apiGroups:
+      - ""
+    resources:
+      - services
+      - endpoints
+      - secrets
+    verbs:
+      - get
+      - list
+      - watch
+  - apiGroups:
+      - extensions
+    resources:
+      - ingresses
+    verbs:
+      - get
+      - list
+      - watch
+---
+kind: ClusterRoleBinding
+apiVersion: rbac.authorization.k8s.io/v1beta1
+metadata:
+  name: traefik-ingress-controller
+roleRef:
+  apiGroup: rbac.authorization.k8s.io
+  kind: ClusterRole
+  name: traefik-ingress-controller
+subjects:
+- kind: ServiceAccount
+  name: traefik-ingress-controller
+  namespace: kube-system
+

examples/binderhub/traefik/values.yaml

@@ -0,0 +1,7 @@
+#loadBalancerIP: "your CloudProvider LoadBalancer IP"
+dashboard:
+  enabled: true
+  serviceType: NodePort
+  ingress:
+    annotations: {}
+  domain: fadi.minikube

examples/binderhub/values.yaml

@@ -0,0 +1,8 @@
+---
+# Default values for FADI are defined here: https://github.com/cetic/helm-fadi.
+# This is a YAML-formatted file.
+# Declare variables to be passed into your templates.
+# Values you define here will overwrite default values from helm-fadi.
+
+jupyterhub:
+  enabled: false