Merge pull request #4 from eclipse-iofog/release/1.2.0

Release 1.2.0

CONTRIBUTING

@@ -0,0 +1,52 @@
+# Contributing to Eclipse test-runer
+
+Thanks for your interest in this project.
+
+## Project description
+
+test-runner is a simple framework to test across our ECN nodes in a networked environment to ensure changes don't break functionality
+of the project
+
+* https://projects.eclipse.org/projects/iot.iofog
+
+## Developer resources
+
+Information regarding source code management, builds, coding standards, and
+more.
+
+* https://projects.eclipse.org/projects/iot.iofog/developer
+
+The project maintains the following source code repositories
+
+* https://github.com/eclipse-iofog
+* http://git.eclipse.org/c/iofog/org.eclipse.iofog.git
+
+This project uses Bugzilla to track ongoing development and issues.
+
+* Search for issues: https://eclipse.org/bugs/buglist.cgi?product=IoFog
+* Create a new report: https://eclipse.org/bugs/enter_bug.cgi?product=IoFog
+
+Be sure to search for existing bugs before you create another one. Remember that
+contributions are always welcome!
+
+## Eclipse Contributor Agreement
+
+Before your contribution can be accepted by the project team contributors must
+electronically sign the Eclipse Contributor Agreement (ECA).
+
+* http://www.eclipse.org/legal/ECA.php
+
+Commits that are provided by non-committers must have a Signed-off-by field in
+the footer indicating that the author is aware of the terms by which the
+contribution has been provided to the project. The non-committer must
+additionally have an Eclipse Foundation account and must have a signed Eclipse
+Contributor Agreement (ECA) on file.
+
+For more information, please see the Eclipse Committer Handbook:
+https://www.eclipse.org/projects/handbook/#resources-commit
+
+## Contact
+
+Contact the project developers via the project's "dev" list.
+
+* https://dev.eclipse.org/mailman/listinfo/iofog-dev

Dockerfile

@@ -7,19 +7,22 @@ FROM python:3
 # Install our deps
 RUN apt-get update -qq && apt-get install -y \
     sudo \
+    jq \
     python-pycurl \
-    vim \
-    jq
+    rubygems
+
+# Set the Kubernetes version as found in the UCP Dashboard or API
+RUN curl -LO https://storage.googleapis.com/kubernetes-release/release/v1.11.5/bin/linux/amd64/kubectl && \
+    chmod +x ./kubectl && mv ./kubectl /usr/local/bin/kubectl
 
 RUN git clone https://github.com/sstephenson/bats.git && cd bats && ./install.sh /usr/local
 
 # Install the pyresttest and deps (the basis of all our smoke tests)
-RUN pip install pyresttest jsonschema future shyaml
+RUN pip install pyresttest jsonschema future shyaml jmespath
 
 # Copy over all the files we need
-COPY functions.bash /
 COPY run.bash /
 COPY tests /tests/
 
 # Run our tests
-CMD [ "/run.bash" ]
+ENTRYPOINT ["/bin/sh", "-c", "/run.bash"]

NOTICE

@@ -0,0 +1,41 @@
+# Notices for Eclipse ioFog
+
+This content is produced and maintained by the Eclipse ioFog project.
+
+* Project home: https://projects.eclipse.org/projects/iot.iofog
+
+## Trademarks
+
+Eclipse ioFog is a trademark of the Eclipse Foundation.
+
+## Copyright
+
+All content is the property of the respective authors or their employers. For
+more information regarding authorship of content, please consult the listed
+source code repository logs.
+
+## Declared Project Licenses
+
+This program and the accompanying materials are made available under the terms
+of the Eclipse Public License v. 2.0 which is available at
+http://www.eclipse.org/legal/epl-2.0.
+
+SPDX-License-Identifier: EPL-2.0
+
+## Source Code
+
+The project maintains the following source code repositories:
+
+* https://github.com/eclipse-iofog
+* http://git.eclipse.org/c/iofog/org.eclipse.iofog.git
+
+## Third-party Content
+
+## Cryptography
+
+Content may contain encryption software. The country in which you are currently
+may have restrictions on the import, possession, and use, and/or re-export to
+another country, of encryption software. BEFORE using any encryption software,
+please check the country's laws, regulations and policies concerning the import,
+possession, or use, and re-export of encryption software, to see if this is
+permitted.

README.md

@@ -1,56 +1,53 @@
 # Test Runner
 
+The ioFog Test Runner provides a convenient way to run multiple smoke test suites on a deployed Edge Compute Network (ECN).
+
+The suite by default does not run any non-smoke tests, i.e. tests that would invalidate or potentially break the ECN, therefore it can be used on staging a productions ECNs to verify correct deployment.  
+
+The following test suites are available:
+* Controller REST API smoke tests
+* Agent CLI smoke tests
+* Basic microservice deployment integration tests
+
+Note that some of additional test suites are automatically skipped as of this release of Test Runner.
+
 ## Usage
 
-It is recommended you run Test Runner using docker-compose.
+| Test suite | Description | Required configuration |
+| --- | --- | --- |
+| Controller REST API smoke tests | Basic REST API tests on Controller instance | <ul><li>CONTROLLER</li><li>CONTROLLER_EMAIL</li><li>CONTROLLER_PASSWORD</li></ul> |
+| Agent CLI smoke tests | Runs Agent tests by SSHing into the Agent nodes and interacting using Agent CLI | <ul><li>AGENTS</li></ul> |
+| Basic microservice deployment integration tests | Sets up users and catalog entries, deploys and destroys microservices on each Agent | <ul><li>CONTROLLER</li><li>CONTROLLER_EMAIL</li><li>CONTROLLER_PASSWORD</li><li>AGENTS</li></ul> |
 
-You must provide agents.conf and a corresponding id_ecdsa/id_ecdsa.pub pair in /conf of the runner. This should be done through a volume in your docker-compose.yml.
 
-For local deployments:
-```
-version: "3"
-services:
-    test-runner:
-        image: iofog/test-runner-develop:latest
-        container_name: test-runner
-        environment:
-            - LOCAL=1
-        network_mode: "bridge"
-        external_links: 
-            - iofog-controller
-            - iofog-connector
-            - iofog-agent
-        volumes:
-            - /path/to/host/conf:/conf
-volumes:
-  conf:
-```
+The format of the environment variables is the following:
 
-For remote deployments:
-```
-version: "3"
-services:
-    test-runner:
-        image: iofog/test-runner-develop:latest
-        container_name: test-runner
-        network_mode: "bridge"
-        volumes:
-            - /path/to/host/conf:/conf
-volumes:
-  conf:
-```
+* _CONTROLLER_ - IP:PORT format (e.g. "1.2.3.4:51121")
+* _CONTROLLER_EMAIL_ - existing user identifier in Controller to use for testing (e.g. "user@domain.com")
+* _CONTROLLER_PASSWORD_ - login password for the user (e.g. "#Bugs4Fun")
+* _AGENTS_ - comma separated URI with user and optional port (e.g. root@1.2.3.4:6451,user@6.7.8.9)
+
+Note that whenever _AGENTS_ is specified, you need to mount appropriate ssh keys to /root/.ssh of the test-runner containers. The keys can be in any default SSH position: ~/.ssh/id_dsa, ~/.ssh/id_ecdsa, ~/.ssh/id_ed25519 and ~/.ssh/id_rsa.
 
-Once the docker-compose.yml is ready, run the following.
+Example usage of the test runner with full configuration:
 
+```bash
+docker run --name test-runner \
+        -v ~/.ssh/my_iofog_ssh_key:/root/.ssh/id_rsa \
+        -e CONTROLLER="1.2.3.4:51121" \
+        -e CONTROLLER_EMAIL="user@domain.com" \
+        -e CONTROLLER_PASSWORD="#Bugs4Fun" \
+        -e AGENTS="root@1.2.3.4:6451,user@6.7.8.9" \
+        iofog/test-runner:latest
 ```
-docker-compose pull test-runner
 
-docker-compose up \
-    --build \
-    --abort-on-container-exit \
-    --exit-code-from test-runner \
-    --force-recreate \
-    --renew-anon-volumes
+## Test Results
+
+The output of this test-suite currently is a single XML File that is Junit-XML compliant.
+This can be used in development pipelines on most DevOps infrastructure to display test results
+in easy to consume factor.
+
+This is used in Azure Pipelines to display each builds pass/fail for both entire test suites
+as well as for specific tests we may see multiple failures in, to help identify issues that may not be consistent.
 
-docker-compose down -v
-```
+The test output is file is `TEST-RESULTS.xml` and is stored in `/root/test-results` of the test-runner container.   

azure-pipelines.yml

@@ -1,89 +1,58 @@
 trigger:
   tags:
     include:
-      - v.*
+      - v*
   branches:
     include:
       - develop
       - master
-      - release
+      - release*
   paths:
     exclude:
       - README.md
 
 pr:
   - master
+  - develop
 
 variables:
   repository: 'focal-freedom-236620/test-runner'
-  type: $(Build.SourceBranchName)
-  commit: $(Build.SourceVersion)
+  buildTag: $(Build.BuildId)
 
 jobs:
-  - job: TriggerOtherJobs
-    dependsOn: TestRunner
-    pool: server
-    variables:
-      tag: $[ dependencies.TestRunner.outputs['setVarStep.tag'] ]
-
-    steps:
-
-      - task: InvokeRESTAPI@1
-        displayName: 'Trigger platform job'
-        inputs:
-          connectionType: 'connectedServiceName'
-          serviceConnection: 'Pipelines'
-          method: 'POST'
-          urlSuffix: '/edgeworx/_apis/build/builds?api-version=5.0'
-          body: "{\"Parameters\":\"{\\\"images.runner\\\": \\\"gcr.io/$(repository):$(tag)\\\"}\", \"Definition\":{\"id\":5}}"
-          waitForCompletion: 'false'
-
-      - task: InvokeRESTAPI@1
-        displayName: 'Trigger demo job'
-        inputs:
-          connectionType: 'connectedServiceName'
-          serviceConnection: 'Pipelines'
-          method: 'POST'
-          urlSuffix: '/edgeworx/_apis/build/builds?api-version=5.0'
-          body: "{\"Parameters\":\"{\\\"images.runner\\\": \\\"gcr.io/$(repository):$(tag)\\\"}\", \"Definition\":{\"id\":13}}"
-          waitForCompletion: 'false'
-
   - job: TestRunner
     pool:
       vmImage: 'Ubuntu-16.04'
 
     steps:
-
-      - bash: |
-          if [ $(type) = 'develop' ]; then
-             echo "##vso[task.setvariable variable=tag]dev-$(commit)"
-             echo "##vso[task.setvariable variable=tag;isOutput=true]dev-$(commit)"
-          elif [ $(type) = 'release' ]; then
-             echo "##vso[task.setvariable variable=tag]rel-$(commit)"
-             echo "##vso[task.setvariable variable=tag;isOutput=true]rel-$(commit)"
-          elif [ $(type) = 'master' ]; then
-             echo "##vso[task.setvariable variable=tag]0.0.0"
-             echo "##vso[task.setvariable variable=tag;isOutput=true]0.0.0"
-          fi
-        displayName: 'Set image tag variables'
-        name: setVarStep
-      
       - task: Docker@2
-        displayName: 'Build docker'
+        displayName: 'Docker build'
         inputs:
           containerRegistry: 'Edgeworx GCP'
           repository: $(repository)
           command: 'build'
           Dockerfile: "Dockerfile"
           tags: |
-            $(tag)
+            $(buildTag)
+            latest
 
       - task: Docker@2
-        displayName: 'Push docker'
+        displayName: 'Docker push'
         inputs:
           containerRegistry: 'Edgeworx GCP'
           repository: $(repository)
           command: 'push'
           Dockerfile: "Dockerfile"
           tags: |
-            $(tag)
+            $(buildTag)
+            latest
+
+      - script: |
+          echo "gcr.io/${REPOSITORY}:${BUILDTAG}" > $(Build.ArtifactStagingDirectory)/docker-image.txt
+        displayName: 'Save Docker image name to artifact'
+
+      - task: PublishPipelineArtifact@0
+        displayName: 'Publish Docker image name artifact'
+        inputs:
+          artifactName: 'docker-image.txt'
+          targetPath: $(Build.ArtifactStagingDirectory)/docker-image.txt