RapiTest

Validate API's Continuously with RapiTest!

Table of Contents

1. About The Project
2. RapiTest
   • Built With
3. Getting Started
   • Docker
   • Locally
4. Usage
5. Test Specific Language, TSL
   • Basic Structure
     • Workflow
     • Stress Test
     • Test
     • Dictionary File
     • Verifications
     • Retain
6. Editor
7. Contributing
8. Contact and Acknowledgements

About The Project

This project was developed as the final project for the obtention of the Master's degree (MEIC - Mestrado em Engenharia Informática e de Computadores) at ISEL (Instituto Superior de Engenharia de Lisboa).

ISEL: https://www.isel.pt/

MEIC: https://www.isel.pt/curso/mestrado/mestrado-em-engenharia-informatica-e-de-computadores

The project was developed in 2 separate cycles, see Contact and Acknowledgements for further details.

RapiTest

RapiTest is a web application for automated and semi-automated black-box testing of RESTful web APIs. It follows a model-based approach, where test cases are automatically derived from the OpenAPI Specification (OAS) of the API under test or manually derived from the Test Specification File (TSL). No access to the source code is required, which makes it possible to test APIs written in any programming language, running in local or remote servers.

The test cases derived from the TSL file allow for some greater customization compared to other tools that only use the OAS, such as:

• Custom HTTP query strings or headers for specific APIs that need an API key
• Native or Custom verifications to specific HTTP requests
• Workflow testing of certain endpoints
• Stress tests to specific workflows

(back to top)

Built With

This app was developed using these frameworks:

• ASP.NET Core
• React.js
• Bootstrap
• Microsoft SQL Server
• RabbitMQ

(back to top)

Getting Started

Docker

The latest versions of the app are available as images in DockerHub, the docker-compose file is available under the folder dockercompose, simply copy it and run the command:

• docker-compose

  docker-compose up

After all images are up and running you can access the app at: https://localhost:8080

Locally

If you want to run it directly without the use of docker here are the steps you need to make

Prerequisites

• Install Visual Studio 2019 (Compatibility with other versions is not guaranteed)
• Install Sql Server 2019 (Compatibility with other versions is not guaranteed)
• (Optional) Install Sql Server Management Studio
• Install RabbitMQ (Default Installation)
• Install Node.js and NPM

Installation

After having installed all the required software:

1. Clone the repo

   git clone https://github.com/DuarteFelicio/RAPITest.git

2. Create a Database

   Example name: RapiTestDB

3. Change Connection String Values
   Open the solution with Visual Studio and go to the appsettings.json file of RAPITest, RunTestsWorkerService and SetupTestsWorkerService projects and change the line:

   //local
   "DefaultConnection": type your connection string here

4. Install NPM packages
   Open a command line in the folder RAPITest\RAPITest\ClientApp and run the command

   npm install

5. Configure the solution
   Make sure the solution is set to run multiple projects, start RAPITest, RunTestsWorkerService and SetupTestsWorkerService and none for ModelsLibrary
6. Create Database Tables
   Open the package manager console (tools -> nuget manager -> package manager console)
   Make sure the default project is RAPITest
   Run the command:

   EntityFrameWorkCore\Update-Database -Context ApplicationDbContext

7. Run and enjoy!

(back to top)

Usage

Demonstration on youtube coming soon.

(back to top)

Test Specific Language, TSL

As mentioned in the introduction, this app takes advantage of TSL files.

These are YAML files with the purpose of defining specific tests based on HTTP requests in order to specify tests that couldn't reliably be made automaticly with just the API's specification.

The app supports the creation of these files trough a simple UI, however not every functionality is supported trough this UI, leaving some functionalities to only a manual creation.

Basic Structure

You can write TSL files in YAML. A sample TSL definition written in YAML looks like:

  - WorkflowID: crud_pet

    Stress:
      Count: 40
      Threads: 5
      Delay: 0

    Tests:

    - TestID: createPet
      Server: "https://petstore3.swagger.io/api/v3"
      Path: "/pet"
      Method: Post
      Headers:
        - Content-Type:application/json
        - Accept:application/json
      Body: "$ref/dictionary/petExample"
      Retain:
        - petId#$.id
      Verifications:
        - Code: 200
          Schema: "$ref/definitions/Pet"

    - TestID: readPet
      Server: "https://petstore3.swagger.io/api/v3"
      Path: "/pet/{petId}"
      Method: Get
      Headers:
        - Accept:application/xml
      Verifications:
        - Code: 200
          Schema: "$ref/definitions/Pet"
          Custom: ["CustomVerification.dll"]

    - TestID: updatePet
      Server: "https://petstore3.swagger.io/api/v3"
      Path: "/pet/{petId}"
      Query:
        - name=doggie
        - status=sold
      Method: Post
      Headers:
        - Accept:application/xml
      Verifications:
        - Code: 200
          Schema: "$ref/dictionary/petSchemaXml"

    - TestID: deletePet
      Server: "https://petstore3.swagger.io/api/v3"
      Path: "/pet/{petId}"
      Method: Delete
      Headers:
        - Accept:application/json
      Verifications:
        - Code: 200     

Workflow

Every TSL file must include atleast one workflow

  - WorkflowID: crud_pet

The workflow needs an ID, which must be unique across all workflows.
One workflow must be comprised of one or more tests and optionally one stress test. All the tests inside the workflow are guaranteed to be executed sequentially which is usefull for situations where the output of one test influences the input of the other.

Stress Test

Optionally, one workflow can have one stress test

  Stress:
      Count: 40
      Threads: 5
      Delay: 0

The stress test has 3 fields, Count, Threads and Delay.

• Count defines the number of times the complete workflow will be executed
• Threads defines the number of threads by which Count will be divided (Count: 10 and Threads: 2 means each thread will execute the workflow 5 times)
• Delay defines the delay in milliseconds between every full execution, which can be usefull to prevent errors of too many executions

Test

You can define one or more tests within each workflow

  Tests:

    - TestID: createPet
      Server: "https://petstore3.swagger.io/api/v3"
      Path: "/pet"
      Method: Post
      Headers:
        - Content-Type:application/json
        - Accept:application/json
      Body: "$ref/dictionary/petExample"
      Retain:
        - petId#$.id
      Verifications:
        - Code: 200
          Schema: "$ref/definitions/Pet"

    - TestID: readPet
      Server: "https://petstore3.swagger.io/api/v3"
      Path: "/pet/{petId}"
      Method: Get
      Headers:
        - Accept:application/xml
      Verifications:
        - Code: 200
          Contains: id
          Count: doggie#1
          Schema: "$ref/definitions/Pet"
          Match: /Pet/name#doggie
          Custom: ["CustomVerification.dll"]

    - TestID: updatePet
      Server: "https://petstore3.swagger.io/api/v3"
      Path: "/pet/{petId}"
      Query:
        - name=doggie
        - status=sold
      Method: Post
      Headers:
        - Accept:application/xml
      Verifications:
        - Code: 200
          Schema: "$ref/dictionary/petSchemaXml"

    - TestID: deletePet
      Server: "https://petstore3.swagger.io/api/v3"
      Path: "/pet/{petId}"
      Method: Delete
      Headers:
        - Accept:application/json
      Verifications:
        - Code: 200     

The test is identified with its id, which MUST be unique across all tests of every workflow

   - TestID: createPet

It is then followed by 3 mandatory fields for a successful HTTP request, the server, the path and the method

    Server: "https://petstore3.swagger.io/api/v3"
    Path: "/pet"
    Method: Post

(Currently the only supported methods are Get, Post, Put and Delete)

You can also define headers following a key/value structure

  Headers:
    - Content-Type:application/json
    - Accept:application/json

Which is similar for Query string parameters

  Query:
    - name=doggie
    - status=sold

The body data can be defined directly on the TSL file, however they can sometimes be extremely large which would hurt the clarity and readability of the file. You can however create an auxiliary text file (dictionary file) which contains the actual body and a unique identifier within the dictionary, leaving only the reference to said identifier on the TSL file

  Body: "$ref/dictionary/petExample"

Dictionary File

The dictionary file is a text file containing all the body data and schemas in order to improve clarity on the actual TSL file

  dictionaryID:petExample
  {
    "id": 10,
    "name": "doggie",
    "status": "available"
  }

Every entry on the file requires the dictionaryID which must be unique across all entries, followed by the actual data, followed by an empty line to separate them

Verifications

Each test can have multiple verifications, only the Code verification is mandatory

  Verifications:
    - Code: 200
      Contains: id
      Count: doggie#1
      Schema: "$ref/definitions/Pet"
      Match: /Pet/name#doggie
      Custom: ["CustomVerification.dll"]

Currently 6 different verifications are supported:

Requirement	Name	Input Type	Description
Mandatory	Code	Integer	Response code matches the given code
Optional	Contains	String	Response body contains the given string
Optional	Count	String#Integer	Response body contains given string # times
Optional	Schema	String	Response body matches the given schema*
Optional	Match	StringPath#String	Response body matches the given value present in the StringPath
Optional	Custom	[String]	Runs Custom verifications given by the user

*The schema verification can be supplied directly, or through reference to the dictionary file or to any schema present in the supplied OAS

Retain

In some requests, the input is based on the output of a previous request, usually in simple workflows, like create read

    Retain:
      - petId#$.id

The keyword Retain allows the user to retain some information from the response body of the request to then be used in other tests of the same workflow.
For instance the value present at the json path $.id will be retained with the identifier petId which can then be used in following tests

    Path: "/pet/{petId}"

(back to top)

Editor

After the initial project that developed and built most of the RapiTest app, a new development cycle was started with the goal of improving upon the application, specifically by implementing a new graphical editor. This new editor would have the goal of providing the RapiTest app with an improved interface for test creation, allowing users to leverage the capabilities of the TSL language.

Most of the work done during this cycle can be found under /RAPITest/ClientApp/src/pages/editor, although changes were made in other places as well, including in the server.

For an overview and quick guide on the editor, please check the document under /others/editor_quick_guide.odt (if it's not available yet, it will soon be).

Contributing

Contributions are what make the open source community such an amazing place to learn, inspire, and create. Any contributions you make are greatly appreciated.

If you have a suggestion that would make this better, please fork the repo and create a pull request. You can also simply open an issue with the tag "enhancement". Don't forget to give the project a star! Thanks again!

1. Fork the Project
2. Create your Feature Branch (git checkout -b feature/AmazingFeature)
3. Commit your Changes (git commit -m 'Add some AmazingFeature')
4. Push to the Branch (git push origin feature/AmazingFeature)
5. Open a Pull Request

(back to top)

Contact and Acknowledgements

This project was developed as part of a partnership between the Lisbon City council (CML) and Lisbon School of Engineering (ISEL).

Duarte Felício - A42197@alunos.isel.pt

Project Link: https://github.com/DuarteFelicio/RAPITest

This project was later continued and improved with the addition of a graphical node-based editor (see Editor ).

Alexandre Rocha - A44789@alunos.isel.pt

Project Link: https://github.com/Alexandre-Rocha/RAPITest

(back to top)