Travis CI is failing *not due* to PR issues

[tubaman]

Travis CI is failing even on a trivial PR: #870

To Reproduce

1. Create a trivial PR
2. See Travis CI fail

Expected behavior
Travis CI should only fail if there are issues with a PR

[ssbarnea]

That is true but someone needs to address this, and I am currently not using the library.

By addressing it, I do not mean to disable CI testing, I mean finding a way to make it work. Atlassian test instance is working.

Hint: either make a PR that fixes the issue or use the ❤️ button at top of page and trying to persuade someone to make it work.

[tubaman]

What's the Atlassian test instance?

[Addono]

Been looking a bit into this today, the problem is that "Encrypted environmental variables" are used, which are not exposed on PR branches, as you will see when you inspect the build logs for a PR:

Encrypted environment variables have been removed for security reasons.
See https://docs.travis-ci.com/user/pull-requests/#pull-requests-and-security-restrictions

And indeed, comparing the exposed variables in a PR:

Versus one by the maintainer:

Solution wise, Travis' documentation states this:

If your build relies on encrypted variables to run, for instance to run Selenium tests with BrowserStack or Sauce Labs, your build needs to take this into account. You won’t be able to run these tests for pull requests from external contributors.

To work around this, restrict these tests only to situations where the environment variables are available, or disable them for pull requests entirely, as shown in the following example:
...

Some solutions which come to mind:

• Spin up a JIRA test instance as part of the CI/CD, then use that for testing, and afterwards tear it down. I am currently doing something like that in a Jenkins pipeline, which works fine, although it does take a minute or so for JIRA server to boot. But given that the current Travis pipeline already takes a couple of minutes this shouldn't be too much of an issue.
• Make the CI user credentials public. Given that the Jira instance is solely used for testing is there not much damage one can do with these accounts (e.g. delete projects), although that would only be an option if it can be configured such that it has enough rights to pass the tests, but not enough to escalate its rights. For which I do not know if this is feasible.
• Let people configure Travis for their own forks and deem that sufficient. Upside: Easiest (just add documentation). Downside: The PR itself will not pass, just the pipeline on the commits in the fork.

[Addono]

What's the Atlassian test instance?

https://pycontribs.atlassian.net/

[ssbarnea]

See #896