- add repo id to secret scanning finding _key #307 (@RonaldEAM)
- Ronald Arias (@RonaldEAM)
- remove secret property from secret scanning findings #306 (@RonaldEAM)
- Ronald Arias (@RonaldEAM)
- include secretScanningAlerts step back again #305 (@RonaldEAM)
- Ronald Arias (@RonaldEAM)
- Remove actions handler #302 (@Nick-NCSU)
- Fix tests dates #301 (@Nick-NCSU)
- Nick Thompson (@Nick-NCSU)
- Update Dockerfile #300 (@zemberdotnet)
- Matthew Zember (@zemberdotnet)
- skip empty pull requests response object #299 (@RonaldEAM)
- Ronald Arias (@RonaldEAM)
- change pull requests query approach to consume > 1000 PRs #298 (@RonaldEAM)
- Ronald Arias (@RonaldEAM)
- fix scopes' info log #297 (@RonaldEAM)
- Ronald Arias (@RonaldEAM)
- add scopes log; fix issues totalCount field on repos #296 (@RonaldEAM)
- Ronald Arias (@RonaldEAM)
- add config options to change ingestion window for PRs and issues #295 (@RonaldEAM)
- Ronald Arias (@RonaldEAM)
- Add authentication type for enterprise PATs #294 (@RonaldEAM)
- Ronald Arias (@RonaldEAM)
- log metrics to debug external identity issues #293 (@RonaldEAM)
- Ronald Arias (@RonaldEAM)
- fix pagination for external identities query #292 (@RonaldEAM)
- Ronald Arias (@RonaldEAM)
- INT-10119 - split issues query for labels and assignees #291 (@RonaldEAM)
- Apply remove-codeql with multi-gitter [ci skip] (@electricgull)
- Cameron Griffin (@electricgull)
- Ronald Arias (@RonaldEAM)
- INT-10071 - handle gateway timeout for repos step #290 (@RonaldEAM)
- Fix x-cortex-service-groups where tier-4 was set incorrectly (@jablonnc)
- Noah Jablonski (@jablonnc)
- Ronald Arias (@RonaldEAM)
- Release graphql improvements #289 (@RonaldEAM)
- Setup beta branch #285 (@RonaldEAM)
- Ronald Arias (@RonaldEAM)
All notable changes to this project will be documented in this file.
The format is based on Keep a Changelog, and this project adheres to Semantic Versioning.
- Organization external identifiers are now used when available to resolve member email addresses.
- Added validate invocation to the invocation config.
- Refactored validateInvocation to support SDK changes.
- Introduced an additional configuration field PULL_REQUEST_MAX_SEARCH_LIMIT for limiting pull request searches.
- Removed
config.enableDependabotAlerts
option since Ingestion Source Configs replaces this feature. - Removed
ENABLE_DEPENDABOT_ALERTS
as an environment variable since Ingestion Source Configs replaces this feature.
- Add
sourceRefOid
andtargetRefOid
properties to the PullRequest entity.
- Improved labels/descriptions for ingestion sources.
- Upgraded to Node 18
- Upgraded to SDK packages 8.41.0
- Introduced Ingestion Sources to enable the Data Config feature
- Upgraded to SDK packages 8.40.0
- The following property has been added to
github_finding
:
Property | Type |
---|---|
dependencyScope |
string |
- The following properties have been added to
github_repo
:
Property | Type |
---|---|
visibility |
string |
-
BREAKING: The
fixReason
property has been removed fromgithub_finding
entities. ThefixReason
field returned by GitHub'svulnerabilityAlerts
GraphQL API has been deprecated and is set to be removed at the end of 03-2023. -
Revert changes released in v2.2.1. The
fixedOn
is not being removed from the GitHub API. Instead, thefixReason
property is being removed.
- BREAKING: The
fixedOn
property has been removed fromgithub_finding
entities. ThefixedAt
field returned by GitHub'svulnerabilityAlerts
GraphQL API has been deprecated and is set to be removed at the end of 03-2023.
- Code Scanning alert step will now self-disabled if the GHE version (<3.5.0) does not support Organization level code scanning alerts ingestion.
- Fixed type error seen in updated deployment
- Re-enabled Branch Protection Rule test
- Branch Protection Rule query support for GitHub Enterprise versions 3.6.0 and 3.5.0 contained a bug and was fixed.
- New data collection step was added: Code Scanning Alerts
The following entities were added:
Resources | Entity _type |
Entity _class |
---|---|---|
GitHub Code Scanning Alerts | github_code_scanning_finding |
Finding |
The following relationships were added:
Source Entity _type |
Relationship _class |
Target Entity _type |
---|---|---|
github_repo |
HAS | github_code_scanning_finding |
- @octokit/rest was updated to v19.0.3
- @jupiterone/integration-sdk-* were updated to v8.30.2
- Testing har and snap files now use JupiterOne-Sandbox GitHub organization
- Added Branch Protection Rule query support for GitHub Enterprise versions 3.6.0 and 3.5.0.
- Added
filesChangedCount
property togithub_pull_request
- Add a buffer (of one polling interval) to the start datetime of PR ingestion. This will allow any missed updates to be re-queried and included in the next ingestion round.
- Updated permissions for branch protection rules step to include
repo discussions:read-only
to allow private repo rules to be ingested.
- ingest github pages (new permission will be requested by github app)
-
Added relationship between two pull requests that share the same merge commit. This allows for discovery of pull request merges without approval given the following scenario:
PR1 - Branch A -> main with commit {A} PR2 - Branch B -> main with commits {A, B} PR3 - Branch C -> main with commits {A, B, C}
If PR3 is merged first, PR1 and PR2 will be marked as MERGED and potentially without approval in some circumstances. This new CONTAINS relationship will indicate that PR3 ->CONTAINS-> (PR1 | PR2)
Source Entity _type |
Relationship _class |
Target Entity _type |
---|---|---|
github_pullrequest |
CONTAINS | github_pullrequest |
github_pullrequest
entity propertyallCommitsApproved
will now only be set if commits are available on a given pull request. If a repo is private, permissions don't allow us to pull commit history.
- Fixed branch protection rules logic to only add relationship when the user is known.
- Added
bypassPullRequestAllowance
property to OVERRIDES relationship between (github_user|github_app|github_team) and branch protection rule.
- Changed step permissions for branch protection rules.
- Added
branchProtectionRules
step with the following entity and relationships:
Resources | Entity _type |
Entity _class |
---|---|---|
GitHub Branch Protection Rules | github_branch_protection_rule |
Rule |
Source Entity _type |
Relationship _class |
Target Entity _type |
---|---|---|
github_team |
OVERRIDES | github_branch_protection_rule |
github_repo |
HAS | github_branch_protection_rule |
github_user |
OVERRIDES | github_branch_protection_rule |
github_app |
OVERRIDES | github_branch_protection_rule |
- Property
databaseId
type was change fromstring
tonumber
- Added
pullRequestMaxResourcesPerRepo
to config to allow the max limit of pull requests to be adjusted. Default remains at 500 per repo.
- Added
pullRequestIngestStartDatetime
to config to allow pull request ingestion to start at a specified date. - Update the GitHub Finding property name. (Summary -> description).
- Added disabled step reasons to getStepStartStates
- Updated integration-sdk-* versions to 8.18.0
- Duplicate PR keys now also prevent duplicate relationships from being created.
- Duplicate PR keys are now being caught, logged, and skipped. Duplication should only be happening in rare instances such as when a new PR is entered during ingestion and pagination is shifted.
- Eliminated the unused config value
githubInstallationId
. - Update action return types.
validateInvocation
now uses /meta endpoint. Establishes the version of GHE Server, as applicable.- Based on the GHE Server version, the Vulnerability Alert queries are modified.
- Vulnerability Alert converter was updated to handle the less verbose response from older versions.
- Finding relationships to CWE and CVE are now mapped relationships and do not create direct relationships.
- Added ability to enable/disable ingestion of Dependabot vulnerability findings. Includes finding filter options on status and severity.
- Modified Vulnerability to CVE/CWE mapped relationship to correct properties.
- Fixed buildVulnAlertRecommendation() to handle responses without
firstPatchedVersion
. - Changed Vulnerability Alert to CVE/CWE from a direct relationship to a mapped relationship.
- Added Vulnerability Alerts (aka Dependabot Alerts) ingestion step. In order to enable, this requires the Dependabot Alert read-only permission.
- Added action handler to process real-time entity ingestion. (v1)
integration-question-github-how-often-are-github-secrets-rotating
90 days managed question to have- 90 days
- Added
code-ql
andquestions
workflow
- Promoted all 1.12.0-beta* fixes.
- Fixed refreshToken in GraphQL client to use correct baseUrl.
- Updated error handling to match new @octokit/graphql.js error structure.
- Updated README for GraphQL client.
- Errors now return partial data responses when possible.
- Moved to @octokit/graphql.js as our client to improve error handling.
- Updated @octokit/* packages and removed redundant packages from
devDependencies
.
- Added improve retry logic during timeouts.
- Added enhanced debug logging. Set environment variable
LOG_LEVEL=trace
to see.
- Updated integration-sdk-* packages to version 8.12.1
- Fixed issue when a single PR was queried for multiple inner resources.
- Rate limit is no longer a required value in GraphQL responses. This supports GHE configuration where rate limiting is turned off.
- Marked GitHub Issues ingestion step as
partial
allowing existing issues to remain in the graph.
- Identified known GraphQL errors (see README.md)
- Introduced clear error handling
- Upgraded graphql.js package
- Continued GraphQL pagination refactor for all queries.
- Renamed
createApiClient()
togetOrCreateApiClient()
and moved to a singleton pattern to prevent unnecessary authentication requests. createQueryExecutor
now accepts arateLimit
object, as a starting point, and the GraphQLClient keeps track of therateLimit
after each step.- Additional fixes
- Introduces new GraphQL pagination pattern for Pull Request entities. This beta release is to prove the changes in various environments before completing the refactor.
- Updated support for GitHub Enterprise Server within a managed environment. Allowing users to specify config for their instance.
- Added support for a self-hosted GitHub Enterprise Server. Supply the URL to the instance via GITHUB_API_BASE_URL environment variable.
- Added property
organizationId
togithub_user
Entity, to make it more convenient to track users from multiple organizations.
- Optimized queries in
fetch-team-members
andfetch-team-repos
to fetch the ideal number of teams necessary to reach desired data.
github_user
propertyactive
added, a boolean which is always true in the case of GitHub Users, to conform with new SDK standards for User entities
- Added logging of token scopes for troubleshooting purposes
- Updated documentation with notes on GitHub App scope details
- Truncate potentially large
body
andbodyText
property values fromgithub_issue
- Remove
body
andbodyText
property values fromgithub_issue
raw data
- Additional logging to pull request iteration
- If multiple teams are returned by the GraphQL API for a single slug, the integration filters for the team it is working with at that moment instead of throwing duplicate relationship errors
- Issues can now discover outside collaborators for CREATED and ASSIGNED relationships
- Updated tests to match the single-team queries for team-members and team-repos that are being used by the ingestion code
- The
handleTimeout
method when handling graphQL calls as it caused a promise to never get resolved. Timeout errors are now handled in a way that resolves all promises properly.
- Order to team-repos and team-members graphQL queries to ensure the correct team is selected.
- Moved GraphQL sleep for rate limit outside of retry functions, and restored GraphQL retry timeout to 3 minutes so catch errors where Node hangs without a thrown error.
github_team_has_user
,github_user_manages_team
, andgithub_repo_allows_team
relationships will not be made if a relationship with the same key has already been ingested in the same integration run.
- Children resources in graphQL queries will now be cursed through instead of only getting the first 100 resources.
- Children resource cursors are now being properly handled which prevents ingesting the same child resource muliple times.
- Changed GraphQL timeout 1 hour so it doesn't interfere with GraphQL sleep for rate limit
- A GraphQL timeout of two minutes to prevent infinite hanging of integration runs.
- Made
fetch-collaborators
,fetch-team-repos
andfetch-team-members
ingest entities in small batches to eliminate cursor confusion and timeouts on large datasets
- Fixed broken cursor handling for nested objects, which made some larger GraphQL queries never terminate.
- Added logs for pagination details
- Separated
fetch-teams
step into three steps so that any crashes have less impact. The 3 steps arefetch-teams
,fetch-team-repos
andfetch-team-members
. Stepfetch-teams
just gets the team entities and the relationshipgithub_account
HASgithub_team
. Stepfetch-team-repos
ingests thegithub_repo
ALLOWSgithub_team
relationship. Stepfetch-team-members
ingests the relationshipsgithub_team
HASgithub_user
andgithub_user
MANAGESgithub_team
. - Improved logs
- Fixed crash on
fetch-env-secrets
when the GitHub App was not authorized for environmental secrets on a subset of repos (403 error) - Fixed unnecessary dependency where
fetch-collaborators
was waiting onfetch-teams
, which in turn delayedfetch-prs
. - Fixed some instances where async functions were not being properly awaited.
- Added check to avoid invoking and logging sleep function for negative time.
- Fixed duplicate key error for when Environments of the same name exist on multiple repositories, and they have Secrets with the same name.
- Fixed crash on
fetch-repo-secrets
when the GitHub App was not authorized for all repos (403 error) - Fixed token expiration after sleep due to rate-limiting
- Properly handle cases when GitHub does not return expected array properties from the GraphQL API
-
Increased tolerance for delays due to GraphQL API errors, to upto 1 hour.
-
Integration now respects
resetAt
property of rate-limiting messages to better predict when the API will permit further usage -
Integration begins to slow down when rate limits are 90% used, to reduce impact and contention with any other automation running against this account's API limits
- Additional GraphQL failure debug logging
- Improved error messaging when a non-rate limit error occurs
- An error in the error handler for certain GraphQL API failures
- When GitHub returns
null
forhasTwoFactorEnabled
due to permissions limitations,github_user
propertymfaEnabled
will now be set toundefined
instead offalse
- When a step takes more than an hour to complete, the GraphQL client can now request a new API token on the fly without losing track of where it was
- Better error handling, including support for when GitHub rate-limiting errors are provided with non-error (200) codes.
- Property
id
togithub_member
entities
- Always throw an
IntegrationError
when repository environment secrets can not be ingested. - Separated out
fetch-env-secrets
step from thefetch-environments
step
- Changed incorrectly spelled
github_app
propertyrespositorySelected
torepositorySelected
- Properties
forkingAllowed
andforkCount
to RepoEntity
- Only retry graphQL errors 5 times instead of 10.
- Only query for the limit of
github_pull_request
s andgithub_issue
s instead of the limit +1. - Request fewer repositiories in a single calls in the
fetch-teams
step. - Request fewer repositories and collaborators in a single call in the
fetch-collaborators
. - Format graphQL errors even better.
- Do not throw an
IntegrationError
when the integration does not have access to environment secrets. - Add a JobLog item when environment secrets could not be ingested due to a
403
error.
- Log graphQL errors properly.
- Increase
github_pull_request
limit from 100 to 500.
- Crash on fetch-environments for private repos in accounts that are not Enterprise level
- Added support for ingesting labels on PullRequests
- Added support for ingesting the following new entity:
Resources | Entity _type |
Entity _class |
---|---|---|
GitHub Issue | github_issue |
Issue |
- Added support for ingesting the following new relationships:
Source Entity _type |
Relationship _class |
Target Entity _type |
---|---|---|
github_repo |
HAS | github_issue |
github_user |
CREATED | github_issue |
github_user |
ASSIGNED | github_issue |
- Migrated fetch-collaborators to GraphQL instead of REST for improved performance
- Removed redundant property API calls in several GraphQL queries
- All new boolean properties added in v1.5.0 that included
is
to no longer includeis
. Ex:isLocked
becamelocked
.
- Added support for ingesting the following new entities:
Resources | Entity _type |
Entity _class |
---|---|---|
GitHub Environment | github_environment |
Configuration |
GitHub Env Secret | github_env_secret |
Secret |
- Added support for ingesting the following new relationships:
Source Entity _type |
Relationship _class |
Target Entity _type |
---|---|---|
github_repo |
HAS | github_environment |
github_environment |
HAS | github_env_secret |
github_env_secret |
OVERRIDES | github_org_secret |
github_env_secret |
OVERRIDES | github_repo_secret |
github_repo |
USES | github_env_secret |
- Extra 9 properties to
github_account
(createdOn
,updatedOn
,description
,email
,node
,databaseId
,verified
,location
,websiteUrl
,webLink
) - Extra 6 properites to
github_team
(createdOn
,updatedOn
,databaseId
,description
,node
,privacy
) - Extra 8 properties to
github_user
(company
,createdOn
,updatedOn
,databaseId
,node
,employee
,location
,websiteUrl
,email
) - Extra 19 properties to
github_repo
(autoMergeAllowed
,databaseId
,deleteBranchOnMerge
,description
,homepageUrl
,node
,disabled
,empty
,fork
,inOrganization
,locked
,mirror
,securityPolicyEnabled
,template
,userConfigurationRepository
,lockReason
,mergeCommitAllowed
,pushedOn
,rebaseMergeAllowed
) - Extra 5 properties to
github_pullrequest
(databaseId
,node
,commitsCount
,approvalsCount
,approvalLastAt
) - Pull requests Opened, Reviewed, or Approved by a user who is not part of the current organization or collaborator list now have a mapped relationship to a GitHub user with the login recorded in the PR properties.
createdOn
andupdatedOn
properties forgithub_org_secret
,github_repo_secret
, andgithub_app
are now time-since-epoch integers instead of strings, matching other entities.- Steps that do not have enough token scope permission are now disabled instead of throwing errors.
- Prevent error for when the head repository could not be determined for a pull request.
- To query 25 pull requests at a time instead of 50 to prevent Github errors.
- Retry plugin to Octokit, which automatically retries upto 3 times for server 4xx/5xx responses except 400, 401, 403 and 404.
- Issue where collaborators step could fail when one repo has special permissions settings that prevent access to collaborators.
- Do not throw on 404 errors when fetching pull requests.
- Omit
members
andrepos
properties from the raw data ofgithub_team
entities.
- Properly log graphQL errors.
- Error logging to "Fetch Pull Requests" step.
- Issue where changing data during the integration run could cause duplicate key errors and failure of some steps.
- Issue where fallback to the REST API for certain accounts could cause Repo ALLOWS Team relationships to only appear for one team.
github_pull_request
ingestion to use octokit v4 graphQL instead of the v3 rest api.- Commit analysis to be done on every
github_pull_request
regardless of if theanalyzeCommitApproval
config variable is set or not.
analyzeCommitApproval
config variable.
- Retrying "Secondary Rate Limit" errors on graphQL queries.
- Better logging of graphQL queries.
- Added support for ingesting the following new entities:
Resources | Entity _type |
Entity _class |
---|---|---|
GitHub Org Secret | github_org_secret |
Secret |
GitHub Repo Secret | github_repo_secret |
Secret |
- Added support for ingesting the following new relationships:
Source Entity _type |
Relationship _class |
Target Entity _type |
---|---|---|
github_account |
HAS | github_org_secret |
github_repo |
HAS | github_repo_secret |
github_repo_secret |
OVERRIDES | github_org_secret |
github_repo |
USES | github_org_secret |
github_repo |
USES | github_repo_secret |
- Remove
suspendedBy
property fromgithub_app
entity. ThesuspendedBy
property is typeobject
, which is not a supported entity property value type.
-
Properties added to graph objects:
| Entity / Relationship | Property | Notes | | ------------------------- | ----------------------------- | ------------------- | ------- | ---------- | -------- | --- | |
github_repo_allows_team
|adminPermission: boolean
| | |github_repo_allows_team
|maintainPermission: boolean
| | |github_repo_allows_team
|pushPermission: boolean
| | |github_repo_allows_team
|triagePermission: boolean
| | |github_repo_allows_team
|pullPermission: boolean
| | |github_repo_allows_user
|role: 'READ' | 'TRIAGE' | 'WRITE' | 'MAINTAIN' | 'ADMIN'
| | |github_user
|webLink: string
| GitHub user profile | -
Properties changed on graph objects:
Entity / Relationship Old New Notes github_repo_allows_team
permissions: string
role: string
Match GitHub UI
- Role property for outside collaborators is now 'OUTSIDE'
-
Added support for ingesting the following new entities:
Entity github_app
github_user
(role: 'OUTSIDE'
) -
Added support for ingesting the following new relationships:
Source class Target github_account
INSTALLED github_app
github_repo
ALLOWS github_user
- Changed relationship
github_team_allows_repo
togithub_repo_allows_team
, and addedpermissions
property ('READ' | 'TRIAGE' | 'WRITE' | 'MAINTAIN' | 'ADMIN')
- Removed incorrect relationship listing in documentation.
- New properties on
github_pull_request
mergedOn
mergeCommitHash
head
andbase
properties fromgithub_pull_request
entities' rawData.
- TypeError in
getCommitsToDestination
when a commit does not exist.
- new optional config variable
useRestForTeamRepos
that can is sometimes needed to get around a GitHub error when fetching team repos via GraphQL.
- octokit packages to be dependencies instead of peer dependencies.
- Integration now uses the
@jupiterone/integration-sdk-core
- JupiterOne/integrations#5
Use
name || login
fordisplayName
ofAccount
andUser
entities.
- Duplicate key bug in
github_user
APPROVEDgithub_pullrequest
relationships.