Pull requests are always welcome, and the MongoDB engineering team appreciates any help the community can give to make the MongoDB tools better.
For any particular improvement you want to make, you can begin a discussion on the MongoDB Developers Forum. This is the best place to discuss your proposed improvement (and its implementation) with the core development team.
If you're interested in contributing, we have a list of some suggested tickets that are easy enough to get started on here
- Create a MongoDB JIRA account.
- Create a Github account.
- Fork the repository on Github at https://github.com/mongodb/mongo-tools.
- Sign the MongoDB Contributor Agreement. This will allow us to review and accept contributions.
- For more details see http://www.mongodb.org/about/contributors/.
- Submit a pull request against the project for review.
- File a JIRA ticket in the TOOLS project.
- All commit messages to the MongoDB Tools repository must be prefaced with the relevant JIRA ticket number e.g. "TOOLS-XXX add support for xyz".
In filing JIRA tickets for bugs, please clearly describe the issue you are resolving, including the platforms on which the issue is present and clear steps to reproduce.
For improvements or feature requests, be sure to explain the goal or use case, and the approach your solution will take.
There's some "paperwork" that needs to be done for all dependency changes. To simplify this there are several command you can run:
# Adds the latest version.
go run build.go addDep -pkg=github.com/some/package
# Adds the specified version.
go run build.go addDep -pkg=github.com/some/package@v1.2.3
# Updates to the latest version.
go run build.go updateDep -pkg=github.com/some/package
# Updates to the specified version.
go run build.go updateDep -pkg=github.com/some/package@v1.2.3
# Updates all dependencies to their latest versions.
go run build.go updateAllDeps
Note that to run this command you will need to have Podman installed.
This will update our go.{mod,sum} files, vendor the dependency, update the SBOM Lite file
(cyclonedx.sbom.json), and update the THIRD-PARTY-NOTICES file.
Note that you cannot just use go get to add or update dependencies, because it doesn't update
all of these other files that need to be updated when dependencies change.
You will need a MongoDB server listening on localhost:33333 to run the integration tests locally.
You can use the mlaunch tool to make this
simple:
$> mlaunch init --replicaset --port 33333
To run unit and integration tests:
go test -v ./...
If TOOLS_TESTING_UNIT is set to a true value in the shell environment, unit tests will run.
If TOOLS_TESTING_INTEGRATION is set to a true value in the shell environment, integration tests
will run.
Integration tests require a mongod (running on port 33333) while unit tests do not.
Example of how to run a specific integration test:
TOOLS_TESTING_INTEGRATION=true go test -v ./... -run TestImportDocuments
To run the quality assurance tests, you need to have the latest stable version of the rebuilt tools,
mongod, mongos, and mongo in your current working directory.
cd test/qa-tests
python buildscripts/smoke.py bson export files import oplog restore stat top
Some tests require older binaries that are named accordingly (e.g. mongod-2.4, mongod-2.6,
etc). You can use
setup_multiversion_mongodb.py to
download those binaries
In the past, we used
github.com/smartystreets/goconvey
as a test harness. However, we are moving to using to
github.com/stretchr/testify instead. If you are
working with existing tests, it's fine to keep using convey, but never mix convey and testify in a
single top-level TestX func. If you like, you can also rewrite the test func you're working on
to use testify. All new test funcs should use testify.
Testify has two primary packages, assert and require. They provide exactly the same functions.
The only difference is that when a test uses require a failure aborts the execution of that test
function.
In general, we prefer to use require over assert, except in cases where we are sure that a test
failure does not make the following tests invalid. If you're not sure which to use, use require.
When using require or assert, you can either use it via functions or by creating an Assertions
struct. The only difference is that the struct holds onto the *testing.T struct so you don't have
to pass it to ever assertion call you make:
func TestSomething(t *testing.T) {
require = require.New(t)
val := callSomeFunc()
// We don't pass `t` here:
require.Equal(42, val, "callSomeFunc returns 42")
}versus:
func TestSomething(t *testing.T) {
val := callSomeFunc()
// We do pass `t` here:
require.Equal(t, 42, val, "callSomeFunc returns 42")
}We use the first style exclusively. The call to require.New should always be the first line of
code in any test function, whether that's a top-level TestX func or a helper function.
Use the require.NoError method to check errors:
err := doSomethingFallible()
require.NoError(err, "doSomethingFallible did not return an error")All test assertions should include a description as their final argument. The description should describe what we expected to happen, not the failure. Here's an example:
require := require.New(t)
val, err := callSomeFunc()
require.NoError(err, "callSomeFunc does not return an error")
require.Equal(42, val, "callSomeFunc returns 42")For example, always use Equal, not Equalf. In practice, the "f" variants work in the same
way as their counterparts, so we will just pick the "f"-less version for consistency.
Use the t.Run() method to group tests into subtests. In particular, use this to avoid having very
long TestX funcs. For example:
func TestX(t *testing.T) {
doSomeSetup(t)
t.Run("variation 1", testVariation1)
t.Run("variation 2", testVariation2)
t.Run("variation 3", testVariation3)
}
func testVariation1(t *testing.T) {
require := require.New(t)
// Check some assertions
}
func TestY(t *testing.T) {
db := openDB(t)
t.Run("variation 1", func (*testing.T) { testVariationWithDb1(t, db) })
t.Run("variation 2", func (*testing.T) { testVariationWithDb2(t, db) })
t.Run("variation 3", func (*testing.T) { testVariationWithDb3(t, db) })
}
func testVariationWithDb1(t *testing.T, db *mongo.Database) {
}Many tests need to write data to disk. Whenever possible, use a temp directory for this. You can use
the testutil.MakeTempDir function to make a temp directory. If the TOOLS_TESTING_NO_CLEANUP env
var is set to a non-empty value then the cleanup func returned by testutil.MakeTempDir won't
delete the directory, which is useful when investigating test failures.
For an example of all of this, see the TestRestoreClusteredIndex func in
mongorestore/mongorestore_test.go.
We use the gosec tool for static analysis of this codebase. You can run this as part of our
linting checks by running the following command:
go run build.go sa:lint
If gosec reports a vulnerability, you have two options:
- Fix the issue so that
gosecstops reporting it. - Mark the issue as a false positive using a
#noseccomment.
If you mark it as a false positive, you must include a justification with the comment:
// #nosec G1234 -- the text here explains why we consider this a false positive
The justification text will end up in the SARIF report we generate as part of the release process.
We do not merge PRs which contain unaddressed high- or critical-severity vulnerabilities.
Note that this will only work for MongoDB employees with access to Snyk.
We use Snyk to check our dependencies for known security issues.
We run these checks in CI, so it's not strictly necessary to do so manually when you add or upgrade a dependency, but you can save some time by checking this locally first.
In order to do this, you will need to first
install the snyk CLI tool.
Then you can run the following commands:
snyk auth
snyk test --org="$org_id" --file=./go.mod
You can get the right organization ID from the Snyk web UI. Go to the "Settings" page and copy the
Organization ID from there. Make sure you are in the dev-prod organization!
If the dependency you just added has any known vulnerabilities this command will report them.
We do not merge PRs which contain unaddressed vulnerabilities in third-party dependencies. All
vulnerabilities found in the master branch must be resolved before a release.
We can address them in one of the following ways:
- Upgrade to a new version of the dependency which contains a fix.
- MongoDB employees only - Use the Silk UI to create a new
ticket in the
VULNproject and transition the ticket to "Rejected". You will need to select a "State" describing why this ticket was rejected, which can be one of "not affected", or "false positive". You will also supply a "Justification", which will end up in the Augmented SBOM.
As part of MongoDB's SSDLC initiative, we've made a number of changes to our development practices. Several of these have already been covered, notably our use of static analysis, the SBOM file, and third-party vulnerability management.
The practices that we are adopting, including producing an SBOM for all releases, doing various types of vulnerability scanning, signing releases, and documentation of all these things.
This is a file format that static analysis tools can output. Silk accepts reports in this format. See https://sarifweb.azurewebsites.net/ for more information.
A machine-readable file containing information about dependencies, including things like the package name, license, etc. This includes a recursive list of all third-party dependencies.
Silk is a third-party SaaS tool that MongoDB as a whole will use for managing all SSDLC-related info for our projects. Silk will be integrated with our Jira instance so that it can do things like create tickets for vulnerabilities in a project’s dependencies.
Snyk is a company that provides a variety of code scanning tools, including tools that do vulnerability checking for third-party dependencies and static code analysis for various languages.
A static analysis tool analyzes code without running it, looking for various issues. For this
particular project, we’re interested in tools that look for security vulnerabilities. For example,
the gosec tool attempts to detect when code creates files with insecure permission.
See the "Static Analysis with gosec" section above for more details on how to run this tool.
We actually have two SBOM files. The first, called the SBOM Lite file, lives permanently in
this repo's root as the cyclonedx.sbom.json file. This file contains a manifest of all of our
dependencies, including transitive dependencies. It includes information on those package's names,
versions, licenses, and other metadata. However, it does not contain information about
vulnerabilities. It must be updated whenever our dependencies change, and we enforce this via CI.
See the section on "Adding or Updating Dependencies" for more details on how to do this.
Vulnerability information lives in our Augmented SBOM files. These files live in the ssdlc
directory, and we create a new one for each release. These files act as a record of our
dependencies, including known vulnerabilities, for each release. The releases include the tag name
of the release, for example ssdlc/100.9.5.bom.json. This file must be created for each release,
and we enforce this via CI.
Generating this file can only be done by MongoDB employees, as it requires access to Silk. See our release documentation for more details.
All releases are recorded using a MongoDB-internal application called Papertrail. This records various pieces of information about releases, including the date and time of the release, who triggered the release (by pushing to Evergreen), and a checksum of each release file.
This is done automatically as part of the release.
All releases are signed automatically as part of the release process.