random tools for dealing with bitbucket.org repos. Created as https://community.atlassian.com/t5/Bitbucket-articles/What-to-do-with-your-Mercurial-repos-when-Bitbucket-sunsets is a very long thread and difficult to search.
This repo contains dump_repos.py which will dump most things to local file system for offline backup. It does NOT import or convert it.
If you have lost content, check out https://bitbucket-archive.softwareheritage.org/ they do not appear to have a complete back up bit there is a good chance that your code is there.
Getting my notes off my machine and in a shareable state.
TL;DR check out:
- https://bitbucket.org/edrandall/bitbucket-hg-to-git/
- https://magnushoff.com/blog/kick-the-bitbucket/
- https://docs.gitlab.com/ee/user/project/import/bitbucket.html
- or https://github.com/clach04/bitbucket_tools/blob/master/dump_repos.py which will export everything (project meta data, logo/avatar, issue text, download metadata and downloads) but not import/push it anywhere
Updated version that works as of 2019, with REST v2.0 API https://developer.atlassian.com/bitbucket/api/2/reference/resource/
https://bitbucket.org/arrizza/bitbucket-cli/src/master/
Also includes a tool convert_to_git (recommend bitbucket-hg-to-git instead)
https://bitbucket.org/edrandall/bitbucket-hg-to-git
creates new BitBucket git repos from mercurial ones. Handles some meta data/tags - handles meta data link description and private repos.
Does NOT handle private repositories (loops forever every 30 seconds with login failure - https://github.com/philipstarkey/bitbucket-hg-exporter/blob/master/bitbucket_hg_exporter/__main__.py#L623 )
UNTESTED/partially tested - claims to handle:
pull request history, issue trackers, wikis, forks, attachments and commit comments
https://github.com/philipstarkey/bitbucket-hg-exporter
Large histories take a long time, one (Jython 2.x) project I had was running for a day and was still chugging away. Handled import of a small project fairly well.
Appeared to handle code (branches) and project meta data.
- Wiki not handled at all.
- Downloads untested so far.
- Avatar/icons untested so far (failed large project did appear to persist to local disk)
- Issues handled BUT prompts before import (can only import once, unlike specific issue migration tools below) so this requires interaction.
Update 7th April 2020: v0.7.0 released - claims to decrease the download time. Can also use a local hg->git conversion tool (such as hg-export-tool instead of the GitHub source importer which has better handling of UTF8 charcters, author attributions and mercurial branches with multiple heads. I have not yet tested these improvements.
UNTESTED - appears to handle private? Partially handles wiki
https://magnushoff.com/blog/kick-the-bitbucket/
UNTESTED - appears to be code and wiki only, appears to handle OAuth. Based on readmeit appears that https://github.com/clach04/bitbucket_tools/blob/master/dump_repos.py is more complete if login/password is a viable option for export.
https://github.com/samkuehn/bitbucket-backup
UNTESTED - shell scripts to dump/export to local file system. May handle Pull Requests - looks like its missing most metadata. And the generated commands do not only retrieve the owner's repositories, they clone all of the repositories referred to as the source for pull requests. Those repositories too will disappear after the purge. WARNING does not appear to handle downloads based on readme https://bitbucket.org/Rarified/bbmigrate/src/master/
UNTESTED Python script, appears to use GitHub import service so same limitations as GitHub but it it does automate all repo imports so fast for simple code only import
https://github.com/brunogfranca/bitbucket2github
Needs work
Appears to handle code repo and issues only; issue migration results in corrupt issue cross references, creates issue tracker (todo) whether original project had one or not. Does NOT handle wiki (sourcehut does not offer hg wiki), does not handle download attachments.
https://hg.sr.ht/~sircmpwn/invertbucket
UNTESTED
https://git.sr.ht/~edik/rerefcommitjson
rerefcommitjson converts all commit hashes in a Bitbucket issue JSON dump from hg to git. See the example section for details and usage warnings.
Philip Starkey's bitbucket-hg-exporter can also translate hashes, but only if they are marked up. rerefcommitjson, however, is able to translate markup-less hashes too.
UNTESTED
https://github.com/fkirc/bitbucket-issues-to-github
UNTESTED - doesn't handle attachments in issues
https://github.com/jeffwidman/bitbucket-issue-migration
Old Fork https://github.com/sphinx-doc/bitbucket_issue_migration
UNTESTED
Perl script - not sure how complete https://github.com/devzendo/hgignore-to-gitignore
UNTESTED
https://github.com/devzendo/bitbucket-to-github-pom-scm
UNTESTED
https://githgmirror.com/. It can keep a git and hg repo (or two hg or two git repos) in sync in one direction or bidirectionally. It's also open source and you can run it yourself too. Only deals with code, so no wiki or issues.
Heptapod is a friendly fork of GitLab that supports mercurial. They offer free hosting for Open Source projects within reason on foss.heptapod.net, and commercial hosting on heptapod.host (currently in free of charge beta).
For the announcement about FOSS hosting, see this blog post. For the announcement of commercial hosting, see this one
as of mid 2020 in free alpha testing. has both hg and git support.
invertbucket can be used to partially migrate. It migrates code 100%. Issue migration is flawed (wrong issue numbers in cross refs). Wiki is NOT migrated and as of July 2020 only supports git wikis. There is no support for download attachments.
In terms of features comparable feature set to BitBucket, in terms of migration tools very basic.
Public hg/git repos can be imported very easily. Unclear on how branches are imported (multiple heads can be an issue). Supports private repos if password is provided (application passwords can be used if 2FA is used), ONLY handles repos (does not handle wiki migration, downloads, project avatar/logo, issues, PRs).
Issues seen with utf8 characters in commit messages.
Lots of options, none complete. https://medium.com/collaborne-engineering/how-to-migrate-a-private-repository-from-bitbucket-to-github-6cddedd5d73 is a good overview on manually migrating code only.
GitHub's code/history import https://github.com/new/import worked well for me to import and convert a Mercurial code repo. It does not handle:
- project description - needs to manually copy/pasted
- project logo/avatar/image - needs to manually set (note you may not be able to get the original image that was used to upload, only a scaled-down copy unless you have the original stored somewhere)
- Issues / history / attachments
- PRs / history
- Downloads
- wiki
https://docs.gitlab.com/ee/user/project/import/bitbucket.html claims to support issues (unclear about attachments), PRs, and wiki. does NOT list downloads.
- https://magnushoff.com/blog/kick-the-bitbucket/ - bash/shell script for moving to GitHub
- https://bitbucket.org/edrandall/bitbucket-hg-to-git - BitBucket to BitBucket git.
needs to manually copy/pasted
curl https://api.bitbucket.org/2.0/repositories/clach04/jython/
WIP can this be scrapped?
avatar (two of them) in meta data is available (redirects, usually to AWS) but is small version cropped version and larger-uncropped (proabbly not original size)
curl https://api.bitbucket.org/2.0/repositories/clach04/jython/
needs to manually set (note you may not be able to get the original image that was used to upload, only a scaled-down copy unless you have the original stored somewhere)
WIP
- https://github.com/fkirc/bitbucket-issues-to-github
- https://github.com/jeffwidman/bitbucket-issue-migration
From https://www.reddit.com/r/mercurial/comments/elu3m7/migrating_from_bitbucket_to_github/
arganoid
I have now tried the second script. Overall I would say the first is quicker and easier, as with the second one you have to export issues via the Bitbucket interface, and also generate an API key from Github. The first script just uses your Bitbucket and GitHub passwords (although that could be a security risk if you don't trust the script).
I noticed the issues brought across by the first script retained their timestamps whereas the second one saves issues with the current time, albeit with a note in the text indicating the original date
curl https://api.bitbucket.org/2.0/repositories/clach04/jython/issues # no issues example
use V2 api to see if there issues at all
WIP
meta data and list:
curl https://api.bitbucket.org/2.0/repositories/clach04/dava33display/downloads # no downloads
curl https://api.bitbucket.org/2.0/repositories/clach04/jython/downloads
each download:
where name is in array from above
curl -S -L https://api.bitbucket.org/2.0/repositories/clach04/jython/downloads/jython251hacked_nopydoc.jar --output jython251hacked_nopydoc.jar
WIP anyway to create initial wiki page? No API, potentially could scrap/robot webpage
wiki declared in meta data with has_wiki attribute (see "project description").
Create an initial page and then force push the new git repo. The wiki content will need to be edited:
- Links seemed to work
- images links needed conversion from mediawiki to Markdown
- headers need converting to markdown
github's hg importer messes up the order of merge parents ... hg-fast-export gets it right:
https://github.com/frej/fast-export/
If you use bookmarks in mercurial, hg-fast-export can't deal with them though. I made a tool that wraps hg-fast-export, first converting bookmarks to named branches so it can deal with them: