Developer meeting – Gitlab

This week we talked about Gitlab, and our migration from using Sourcerepo for Git hosting to using our own self-hosted Gitlab on Amazon EC2.

We’re switching to Gitlab because it has a better UI, has a REST API for creating and managing projects, and has some nice features like protected branches and snippets.

Our migration process was to write some Scala code that used Selenium to scrape our existing list of projects from Sourcerepo, and then used the Gitlab API to create new projects in Gitlab. It also migrated users and transferred issues from our Trac issue management system into the Gitlab issue management.

We discussed using the Gitlab “merge request” process in place of our existing Git Flow approach to code review. We didn’t reach any firm conclusions, but we’re likely to discuss this again in a future developer meeting.

We discussed other uses for the API:

  • It potentially allows for better integration with our Jenkins continuous integration server. We would need to write code for this. There is existing Jenkins code that will scan a GitHub account, and create a Jenkins build for every project it finds containing a “Jenkinsfile” pipeline configuration – we could use this as a basis. This would mean that we didn’t need to configure Jenkins projects explicitly, just create a new repo based on our standard project template and the Jenkins project would be created automatically.
  • In addition to the normal Gitlab backup process, we can use the API to maintain a local copy of all the Git repositories on the server. The API allows us to enumerate all of the repositories and create directories for them, then we can use a script to check out the latest contents of each on a schedule.

Ansible and other orchestration tools

In our developer meeting, Richard B talked about orchestration tools, reporting back on his evaluation of Chef, Puppet, Ansible and Salt.

The full details are in the slides that he produced, attached below. The high-level summary is that he thinks that Ansible is most appropriate for our purposes. Ansible playbooks let us set up repeatable scripts for provisioning a server – so the server setup instructions that we currently write can be replaced by an executable and testable script. The main downsides for us are that Amazon Linux doesn’t work well with Ansible, and that we will need to create modules from scratch for some of the software that we use such as MarkLogic.

Richard has set up one of our current projects to work with Ansible, and we will experiment with using Ansible for new projects.

Slides for the Ansible dev talk (PDF)