The debate between the Git Rebase and Git Merge workflows is a long-lasting and heated one, and at SingleStore we use the rebase workflow (my favorite method!)
In this blog post, I want to share some of our experiences with using GitLab Code Review and GitLab CI together to iterate on the Singlestore Helios platform with the rebase workflow. This repository has 10-40 commits per day, with engineers working across many different time zones on different components (frontend UIs, backend services, Kubernetes operators, scripts, infrastructure, etc).
We use GitLab's CI/CD system for linting, building, testing and automated deployments (and a few more odd things). Our main repository consists of 160 CI jobs, most of which are test jobs configured with matrix jobs that run in parallel. The pipeline execution time can range from 25 minutes to 60 minutes, depending on the code that was changed (and some other external factors).
One thing to note before we dive in is that we used Phabricator for several years before we switched to GitLab around 6 months ago.
Rebase Workflow and Code Review
GitLab can be configured to use the rebase workflow quite easily:
- Set the "Merge method" to "Fast-forward" merge in the Merge Requests configuration
- Set "Squash commits when merging" to "Require"
When you do this, GitLab will force you to have your side branch rebased against the main branch before allowing you to Merge. Merging a code change then becomes a two-step operation if your branch is not rebased against main. There's actually a ticket open in GitLab's backlog to make this operation one-click. (The rebase operation can also be performed with the git CLI).
Now, GitLab can be configured to block people from merging their changes to the main branch if the latest pipeline hasn't passed by enabling "Pipelines must succeed" ("Merge requests can't be merged if the latest pipeline did not succeed or is still running."). However, this presents a challenge to teams that have pipelines that are even moderate in duration. This is because by the time the pipeline has completed, the branch most likely needs to be rebased again.
Because of this, we've had to disable the "Pipelines must succeed" requirement entirely and trust the engineers to only merge if they have a somewhat recently rebased and successful pipeline. This is not ideal, and we'd like to be able to configure a different behavior. Here are some ideas for GitLab, from simpler to more complex:
- Having a check that forces the merge request to have had at least one successful pipeline.
- Having a check that forces the last successful pipeline in the merge request to be X hours/days old at most.
- Merge trains! (more below)
Merge Trains
GitLab has a feature called Merge Trains for flowing code changes to the main branch. Here's a great description from another blog post about these:
With merge trains, each merge request joins as the last item in that train with each merge request being processed in order. However, instead of queuing and waiting, each item takes the completed state of the previous (pending) merge ref (the merge result of the merge), adds its own changes, and starts the pipeline immediately in parallel under the assumption that everything is going to pass.
If all pipelines in the merge train are completed successfully, then no pipeline time is wasted on queuing or retrying. Pipelines invalidated through failures are immediately canceled, the MR causing the failure is removed, and the rest of the MRs in the train are requeued without the need for manual intervention.
Unfortunately, merge trains do not currently work with the Fast-forward merge flow. If they did, we could push all code changes into the train instead of to the main branch. These commits and their associated pipelines would automatically end up in the main branch if successful, or the MRs would be reopened if unsuccessful. This would be perfect for our team's way of working.
Rebase Workflow and Commit Messages
One of the main advantages of the rebase workflow is that the commit history in the main branch will be entirely linear. I find it important to make sure that the commit messages follow a certain template. We've been able to easily configure our MR summary template in the "Default description template for merge requests" configuration in GitLab. As the commits land in the main branch, the commit message is inherited from the merge request with a link to the merge request. This is perfect and makes studying the history of our repository really easy.
Stacked Merge Requests
Our team is using GitLab after several years using Phabricator (which has been EOL'd). One of the distinguishing features of Phabricator is how easy it is to manage stacked MRs ("diffs"). With GitLab, this is also possible, although it's a little bit more complicated: we can specify in the merge request that we would like to merge into another branch. This is usually enough to scope the changes in the merge request as we'd like. However, an issue can arise if the author of the merge request and the underlying branch are not the same person.
If the author of the underlying branch prefers not to rebase against the main branch as they work, but the merge request author prefers to rebase, the changes in the merge request can become polluted. If the merge request is set to merge into the underlying branch, we will see all the changes from main that have yet to be added to the underlying branch. If the merge request is set to merge into main, we will see the underlying branch's changes.
Phabricator's stacked diffs allowed us to open a diff comparing changes between two local branches. As a result, even if the underlying branch author preferred not to rebase until the end, it would not affect the changes in the diff.
Triggering Pipelines
Because we have different CI pipeline layouts for changes to different parts of the codebase (frontend, backend, etc.), we leverage the [changes](https://docs.gitlab.com/ee/ci/yaml/#ruleschanges) keyword in the CI configuration a lot. This allows us to design a different pipeline altogether depending on what changed in the commit that the CI is running against. Unfortunately, this only works in Merge Request Pipelines and not in pipelines that come as a result of pushing to a side branch in origin. So, we've had to disable the former type of pipelines and rely exclusively on Merge Request Pipelines.
It can however be cumbersome to have to create a MR just to run the CI for a side branch. So, we've come up with a clever hack that allows us to still run CI even if a MR wasn't triggered. By adding custom roles based on how the pipeline is created in a few places in our CI config, we can ensure that pipelines that were manually started via the UI/GitLab API for a given side branch also work as expected:
.frontend-rules: &frontend-rules
- if: '$CI_COMMIT_BRANCH == "main"'
- if: '$CI_PIPELINE_SOURCE == "web"'
- if: '$CI_PIPELINE_SOURCE == "api"'
- if: '$CI_PIPELINE_SOURCE == "merge_request_event"'
changes:
- 'frontend/**/*'
Of course, we need to have this for all the different pipeline shapes that we support.
Deployments
We use GitLab CI for automated deployments of our code (as well as manually rolling back changes if needed). This is done automatically for successful pipelines in the main branch, depending just slightly on the changes in the commit. The GitLab Environments feature is particularly useful to make it easy to check what commits are deployed to the various environments that we deploy to from CI. If there's an incident, it's critical that it's as fast as possible to identify what commits are running in the various environments.
Slack Bot for Deployment Notifications
We've also implemented a Slack bot that notifies a release change-log channel whenever deployments go through to production. This is very convenient, and since the implementation of this bot warrants its own blog post, I won't get into more details here. If people are interested in learning how we built this, please let me know!
Wrapping Up
A friend of mine who works at GitLab told me that migrating from Phabricator to GitLab could be painful for our team given the differences between the two platforms. After more than 6 months, we're comfortable with our way of working with GitLab Code Review and GitLab CI, but there are definitely various things that we'd like to see improved particularly when it comes to how the Git Rebase Workflow is supported (Epic 4911, Issue 349733, Issue 895, Issue 118825).
If you have any questions, feel free to reach out to me on Twitter.
