Our motivation at Promyze was to have an overview of “what’s a typical public PR on GitHub ?”. This platform is used today by a large panel of organizations: single developers for their side projects, private companies, open-source organizations, universities, and so on. Our purpose is not to highlight data on PRs management in private companies, with teams working 40 days a week together, but instead to provide a humble overview of public data available on GitHub.
We used the GitHub REST API to browse public organizations. Then, within each of them, we searched for a pull request that matched the following criterion:
We ran queries on the REST API for few days, and finally gathered 10,193 Pull Requests from distinct organizations.
To evaluate the size of the PR, we computed the following metrics:
And here are the extracted values:
We can observe that at least 75% of the PRs have a reasonable size for a code review, as the state of the art recommends no more than 500 lines of code. But, spoiler: all of these PRs won’t be necessarily reviewed!
The average values are quite large because some outliers have extremely high values.
Developers write comments on a PR to start discussions on its content, ask for clarifications or suggest improvements. We wanted to highlight data regarding those discussions.
First of all, how many users participate in the PR discussion? (including the author of the PR and Bots as we will discuss below). Here are the key insights:
31,2 % of the PRs involve only the author, this shows that Pull Requests are not always collaborative workspaces.
Comments can also be automatically generated by Bots, which are applications that GitHub users can set up on their projects to perform quality or security analysis for instance.
We found that 748 PRs (7,33 % of our dataset) have at least one comment from a Bot. On this sample, only 50 PRs involved more than one Bot (45 have 3 bots and 5 have 3 Bots).
What about the number of comments in the PR?
Here are the main insights:
In most cases, discussions (when there exist) are quite short and go seldom beyond 10 comments.
We wanted to investigate more on the approval feature offered by GitHub.
We found that 34% of the PRs have at least one approval.
In this example, the user “jacogr” indicated an approval on the PR:
As you may know, multiple users can be required on a PR for approval. For PRs getting at least one approval, representing 3,498 PRs, here is the distribution of the number of approvals.
As you can see, 84,33 % of the approved PRs have a single developer that approved the PR. 0,8 % of the PRs have at least 3 contributors.
We compute the PR lifetime as the duration between its creation and the moment it’s been merged and closed.
We found that 2,861 PRs (28,60%) have a zero-minute lifetime, meaning they are either automatically merged or instantly manually merged by a user. 238 PRs of this sample involve more than 1 user, meaning that about 1/4 of the PRs are opened just in order to merge code from one branch to another, without any validation process.
For the rest of the PRs, having a lifetime > 0 minutes, here are the insights:
Even though a majority is merged within 6 hours, there are still 18,05 % of the PRs that wait more than a day to be merged.
Consider now the wait time per line of code metric, which reveals how long a line of code waits to be merged. The lower this value is, the most the latency is reduced. A value if 0 means a line of code is instantly merged. A PR of 10 new lines added and merged in 10 minutes will have higher throughput than a PR with 2 lines added in the same timeframe.
For readability reasons, we decided to skip “zero-minute PRs”.
Here is the following distribution in minutes:
Wait time per line of code
This is highly heterogeneous depending on the projects. The median value indicates that on average a line of code waits 10,5 minutes before being merged.
As we already said, projects on GitHub are heterogeneous: personal projects, students projects, open-source projects maintained by private companies, … We’d like now to narrow the focus on PRs from projects where we assume some structural rules have been set due to their size:
We’re aware these are arbitrary values, but this seems quite fair to us. This sample represents 1,994 PRs (19,5% of the original dataset), we’ll call it “Category A”.
Let’s compare the main metrics with another population of projects having both less than 10 contributors and less than 50 stars representing 6,560 PRs in our dataset (64,3 % of our dataset), we’ll call it “Category B”.
If we look at the other remaining metrics:
1,100 (55,2 %)
2,624 (40 %)
Not so surprisingly, we can observe that in the Category A projects, the comments and approvals are more frequent. Twice more PRs have approval in this population, and it’s even more than half of the sample. Also, we observe that there’s a small minority of zero-minute PRs in that sample, meaning projects tend to set up a validation process in their PR.
We observe that PRs’ content in category A is more atomic than in category B. The median number of lines added is 11 vs 28, and on average, the differences are much more significant in terms of lines added, removed, and files involved.
That’s all folks, feel free to comment if you have any suggestions or questions regarding this dataset.
Want to go deeper on Pull Requests? Find out how to save time during your code reviews on GitHub.
Promyze, the collaborative platform dedicated to improve developers’ skills through best practices sharing and definition.
Crafted from Bordeaux, France.