Understanding the Impact of Generative AI Tools on Code Quality and Developer Productivity in Open Source Software

Introduction

Generative AI (GenAI) tools such as GitHub Copilot and ChatGPT are rapidly transforming software development. According to the 2024 Stack Overflow Developer Survey [1], over 60% of professional developers now integrate some form of AI tools into their workflows. GitHub reported that developers code up to 55% faster, and that 85% of developers feel more confident in their code quality [2]. While these tools promise faster development cycles and improved productivity, growing concerns have emerged regarding the quality of AI-generated code.

Table 1. Trends in code changes from 2020 to 2024 (directly taken from [3]).

GitClear [3] analyzed a dataset of 211 million lines of code across more than 100 repositories between 2020 and 2024. GitClear reported a rise in code duplication, increased churn, and a decline in refactoring activity that coincided with the adoption of GenAI tools (as shown in Table 1). However, their dataset is limited in terms of size, timespan, and public availability.

This study aims to investigate the impact of GenAI tools on both code quality and developer productivity in open source software (OSS). We use the World of Code (WoC) dataset, which is a large-scale and frequently updated (most recent update in 2024) collection of version control data in the entire OSS ecosystem. The goal is to analyze large-scale trends before and after the widespread availability of GenAI tools. Specifically, we study the following research questions (RQs):

RQ1: How have the proportions of different code change operations evolved over time?
RQ2: How have code clone operations evolved over time?
RQ3: How has developer productivity evolved over time?

Methodology

Dataset

We use the World of Code (WoC) dataset [5].

Code Change Operations

We use the taxonomy from GitClear [3]:

• Added: New lines introduced
• Deleted: Lines removed
• Updated: Minor edits to existing lines
• Moved: Identical or near-identical lines relocated across files/modules
• Copy/Pasted: Identical line-level duplication within or across files
• Find/Replaced: Repetitive pattern-based edits
• Churned: Lines modified within 14–30 days of being added

This taxonomy may be modified or extended based on feasibility when using WoC.

Code Clone Detection

We analyze two types of code clones:

• Block clones: Contiguous cloned lines in the same repository
• Whole-file clones [4]: Entire cloned files in the same repository

The types of code clones detected may be adjusted depending on feasibility when using WoC.

Productivity Measurement

To measure developer productivity, we use the same metric as in [5]: the ratio of the number of commits to the number of developers.

Analysis

1. Plot overviews of metrics annually/monthly for visualization.
2. Group the data by programming language, developer, or project.
3. Use either June 21, 2022 (GitHub Copilot generally available [6]) or November 30, 2022 (ChatGPT release [7]) as the intervention date.
4. Examine data distributions and apply appropriate statistical tests (e.g., Mann-Whitney U) to detect significant differences between pre-GenAI and post-GenAI. Also, explore time series analysis.

Reference

[1] 2024 Stack Overflow Developer Survey, https://survey.stackoverflow.co/2024/ai#1-ai-tools-in-the-development-process
[2] Research: Quantifying GitHub Copilot’s impact in the enterprise with Accenture, https://github.blog/news-insights/research/research-quantifying-github-copilots-impact-in-the-enterprise-with-accenture/
[3] AI Copilot Code Quality: 2025 Look Back at 12 Months of Data, https://www.gitclear.com/ai_assistant_code_quality_2025_research
[4] Jahanshahi, M., Reid, D., & Mockus, A. "Beyond Dependencies: The Role of Copy-Based Reuse in Open Source Software Development." Accepted in ACM Transactions on Software Engineering and Methodology (TOSEM).
[5] Ma, Y., Dey, T., Bogart, C. et al. World of code: enabling a research workflow for mining and analyzing the universe of open source VCS data. Empir Software Eng 26, 22 (2021). https://doi.org/10.1007/s10664-020-09905-9
[6] GitHub Copilot is generally available to all developers, https://github.blog/news-insights/product-news/github-copilot-is-generally-available-to-all-developers/
[7] Introducing ChatGPT, https://openai.com/index/chatgpt/