Polski frontendu and American Back-end: GitHub Profile Recruitment Bias in GPT-4

This repository serves as the online appendix for the paper "Polski frontendu and American Back-end: GitHub Profile Recruitment Bias in GPT-4". Large Language Models (LLMs) such as Generative Pre-trained Transformers (GPTs) have taken the world by storm, demonstrating their ability to automate not only tedious tasks but also showing some degree of proficiency in software engineering tasks. A key concern for GPTs is their "black box" nature, where the system is viewed only in terms of its inputs and outputs (or transfer characteristics), without any knowledge of its internal workings, ultimately leading to potential societal bias in its results. In this short paper, we empirically explore the software engineering context to understand how well a GPT model can perform recruitment tasks for a geographically-diverse software team. We conduct an initial set of experiments using GitHub Profiles from four countries (i.e., the United States, India, Nigeria, and Poland) to recruit a 6-person software development team, analyzing a total of profiles over a 5-year period (2019–2023). Results indicate that GPT tends to prefer some countries over others, even when some of the profiles have been manipulated to contain counterfactuals, such as having two profiles' location string swapped. Furthermore, GPT was more likely to assign certain developer roles to users of a specific nationality. Overall, this study reveals insights into the inner workings of the GPT and has implications for mitigating these potential biases.