The newer git version seems to remove the branches directory for some odd reason. I've attempted debugging this but that git version isn't available for Ubuntu 24.04 - Github Actions seems to install it regardless.
Rather than waste cycles, just remove it from CI.
Git worktrees do not include the full .git directory - only a pointer to the actual directory - when keep-git-dir is set to true, which is incorrect for built containers. This change special-cases that combination, allowing folks on newer git installs to actually keep the .git directory during the build process.
Closes#4415
