A subset of upload jobs were hanging. Investigation pointed to a handful of VMs that were recently launched in our cloud which were firewalled off from access. Condor, our job scheduler, needs to be able to reach the VMs in order to send jobs to it. When jobs were scheduled for those VMs, they just hung indefinitely. We have now corrected the issue in code, so we won’t launch any more VMs with this issue.