I believe the records you seek can be found here within RG-249:
Memorandum Slips Containing Individual Histories of Federal Prisoners of War, ca. 1867 - ca. 1905.
"This series includes information on individuals who served in the Regular Army and the U.S. Navy; unknown organizations; citizens, clerks, scouts, employees and mechanics; teamsters and Virginia Militia members; and miscellaneous."
Dear Mr. Morgan,
Thank you for posting your request on History Hub!
We searched the National Archives Catalog and located the series Memorandum Slips Containing Individual Histories of Federal Prisoners of War, ca. 1867 - ca. 1905 and the series Memoranda and Reports Relating to Federal and Confederate Prisons and Prisoners of War, ca. 1867-1905 in the Records of the Commissary General of Prisoners, 1861-1905 (Record Group 249) that may be helpful for your research. In addition, we located the series Index to Prisoners of War Relating to U.S. Troops, U.S. Navy and Citizens, 1862-1865 in Record Group 249 that contains the name of the prisoner; his company and regiment (if applicable); if he was in the U.S. Navy and his position or rank; if he was a civilian, deserter, teamster, or sutler. Finally, we also located 15 additional series related to Andersonville in Record Group 249 that may contain some relevant information. Most of these records have not been digitized. For information about and access to these records, please contact the National Archives at Washington, DC - Textual Reference (RDT1) at email@example.com.
Due to the COVID-19 pandemic and pursuant to guidance received from the Office of Management and Budget (OMB), NARA has adjusted its normal operations to balance the need of completing its mission-critical work while also adhering to the recommended social distancing for the safety of NARA staff. As a result of this re-prioritization of activities, you may experience a delay in receiving an initial acknowledgement as well as a substantive response to your reference request from RDT1. We apologize for this inconvenience and appreciate your understanding and patience.
We hope this is helpful. Best of luck with your research!
Thank you. It might be helpful for other people to know that you can access the 15 additional series related to Andersonville that you mention are available online for free at Familysearch https://www.familysearch.org/search/image/index?owc=https://www.familysearch.org/service/cds/recapi/collections/2019835/… This collection isn't searchable, so you have to go through the records collection you want page by page to find things.