Seeking records on paupers in the 1850's-1900's

A wide variety of state, local, and charitable organizations would have taken notice of the poor, so what you would need to do is look for their records.  I'd particular suggest looking for things related to county poor commissioners, poor houses (aka alms houses), work houses, poor relief, pauper farms, and relief societies.  For the post-Civil War era, there were also varies efforts by states (but not the federal government) to provide particular relief to Confederate veterans and widows. There would have been minimal federal involvement with poor relief during this time period, however residents of poor houses and pauper farms might be listed on federal censuses.  For the Reconstruction time period there would be Freedmen's Bureau records in the National Archives that would provide insights into the lives of African Americans.

19th century social history and lives of the poor have gotten increased attention from historians since the 1960s, and you can start by searching libraries to find what books have already been written about the topic, and then look at the sources they use.  One example would be Before the New Deal: Social Welfare in the South, 1830-1930, by Elna C. Green, published by University of Georgia Press, 1999.

You can also contact the Georgia Archives, the Georgia Historical Society, local history and genealogy societies, county courthouses, municipal archives, and religious and charitable organizations that date back that far.
  
The following may be of interest.

http://www.poorhousestory.com/index.htm

https://www.georgiaencyclopedia.org/articles/history-archaeology/welfare-and-poverty-during-civil-war

https://www.findagrave.com/cemetery/2624844/old-lowndes-county-poor-farm-cemetery

https://www.findagrave.com/cemetery/2535803/dekalb-county-pauper-farm

The newsletter linked below includes references to a Caroll County Pauper Book from the early 1900s.  I'm not sure if this is an exceptional thing or the norm for Georgian counties, or if there would be any for the 19th century.

http://www.ccgsga.org/wp-content/uploads/2011/10/CCGS-Fall-2010.pdf

This section of the Code of the State of Georgia in 1882 (Part I, Tivle VI, Chapter VIII) deals with "The County Poor."

https://books.google.com/books?id=B81JAQAAIAAJ&pg=PA170&lpg=PA170&dq=County+Pauper+Georgia&source=bl&ots=M6S0IXBR_X&sig=…