dirt poor: extremely poor
This phrase first appears in print during the 1930s, for example in this line from Edna Ferber's 1931 novel, American Beauty: "Judy's family were high and mighty and gave themselves airs—though by that time they were as dirt poor as the rest of us."
The term's exact origin is unclear. One suggestion is that it grew out of the 1930s Dust Bowl. Thousands of Great Plains farmers during this time lost everything except the dirt of their unworkable farms, now worthless. Also, huge dust storms created an unusual amount of dirtiness by blowing sand into every corner of their farmhouses. However, the Dust Bowl was only getting started in 1931 when Ferber and others were already using dirt poor in their writing, so that association probably arose later.
Dirt as a synonym for soil is an American invention. The English generally use the word to mean filth—either real or metaphorical. Dirt farmer, dirt road, hit pay dirt, eat dirt, and do someone dirt were all coined in the United States.