The walls were built by farmers and land owners as land property markers. My grandfather built his wall around 40 acres over the past 50 years. Around Ashford, CT and smaller towns, those markers are still used today as property barriers. Just a little history I've learned from CT family.
The historical record makes clear that many were built by slaves, Indians, indentured servants, and children.