^{James Madison}
When Madison left office in 1817, he retired to Montpelier, his tobacco plantation in Orange County, Virginia, not far from Jefferson’s Monticello. He was 65 years old. Dolley, who thought they would finally have a chance to travel to Paris, was 49. As with both Washington and Jefferson, Madison left the presidency a poorer man than when he entered, due to the steady financial collapse of his plantation, aided by the continued low price of tobacco and his stepson’s mismanagement.

Insight into Madison is provided by the first “White House memoir,” A Colored Man’s Reminiscences of James Madison (1865), told by his former slave Paul Jennings, who served the president from the age of 10 as a footman, and later as a valet for the rest of Madison’s life. After Madison’s death, Jennings was purchased in 1845 from Dolley Madison by arrangement with the senator Daniel Webster, who enabled him to work off the cost and gain his freedom. Jennings published his short account in 1865. He had the highest respect for Madison and said he never struck a slave, nor permitted an overseer to do so. Jennings said that if a slave misbehaved, Madison would meet with the person privately to try to talk about the behavior.

Some historians speculate that Madison’s mounting debt was one of the chief reasons why he refused to allow his notes on the Constitutional Convention, or its official records which he possessed, to be published in his lifetime. “He knew the value of his notes, and wanted them to bring money to his estate for Dolley’s use as his plantation failed—he was hoping for one hundred thousand dollars from the sale of his papers, of which the notes were the gem.” Madison’s financial troubles weighed on him, and deteriorating mental and physical health would haunt him.

In his later years, Madison became extremely concerned about his historic legacy. He took to modifying letters and other documents in his possession: changing days and dates, adding and deleting words and sentences, and shifting characters. By the time he had reached his late seventies, this “straightening out” had become almost an obsession. As an example, he edited a letter written to Jefferson criticizing Lafayette: Madison not only inked out original passages, but imitated Jefferson’s handwriting as well in making changes.

“During the final six years of his life, amid a sea of personal [financial] troubles that were threatening to engulf him…At times mental agitation issued in physical collapse. For the better part of a year in 1831 and 1832 he was bedridden, if not silenced… Literally sick with anxiety, he began to despair of his ability to make himself understood by his fellow citizens.”

In 1826, after the death of Jefferson, Madison was appointed as the second Rector (“President”) of the University of Virginia. He retained the position as college chancellor for ten years until his death in 1836.

In 1829, at the age of 78, Madison was chosen as a representative to the constitutional convention in Richmond for the revising of the Virginia state constitution. It was his last appearance as a legislator and constitutional drafter. The issue of greatest importance at this convention was apportionment. The western districts of Virginia complained that they were underrepresented because the state constitution apportioned voting districts by county, not population. The growing population in the Piedmont and western parts of the state were not reflected in their representation in the legislature. Western reformers also wanted to extend suffrage to all white men, in place of the historic property requirement. Madison tried to effect a compromise, but to no avail. Eventually, suffrage rights were extended to renters as well as landowners, but the eastern planters refused to adopt population apportionment. Madison was disappointed at the failure of Virginians to resolve the issue more equitably.

Madison was very concerned about the continuing issue of slavery in Virginia and the South. He believed that transportation of free American blacks to Africa offered a solution, as promoted by the American Colonization Society (ACS). He told Lafayette at the time of the convention that colonization would create a “rapid erasure of the blot on our Republican character.” The British sociologist, Harriet Martineau, visited with Madison during her tour of the United States in 1834. She characterized his faith in colonization as the solution to slavery as “bizarre and incongruous.” Madison may have sold or donated his gristmill in support of the ACS. The historian Drew R. McCoy believes that “The Convention of 1829, we might say, pushed Madison steadily to the brink of self-delusion, if not despair. The dilemma of slavery undid him.” Like most African Americans of the time, Madison’s slaves wanted to remain in the U.S. where they had been born and believed their work earned them citizenship; they resisted “repatriation”.

Through failing health, Madison wrote several memoranda on political subjects, including an essay against the appointment of chaplains for Congress and the armed forces. He felt it would produce religious exclusion but not political harmony.

Between 1834 and 1835, Madison sold 25% of his slaves to make up for financial losses on his plantation. Madison lived until 1836, increasingly ignored by the new leaders of the American polity. He died at Montpelier on June 28, as the last of the Founding Fathers. He was buried in the Madison Family Cemetery at Montpelier.

In 1842, Dolley Madison sold the Montpelier mansion, and in 1844 sold the extensive plantation lands to Henry W. Moncure. She leased half of the remaining slaves to Moncure. The other half were inherited by her, her son John Payne Todd, and James Madison, Jr., a nephew. Between 1845 and 1849 Todd sold numerous slaves; by 1851 he retained only 15 at his residence. By 1850, the Montpelier plantation was a “ghost of its former self”. In 1851, Montpelier was owned by Thomas Thorton, an Englishman; he held 40 slaves.