The first girl in her family, Dolley Payne was born on May 20, 1768, in the Quaker settlement of New Garden, North Carolina, in Guilford County to John Payne Jr. and Mary Coles Payne. Her parents, both Virginians, had moved there in 1765. Her mother, Mary Coles, a Quaker, had married John Payne, a non-Quaker, in 1761. Three years later, he applied and was admitted to the Quaker Monthly Meeting in Hanover County, Virginia, where Coles’ parents lived. Dolley Payne was reared in the Quaker faith.  By 1769, the family returned to Virginia to live near the Coles family. As a young girl, Dolley grew up in comfort at her parents’ plantation in rural eastern Virginia, and she grew deeply attached to her mother’s Coles family. In total, the Paynes had four boys (Walter, William Temple, Isaac, and John) and four girls (Dolley, Lucy, Anna, and Mary).

In 1783, following the American Revolutionary War, John Payne emancipated his slaves, as did numerous slaveholders in the Upper South. Some, like Payne, were Quakers, who had long encouraged manumission; others were inspired by revolutionary ideals. From 1782 to 1810, the proportion of free blacks to the total black population in Virginia increased from less than one percent to 7.2 percent, and more than 30,000 blacks were free.  Payne moved his family to Philadelphia, where he went into business as a starch merchant. By 1789, however, his business had failed. He died in October 1792. Dolley’s mother Mary Payne initially made ends meet by opening a boarding house. A year later she moved to western Virginia to live with her daughter Lucy, who had married George Steptoe Washington, a nephew of George Washington. The widow Mary Coles Payne took her two youngest children, Mary and John, with her.

In January 1790, Dolley Payne had married John Todd, a Quaker lawyer in Philadelphia. They quickly had two sons: John Payne (called Payne) and William Temple (born July 4, 1793). After their mother left Philadelphia in 1793, Dolley’s sister Anna Payne moved in with the Todds to help with the children.  In August 1793, a yellow fever epidemic broke out in Philadelphia. 5,019 people died during the four months that the epidemic raged. Twenty thousand people fled the city by mid-September. Tragedy struck Dolley, as she lost both her husband and younger son William in the epidemic, as well as her Todd parents-in-law. Husband and child both died on October 24, 1793, at the ages of twenty-nine and three months, respectively. Dolley Todd was a widow at the age of twenty-five, with her young son Payne to support. Her family had perished and she was alone with only one companion, her son.

Dolley Payne Todd and James Madison, who represented Virginia in the U.S. House of Representatives (the capital met in Philadelphia from 1790–1800), likely encountered each other at social events in the temporary federal capital. Some sources state that Aaron Burr, who had been a fellow student with Madison at the College of New Jersey (now called Princeton University) and with whom he had remained friends, stayed at a rooming house where the widow Dolley also resided, and it was Aaron’s idea to introduce the two. In May 1794, Burr made the formal introduction between Madison and the young widow. Madison was seventeen years her senior and, at the age of forty-three, a longstanding bachelor. The encounter apparently went smoothly, for a brisk courtship followed; by August Dolley accepted his proposal of marriage. As he was not a Quaker, she was expelled from the Society of Friends for marrying outside her faith. They were married on September 15, 1794, and lived in Philadelphia for the next three years.

In 1797, after eight years in the House of Representatives, James Madison retired from politics. He returned with his family to Montpelier, the Madison family plantation in Orange County, Virginia. There they expanded the house and settled in. When Thomas Jefferson was elected as the third president of the United States in 1800, he asked Madison to serve as his Secretary of State. Madison accepted, and his family: Dolley, her son Payne Todd (as he was commonly called), and her sister Anna Payne, moved to Washington, with their domestic slaves. They stretched to take a large house, as Dolley believed entertaining would be important in the capital.

Dolley Madison worked with the architect Benjamin Henry Latrobe to furnish the White House, the first official residence built for the president of the United States. In addition, as Jefferson was a widower, she sometimes served as his First Lady for official ceremonial functions.  In the approach to the 1808 presidential election, with Thomas Jefferson ready to retire, the Democratic-Republican caucus nominated James Madison to succeed him. He was elected President, serving two terms from 1809 to 1817, and Dolley became the official First Lady. She was renowned for her social graces and hospitality, and contributed to her husband’s popularity as president. In 1812 Madison was re-elected; the same year that the War of 1812 with Great Britain had begun.

As the invading British army neared Washington in 1814 and the White House staff hurriedly prepared to flee, Dolley Madison ordered the Stuart painting, a copy of the Lansdowne portrait, to be removed:

“Our kind friend Mr. Carroll has come to hasten my departure, and in a very bad humor with me, because I insist on waiting until the large picture of General Washington is secured, and it requires to be unscrewed from the wall. The process was found too tedious for these perilous moments; I have ordered the frame to be broken and the canvas taken out”….. “It is done, and the precious portrait placed in the hands of two gentlemen from New York for safe keeping. On handing the canvas to the gentlemen in question, Messrs. Barker and Depeyster, Mr. Sioussat cautioned them against rolling it up, saying that it would destroy the portrait. He was moved to this because Mr. Barker started to roll it up for greater convenience for carrying.”

Popular accounts during and after the war years tended to portray Dolley Madison as the one who removed the painting, and she became a national heroine. Early twentieth-century historians noted that Jean Pierre Sioussat, a Frenchman, had directed the servants in the crisis.  Dolley Madison hurried away in her waiting carriage, along with other families fleeing the city. They went to Georgetown and the next day crossed over the Potomac into Virginia. When the danger receded after the British left Washington a few days later, she returned to the capital to meet her husband.

On April 6, 1817, a month after his retirement from the presidency, Dolley and James Madison returned to the Montpelier plantation in Orange County, Virginia.  In 1830, Dolley Madison’s son by her first marriage, Payne Todd, who had never found a career, went to debtors’ prison in Philadelphia. The Madisons sold land in Kentucky and mortgaged half of the Montpelier plantation to pay his debts.

James Madison died at Montpelier on June 28, 1836. Dolley remained at Montpelier for a year. One of her nieces, Anna Payne, came to live with her, and Todd also came for a lengthy stay. During this time, Dolley Madison organized and copied her husband’s papers. Congress authorized $55,000 as payment for editing and publishing seven volumes of the Madison papers, including his unique notes on the 1787 convention.  In the fall of 1837, Dolley Madison returned to Washington, charging Todd with the care of the plantation. She moved with her niece Anna Payne into a house located on Lafayette Square. It was bought by her sister Anna and her husband Richard Cutts. Madison took Paul Jennings with her as a butler, and he was forced to leave his family in Virginia.

While Madison was living in Washington, Payne Todd was unable to manage the plantation, due to alcoholism and related illness. Madison tried to raise money by selling the rest of the president’s papers. She agreed to sell Jennings to Daniel Webster, who allowed him to gain his freedom by paying him through work.  Unable to find a buyer for the papers, she sold Montpelier, its remaining slaves, and the furnishings to pay off outstanding debts.

Paul Jennings, the former slave of the Madisons, later recalled in his memoir,

“In the last days of her life, before Congress purchased her husband’s papers, she was in a state of absolute poverty, and I think sometimes suffered for the necessaries of life. While I was a servant to Mr. Webster, he often sent me to her with a market-basket full of provisions, and told me whenever I saw anything in the house that I thought she was in need of, to take it to her. I often did this, and occasionally gave her small sums from my own pocket, though I had years before bought my freedom of her.”

In 1848, Congress agreed to buy the rest of James Madison’s papers for the sum of $22,000 or $25,000.  In 1842, Dolley Madison joined St. John’s Episcopal Church, Lafayette Square in Washington, D.C. This church was attended by other members of the Madison and Payne families.  She died at her home in Washington in 1849 at the age of 81. She was first buried in the Congressional Cemetery, Washington, D.C., but later was re-interred at Montpelier next to her husband.