Wolman was born to an Orthodox Jewish family in Shenandoah, Pennsylvania, the son of a grocer. He worked in the family business into his high school years, when his father had a stroke. Not graduating, Wolman joined the Merchant Marines, returned home, and moved to Washington, D.C. In the 1950s, he began his own construction company, building apartment units and offices.

In 1963, Wolman bought the Eagles franchise from the “Happy Hundred,” a group of investors that owned the team from 1949–1963, for a sale price of $5,505,000, to become the youngest owner in the league. He was also the owner of Connie Mack Stadium.  He was also one of the founding owners, briefly in 1967, of the Philadelphia Flyers of the National Hockey League. Over the next two years, his $100-million financial empire crumbled into bankruptcy, and he was forced to give up his interests in both teams. In 1967, he sold his Flyers interest to his co-owners, with Ed Snider assuming control, along with his partners, Bill Putnam and Joe Scott. In 1969, he sold the Eagles to Leonard Tose for a reported $16.1 million, then a record price for a professional sports team.

Wolman was a member of the Pennsylvania Sports Hall of Fame; one of its chapters is named after him. Wolman was a philanthropic individual who never forgot his coal mining town roots. He kept in contact with his hometown of Shenandoah, Pennsylvania through good and bad times.  Wolman also initiated the development of the John Hancock Center, a 100-story skyscraper in Chicago, Illinois, but withdrew from the project because of financial difficulties.  Wolman lived in Potomac, Maryland, with his wife, Bobbie. He was a father of two and a grandfather of eight. His favorite grandson, Brian Wolman, shattered all wide receiver records playing football while attending Churchill High School, and went on to play football at Hofstra University. He died on August 6, 2013, at the age of 86.