Randolph was born in New York City, the son of Jewish immigrants from Russia and Romania. His mother, Dorothy (née Shorr), was an insurance agent, and his father, Louis Cohen, was a hat manufacturer. In the 1930s, he spent his summers at the Pine Brook Country Club in Nichols, Connecticut which was the summer home of the Group Theatre. He made his Broadway debut in 1938 in Coriolanus. Randolph joined the United States Army Air Forces in World War II. He had a small role in the 1948 film in The Naked City. He and wife Sarah Cunningham were blacklisted from working in Hollywood films and in New York film and television and radio after 1948. In 1955 they were both called before the HUAC to testify concerning ongoing investigations concerning Communist infiltration in the American entertainment industry. Both he and his wife refused to answer questions and cited the Fifth Amendment protection against testifying against themselves. John and Sarah Randolph were very active in AFTRA, SAG and in Actor’s Equity, were elected members of union boards and became vice presidents at various times during their careers. Randolph was one of the last blacklisted actors to regain employment in Hollywood films when director John Frankenheimer cast him in a major role in Seconds in 1966. Randolph was in the original New York stage productions of The Sound of Music (as Von Trapp’s butler, Franz), Paint Your Wagon, and The Visit. He won the 1987 Tony Award for Best Performance by a Featured Actor in a Play for his performance in the Neil Simon play, Broadway Bound. He made his last Broadway appearance in 1991 in Prelude to a Kiss. Randloph made numerous screen and television appearances in secondary roles, among which he played Donna Pescow’s father in-law on the television series, Angie.

In the 1970s he made three appearances as Cornelius “Junior” Harrison, father of Emily Hartley in The Bob Newhart Show. In 1975, Randolph was cast as General Philip Blankenship in The New Original Wonder Woman pilot. He was replaced by Richard Eastham in the television series. He had an uncredited role in the 1976 film All The President’s Men as the voice of Richard Nixon’s Attorney General John Mitchell. In 1979, he had a notable guest appearance with Ed Begley Jr. on M*A*S*H as an adjutant Army general admiring the culinary prowess of a master chef errantly assigned as a foot soldier in a front unit. John also appeared in a made for TV movie entitled The Gathering, a Christmas themed show along with Edward Asner and Maureen Stapleton. This 1977 movie was nominated and won the Emmy for Outstanding Special – Drama or Comedy. In 1982, he appeared in a first season episode of Family Ties as Jake Keaton, Steven Keaton’s father. In 1990, he co-starred in the NBC comedy Grand. He appeared in “The Handicap Spot”, an early episode of the television sitcom Seinfeld as Frank Costanza, George Costanza’s father. He was later replaced by Jerry Stiller. In 1995, the scenes in which Randolph appeared were re-shot with Stiller. The re-shot version is shown in syndication in the United States. The original version, with Randolph, can be seen outside the U.S. and on DVD. He also appeared in a Season 2 episode of Matlock as Sam Gerard, the head of a crime family in “The Investigation”. He costarred with Alec Guinness, Leo McKern, Jeanne Moreau and Lauren Bacall, in the BBC production of A Foreign Field (1993) as a World War II veteran returning to France to find the woman he fell in love with. Among his most famous film roles was Chief Sidney Green in Serpico (1973), directed by Sidney Lumet. He also played the father of Charlie Partana (played by Jack Nicholson) in Prizzi’s Honor and Clark W. Griswold, Sr. in National Lampoon’s Christmas Vacation (with Chevy Chase). One of his last film roles was as Joe Fox’s grandfather in You’ve Got Mail (1998). On February 24, 2004, Randolph died at age eighty-eight.