Little was born in Chickasha, Oklahoma. He was the brother of singer DeEtta Little, best known for her performance of “Gonna Fly Now”, the main theme to Rocky. He was raised in California, graduating in 1957 from Kearny High School and initially attended San Diego City College, and then at San Diego State University, where he earned a bachelor’s degree in dramatic arts. After receiving a full scholarship to graduate school at Juilliard, he moved to New York. After completing studies at Juilliard, Little trained at the American Academy of Dramatic Arts.  Little made his professional debut in February 1967, appearing off-Broadway at the Village Gate as the Muslim Witch in the original production of Barbara Garson’s MacBird. This was followed by the role of Foxtrot in the original production of Bruce Jay Friedman’s long-running play Scuba Duba which premiered in October 1967.  The following year, he made his first film appearance in a small uncredited role in What’s So Bad About Feeling Good?, as well as his first television appearance as a guest star on two episodes of Felony Squad. A series of small roles followed in films such as John and Mary (1969) and Cotton Comes to Harlem (1970).  Little made his Broadway debut in 1969 as Lee Haines in John Sebastian and Murray Schisgal’s musical Jimmy Shine with Dustin Hoffman in the title role. In 1971, he returned to Broadway to portray the title role in Ossie Davis’s musical Purlie, for which he won a Tony Award and a Drama Desk Award for best actor in a musical.

A year later, Little was hired as an ensemble player on the syndicated TV variety weekly The David Frost Revue and he portrayed Shogo in Narrow Road to the Deep North on Broadway. In 1971, Little was chosen to portray the blind radio personality Super Soul in the car-chase movie Vanishing Point. That same year, he played Hawthorne Dooley in the pilot for The Waltons called “The Homecoming: A Christmas Story”, helping John-Boy Walton search for his father; then again in season four, in an episode called “The Fighter”, about a prizefighter who desired to build a church and be a preacher. He also played a burglar in a 1971 episode of All in the Family titled “Edith Writes a Song”.  He then starred in the ABC sitcom Temperatures Rising, which aired in three different iterations from 1972–74, with Little’s character of Dr. Jerry Noland as the only common element. Concurrently, he was cast as Sheriff Bart in the 1974 comedy film Blazing Saddles, after the studio rejected Richard Pryor, who co-wrote the script. Studio executives were apparently nervous over Pryor’s reputation as a racy comedian and thought Little would be a safer choice. This role earned him a BAFTA Award nomination as most promising newcomer.  He played a supporting role to Richard Pryor in the racing movie Greased Lightning (1977), based on the true life story of Wendell Scott, the first black stock car racing winner in America. In 1975, Little returned to Broadway to portray the role of Lewis in the original production of Murray Schisgal’s All Over Town under the direction of Dustin Hoffman. The following year, he appeared as Willy Stepp in the original production of Ronald Ribman’s The Poison Tree at the Ambassador Theatre.

After Blazing Saddles, Little appeared in many less successful films, such as FM, High Risk, Scavenger Hunt, Jimmy the Kid, Surf II, and Toy Soldiers. He also made guest appearances on The Mod Squad, The Rookies, Police Story, The Rockford Files, The Love Boat, Fantasy Island, ABC Afterschool Specials, The Fall Guy, MacGyver, the Waltons and ALF.  Little had a part in Fletch Lives, the sequel to 1985’s Fletch. He co-starred opposite Lauren Hutton and Jim Carrey in the 1985 horror comedy Once Bitten. In 1985, Little returned to Broadway to appear as Midge in Herb Gardner’s Tony Award-winning play I’m Not Rappaport, reuniting with Dear John star Judd Hirsch in New York and later on tour. The Broadway cast also featured Jace Alexander and Mercedes Ruehl.  In 1989, he appeared in the Dear John episode “Stand By Your Man”, for which he won the Outstanding Guest Actor Emmy, defeating Robert Picardo, Jack Gilford, Leslie Nielsen, and Sammy Davis, Jr.  Little was slated to star in the TV series Mr. Dugan, where he was to play a black Congressman, but that series was poorly received by real black Congressmen and was canceled before making it to air. In 1991, he replaced Frankie Faison as Ronald Freeman, a black dentist married to a white housewife, on the Fox sitcom True Colors. That same year, he also had a supporting role in the television series Bagdad Cafe, appearing in 12 episodes. Later that year, he was cast as a civil-rights lawyer in the TV docudrama, Separate but Equal, starring Sidney Poitier, who portrayed the first black U. S. Supreme Court Justice, Thurgood Marshall, NAACP lead attorney in the 1954 Supreme Court case desegregating public schools. He also appeared in the TV series MacGyver as Frank Colton, half of a bounty hunter brother duo.  Little’s last appearance was a guest part on an episode of Tales from the Crypt. Often afflicted by ulcers and general stomach problems during his life, Little died of colorectal cancer on October 22, 1992. His ashes were scattered into the Pacific Ocean.