Rivers was born in El Reno, Oklahoma. His father was a gospel musician who had sung with the Fisk Jubilee Singers and the Silverstone Quartet, exposing Rivers to music from an early age. Rivers was stationed in California in the 1940s during a stint in the Navy. Here he performed semi-regularly with blues singer Jimmy Witherspoon. Rivers moved to Boston, Massachusetts in 1947, where he studied at the Boston Conservatory with Alan Hovhaness.  He performed with Quincy Jones, Herb Pomeroy, Tadd Dameron and others.

In 1959 Rivers began performing with 13-year-old drummer Tony Williams, who went on to have an impressive career. Rivers was briefly a member of the Miles Davis Quintet in 1964, partly at Williams’s recommendation. This edition of the quintet released a single album, Miles in Tokyo, recorded live in concert. However, Rivers’ playing style was a bit too avant-garde for what Davis had in mind for his music at this point, and he was replaced by Wayne Shorter shortly thereafter.  Rivers was signed by Blue Note Records, for whom he recorded four albums as leader and made several sideman appearances. Among noted sidemen on his own Blue Note albums were Jaki Byard, who appears on Fuchsia Swing Song, Herbie Hancock and Freddie Hubbard. He appeared on Blue Note recordings by Tony Williams, Andrew Hill and Larry Young.

Rivers derived his music from bebop, but he was an adventurous player, adept at free jazz. The first of his Blue Note albums, Fuchsia Swing Song (1964), adopts an approach sometimes called “inside-outside”. Here the performer frequently obliterates the explicit harmonic framework (“going outside”) but retains a hidden link so as to be able to return to it in a seamless fashion. Rivers brought the conceptual tools of bebop harmony to a new level in this process, united at all times with the ability to “tell a story”, which Lester Young had laid down as a benchmark for the jazz improviser.

His powers as a composer were also in evidence in this period: the ballad “Beatrice” from Fuchsia Swing Song has become an important standard, particularly for tenor saxophonists. For instance, it is the first cut on Joe Henderson’s 1985 The State of the Tenor, Vols. 1 & 2, and Stan Getz recorded it during the 1989 sessions eventually issued as Bossas & Ballads – The Lost Sessions.

During the 1970s, Rivers and his wife, Bea, ran a jazz loft called “Studio Rivbea” in New York City’s NoHo district. It was located on Bond Street in Lower Manhattan and was originally opened as a public performance space as part of the first New York Musicians Festival in 1970. Critic John Litweiler has written that “In New York Loft Jazz meant Free Jazz in the Seventies” and Studio Rivbea was “the most famous of the lofts”. The loft was important in the development of jazz because it was an example of artists creating their own performance spaces and taking responsibility for presenting music to the public. This allowed for music to be free of extra-musical concerns that would be present in a nightclub or concert hall situation. A series of recordings made at the loft were issued under the title Wildflowers on the Douglas label.

Rivers was also recruited by Clifford Thornton to lead a student world-music/free-jazz ensemble at Wesleyan University in 1971.  During this era Rivers continued to record, including several albums for Impulse!: Streams, recorded live at Montreux, Hues (both records contain different trio performances later collated on CD as Trio Live), the quartet album Sizzle and his first big-band disc, Crystals; perhaps his best-known work from this period though is his appearance on Dave Holland’s Conference of the Birds, in the company of Anthony Braxton and Barry Altschul.

In the early 1990s Sam and wife Beatrice moved to Florida, in part to expand his orchestra compositions with a reading band in Orlando. This band became the longest-running incarnation of the RivBea Orchestra. He performed regularly with his Orchestra and Trio with bassist Doug Mathews and drummer Anthony Cole (later replaced by Rion Smith.) From 1996 to 1998 he toured and recorded three projects for Nato Records in France with pianist Tony Hymas and others. In 1998, with the assistance of Steve Coleman, he recorded two Grammy-nominated big-band albums for RCA Victor with the RivBea All-Star Orchestra, Culmination and Inspiration (the title-track is an elaborate reworking of Dizzy Gillespie’s “Tanga”: Rivers was in Gillespie’s band near the end of the trumpeter’s life). Other recent albums of note include Portrait, a solo recording for FMP, and Vista, a trio with drummers Adam Rudolph and Harris Eisenstadt for Meta. During the late 1990s he appeared on several albums on Postcards Records.

In 2006, he released Aurora, a third CD featuring compositions for his Rivbea Orchestra and the first CD featuring members of his working orchestra in Orlando.  Rivers died from pneumonia on December 26, 2011 at the age of 88 in Orlando, Florida.