Born in Buffalo, New York, before moving to Cleveland, Ohio, Hall was from a musical family, his mother played the piano, his grandfather violin, and his uncle guitar. He began playing the guitar at age ten when his mother gave him an instrument as a Christmas present. As a teenager in Cleveland, he performed professionally, and also took up the double bass. Hall’s major influences since childhood were tenor saxophonists Coleman Hawkins, Lester Young, Paul Gonsalves, and Lucky Thompson. While he copied out solos by guitarist Charlie Christian (and later Barney Kessel), it was horn players from which he took a lead.

In 1955, Hall attended the Cleveland Institute of Music where he studied piano and bass, in addition to theory. About a year later, he moved to Los Angeles, where cool jazz was prominent at the time. He focused on classical guitar, and, from 1955 to 1956, played in Chico Hamilton’s quintet. It was at this time that he began to gain attention.  In the Jimmy Giuffre Three, Hall developed some of his own personal musical preferences, including “challenging arrangements and interactive improvisation in duos and trios.” He taught at the Lenox School of Jazz in 1959; toured with Jazz at the Philharmonic; and worked with Ben Webster (1959), Bill Evans (1959), Paul Desmond (1959–65), Ella Fitzgerald in Europe (1960), Lee Konitz (1960–61), Sonny Rollins (1961-2, 1964), and Art Farmer (1962-1964). Working with all of these prominent and established artists furthered Hall’s career and aided in producing his own bands and own styles.

By 1960, Hall was living in New York. In 1962, he led a trio with Tommy Flanagan and Ron Carter—with the addition of Red Mitchell in 1965. Furthermore, he landed a gig playing with Bill Berry, Bob Brookmeyer, Benny Powell, Art Davis and Jake Hanna as a house band for The Merv Griffin Show on television. Most notably, he arranged and recorded duos with Evans and Carter, which allowed his complex arrangements and improvisations to shine.  Hall had incredible improvisational ability and creativity.[citation needed] He was an arranger as much as an artist, known for developing motives and using blues inflections. These characteristics are showcased in his 1975 album Jim Hall Live!, with Don Thompson and Terry Clarke. Around this time he also recorded with pianist George Shearing and classical violinist Itzhak Perlman. He further continued creating music with Mitchell and Ron Carter until 1985.

In the 1990s, Hall continued to tour and record all over the world. His sidemen included drummers Bill Stewart and Andy Watson; bass players Scott Colley and Steve LaSpina; pianists Gil Goldstein and Larry Goldings. At times, Hall included Chris Potter on the tenor saxophone. These players are featured on Hall’s video Master Sessions with Jim Hall from 1993. Hall appeared as a guest soloist in Michel Petrucciani’s trio with Wayne Shorter in 1986 and performed at the Village Vanguard with Bill Frisell. In 1990, he hosted the JVC Jazz Festival New York, which also featured Pat Metheny and John Scofield. After this, he played a number of duo concerts with Metheny. In 1994, Hall recorded a completely solo album. Furthermore, in 1996, he returned to Europe to lead a quartet with Joe Lovano.

In 1997, Hall received the New York Jazz Critics Award for Best Jazz Composer/Arranger, which was a very important milestone in his career. His pieces for string, brass, and vocal ensembles can be heard on his Textures and By Arrangement recordings. His original composition, “Quartet Plus Four”, a piece for jazz quartet featuring the Zapolski string quartet, was debuted in Denmark, where he was awarded the Jazzpar Prize.  His last orchestral composition was a concerto for guitar and orchestra, commissioned by Towson University in Maryland for The First World Guitar Congress, which was debuted in June 2004 with the Baltimore Symphony. He was awarded an NEA Jazz Masters Fellowship award in January 2004. Hall was one of the first artists to join the fan-funded label ArtistShare and released Magic Meeting in 2005. In November 2008 the double album Hemispheres was released through ArtistShare, featuring fellow guitarist and former student Bill Frisell with Scott Colley (bass) and Joey Baron (drums).

Hall performed in a project titled The Live Project, where he shared his music making process through ArtistShare as well as interviews with other musicians about his lasting influence. In 2010, Hall and Baron recorded a duo album, which listeners can view on the recording Coming to Life. In 2012 at the age of 81, Hall had gigs at the Blue Note in New York City and at a number of jazz festivals in the US as well as in Europe.  Hall died in his sleep in his Manhattan, New York apartment on December 10, 2013.