Harvey Brown was born in Bridgetown (part of present-day Rahway, New Jersey). He graduated from the United States Military Academy at West Point, New York on July 24, 1818, as a second lieutenant of light artillery. He spent the first years of his military career on garrison duty in Boston, Massachusetts and New London, Connecticut as well as commissary duty in St. Augustine, Florida. Upon the reorganization of the U.S. Army in 1821, Brown was assigned to the 1st United States Artillery and was officially promoted to first lieutenant on August 28, 1821. From 1824 to 1825, he served as an aide-de-camp to Major General Jacob Brown, then commanding general of the U.S. Army, and was assigned to quartermaster duty for three years. In 1831, Brown was made a captain and participated in the Black Hawk War the following year. Brown also took part in the Second Seminole War as a lieutenant colonel, participating in the Battle of Wahoo Swamp, and was brevetted a major for “gallant conduct” on November 21, 1836. Between 1839 and 1841, he suppressed a series of disputes on the Canada–US border. During the Mexican-American War, Harvey Brown served as a major under Generals Zachary Taylor and Winfield Scott. He won distinction at the battles of Monterrey, Cerro Gordo, Contreras, and the Gate of Belén during the fall of Mexico City, brevetted a lieutenant colonel and colonel in the last two engagements respectively. Involved with military recruitment from 1848 to 1852, he fought in the Third Seminole War and remained in Florida until 1857. Brown served in various duties and was Inspector of Artillery prior to the American Civil War.

In January 1861, Harvey Brown was appointed military commander of Washington, D.C., and Fort McHenry and remained in this capacity until war broke out four months later. He was initially assigned to the 4th United States Artillery but, declining promotion to brigadier general, he was appointed commanding officer of the new 5th United States Artillery on May 14, 1861. Placed in command of the expedition to hold Fort Pickens in Confederate held Florida, he successfully defended Santa Rosa Island on October 9 and repulsed an attempt by the enemy to capture the fortress during November 22–23, 1861. He was brevetted a brigadier general for “gallantry and good conduct during the engagement of November 22 and 23, 1861 between Fort Pickens and the rebel batteries” and later appointed commander of New York Harbor in April 1862 and full military commander in January 1863. He remained in New York for over a year and, during the New York Draft Riots, Brown oversaw military operations against the rioters and was especially involved in the final suppression of the rioters. He retired from active duty on August 1, 1863, but was brevetted a major general on August 2, 1866, for his efforts during the riot. Harvey Brown was awaiting orders from June 29, 1864, to November 9, 1866, then served as superintendent of the Army recruiting service until April 5, 1867. He died at Clifton, New York, and is buried in Hazelwood Cemetery, Rahway, New Jersey.