Gustavus von Holst was born in Riga, Latvia in 1799 which was at that time part of the Russian Empire.  His father was Matthias Holst, a harp teacher to Russian imperial family.  His mother was Katherine Rogge, sister of Prince Rogge of Russia.  At some point in the early 1800s, Matthias and Katherine fled Russia because of his political views and came to live in England.

Like his father, Gustavus trained as a harpist and gave lessons to the daughters of wealthy families.  The harp was considered a suitable instrument for ladies to learn in the early nineteenth century.  He moved to Cheltenham in the mid nineteenth century due to the number of wealthy and fashionable families living in the town.  In order to sound more aristocratic (and therefore attract wealthier pupils), Gustavus added ‘von’ to his surname, which was soon copied by his brothers and sisters and even his father.
As well as teaching the harp, Gustavus composed music and he would arrange public concerts for his pupils to perform on the Promenade in Cheltenham.