No. U.S. citizens do not need a passport to travel directly between the U.S. mainland and the U.S. Virgin Islands, because the islands are a U.S. territory and are considered part of the United States.
U.S. Customs and Border Protection (CBP) says citizens returning directly from a U.S. territory are not considered to have left the country, so they do not have to present a passport, and the Western Hemisphere Travel Initiative document rules do not apply to travel between the mainland and the territory. The exception is a side trip: if you also visit a place that is not a U.S. territory, you must present a passport to re-enter the United States. CBP still recommends carrying evidence of citizenship, such as a U.S. passport, birth certificate, or Trusted Traveler Program card, because it speeds customs processing when you fly from St. Thomas or St. Croix back to the mainland. Departing travelers also go through customs and agriculture checks, and those who stayed in the territories for at least 48 hours get a $1,600 duty-free exemption, with a flat 1.5% rate on the next $1,000 of purchases.
Source: U.S. Customs and Border Protection, Western Hemisphere Travel Initiative FAQs
Image: Cruise ships in Charlotte Amalie Harbor, Saint Thomas, by OldPine, public domain, via Wikimedia Commons.