The observatory said on Saturday it will issue the Strong Wind Signal No 3 at 3.40pm, as Super Typhoon Yinxing edges closer to Hong Kong.
It said local winds are expected to strengthen later in the day, with winds on high ground occasionally reaching gale force.
The Observatory said Yinxing will skirt more than 300 kilometres to the south of Hong Kong on Sunday morning, so a tropical cyclone warning signal is expected to remain in force until then.
The cross-harbour swim was originally scheduled for Sunday, but the organiser decided to call off the race.
"Due to the potential for inclement weather brought by Tropical Storm Yinxing and to ensure the safety of all swimmers, under careful consideration, the organisers have made the difficult decision to cancel the New World Harbour Race 2024," it wrote on social media.
It added that all qualified swimmers will receive a full refund.