Police blocked roads, preventing pro-democracy protesters from getting to the harbor-front venue close to where the last colonial governor, Chris Patten, tearfully handed back Hong Kong to China in the pouring rain in 1997.
Thousands gathered in the afternoon in a sprawling park named after Britain's Queen Victoria, demanding Xi allow universal suffrage.
"This protest is the most urgent in the past 20 years," said lawmaker Eddie Chu, as some demonstrators marched with yellow umbrellas, a symbol of democratic activism in the city, and held aloft banners denouncing China's Communist "one party rule."
Others criticized China's Foreign Ministry which on Friday said the "Joint Declaration" with Britain over Hong Kong, a treaty laying the blueprint over how the city would be ruled after 1997, "no longer has any practical significance."
A small group of pro-democracy activists near the venue were roughed up by a group of men who smashed up some props in ugly scuffles while surrounded by more than 100 police. Nine democracy protesters, including Joshua Wong and lawmaker "long hair" Leung Kwok-hung, were bundled into police vans while several pro-China groups remained, cheering loudly and waving red China flags.
The activists, in a later statement, said the assailants had been "pro-Beijing triad members."Other protesters unfurled a massive yellow banner, with the words "I want real universal suffrage," on the waterfront of Hong Kong's Victoria Harbour, but were later taken away by police.