Protesters have been removed from an inner-city park as work began on the stadium for the 2032 Olympic Games, despite warnings the project would destroy a culturally and ecologically significant area.

The 2032 Games authority on Monday officially took possession of the Victoria Park site in Brisbane after demonstrators were moved on by police early in the morning.

A handful of activists regrouped outside the park’s fenced-off perimeter as construction begun.

The Games authority is racing the clock to complete the 63,000-seat, $3.8 billion stadium that will be the 2032 centrepiece, almost five years after Brisbane won the Games bid.

Premier David Crisafulli turned the first sod on Monday morning, saying Queenslanders had been urging the government to “get on with it” after years of delays.

“It’s game on – we have a plan to deliver for the 2032 Games and beyond, and today, we get cracking on delivering it,” he said.

“Victoria Park will be the beating sport, cultural and green heart of Brisbane, and a place all Queenslanders can be proud of.”

Crisafulli said more than two‑thirds of Victoria Park would remain free green public space once the project was complete.

However, protesters and Traditional Owners have warned the project will destroy a culturally and ecologically significant green space.

Tensions boiled over in the countdown to the official start of construction. Five people were arrested on Friday as police began to move on protesters in what critics described as a surprise raid.

It was less eventful when protesters took advantage of the last day they could access the site, with hundreds rallying at the sprawling park in the city’s north on Sunday.

The last of the protesters was moved on early on Monday as the Games Independent Infrastructure Coordination Authority took possession of the park.

Fencing was installed and preparatory works began, including service decommissioning, testing and earthworks.

Local resident Josie – who did not want to provide her last name – said she felt compelled to return to the park on Monday after attending Sunday’s protest.

“I have hope,” she said as about 20 police officers blocked public access to the park’s main entrance.

“I want to show up for the First Nations people,” she said.

Save Victoria Park organiser Andrea Lunt said protesters were disappointed but still considering their next steps.

“This space with its heritage and its history is such a special asset for Brisbane,” she said.

“We’re not an anti-Olympics group but we don’t support the Olympics in its current form, which is going to take away precious green space from future generations.”

Lunt said the park held deep significance for Traditional Owners, with dozens of ancient trees she described as “ecologically irreplaceable”.

Indigenous elders have declared Victoria Park a place of importance for many First Nations peoples and want their concerns heard as national Aboriginal heritage laws are tested.

Federal Environment Minister Murray Watt said he had received 10 applications seeking to protect a significant Aboriginal area “under threat of injury or desecration” at the park.

Two have been declined while the others remain under consideration.

Watt said his powers over the proposed stadium were limited, stressing that cultural heritage declarations are not designed to stop the project outright.

“People seem to think that these are stop work orders, and they’re not,” he told ABC Radio Brisbane.

Any declaration he made would instead set out what must be done “to preserve or protect an area from being injured in some way or desecrated in some way”, Watt said.

Crisafulli said while people had a right to protest but they could not remain inside an active work zone.

Deputy Premier Jarrod Bleijie said major building contracts would be awarded within months.

-AAP