The Labor Party is intensifying its attacks on trade union royal commissioner Dyson Heydon and has flagged asking the Governor-General to sack him.

Justice Heydon has been under sustained criticism from Labor for agreeing to speak at a Liberal Party event.

He has withdrawn from the engagement but Labor said his initial willingness to attend had left him appearing biased.

• Heydon admits he ‘overlooked’ Liberal link to speech
• Q&A: ‘That’s so offensive I don’t know where to start’
• ACTU to decide on Heydon disqualification

On Tuesday afternoon, Labor frontbencher Penny Wong confirmed the party would attempt to use Parliament to ask Governor-General Sir Peter Cosgrove to revoke Justice Heydon’s appointment.

However she acknowledged the plan was not guaranteed to succeed.

“What the Governor-General does with this message, or this address as it is formally called, will be a matter, if it is passed by the Senate, is a matter for him,” she said.

In question time, Prime Minister Tony Abbott again hit back at Labor’s criticism of Justice Heydon.

“Stop smearing a High Court judge,” he said.

Attorney-General George Brandis also stood by Justice Heydon.

Mr Brandis said Justice Heydon was “one of the most eminent and distinguished Australians this country has ever produced”.

The Law Council, a professional body representing lawyers, has criticised public attacks on Justice Heydon, saying he is highly regarded.

“The proper way for dealing with any question of bias, including apprehended bias, is to make an application for the commissioner to recuse himself, and for the commissioner to consider and rule on the application,” the Law Council said.

Unions are still considering whether they will make such an application.