"Obviously, I disagree with the conservatorship altogether," said Tom, 82, of his younger brother Robert Girdardi being his conservator.
"I think that we should put together the reasons why the conservatorship should be dissolved, and then we'll address it, address the court," the once-famed attorney continued. "Right now, I have nothing to say to the court."
During the hearing, the judge ruled that Robert be appointed as the conservator of Tom's "person and estate," leaving him responsible for deciding the appropriate care for Tom, as well as giving him control over his estate, according to court records obtained by PEOPLE.
The judge also found that Tom "consents and does not object" to the conservatorship. Robert was previously named his brother's temporary conservator in the months leading up the hearing.
"It's obviously a heartbreaking situation for Robert, but we agree with the court's rulings yesterday," Robert's lawyer, Nicholas Van Brunt, said in a statement to PEOPLE.
In a previous document filed in March, Tom's court appointed lawyer said that during his discussions with his client, Tom "had no objection to Robert becoming his conservator" though he noted that in his opinion, Tom "could not fully comprehend the nature of the proceedings."
A lawyer for Tom did not immediately respond to a request for comment.
Earlier this year, Tom's lawyers claimed that he "has had issues" with "mental competence" and it was later announced that he had been diagnosed with dementia and late-onset Alzheimer's disease.
He underwent a mental assessment on Feb. 26 as part of Robert's petition to become his permanent conservator. Robert was granted a temporary conservatorship in February, which became permanent at Wednesday's hearing.
Dr. Nathan Lavid, a Long Beach forensic and clinical psychiatrist, wrote a sworn declaration submitted to the Superior Court of California on March 10 stating that Tom was medically unfit to attend any court proceedings "for the foreseeable future," according to a capacity declaration previously obtained by PEOPLE.
"Dementia impairs his ability to understand the hearing," Lavid wrote. "His emotional distress is directly related to his dementia and exacerbated by his confusion."
Prior to his diagnosis, Erika, 49, had filed for divorce from Tom, telling PEOPLE in November that it was not "a step taken lightly or easily."
In the filing, Erika - also known as Erika Jayne - sought spousal support and requested the court to terminate its ability to award spousal support to Tom. Tom responded by asking the court to terminate its ability to award spousal support to Erika, requesting that the Bravo star pay his attorney fees and costs.
A month after she filed for divorce, Tom and Erika were sued for allegedly using their split to embezzle money. Erika declined to comment on the case when reached by PEOPLE at the time and Tom did not respond to multiple requests for comment.
In December, Tom and his law firm, Girardi & Keese (GK), were held in civil contempt and had their assets frozen by a judge.
Also in December, Tom was hit with another lawsuit,
this time from his partner, Robert Keese, and fellow business partners,
Robert Finnerty and Jill O'Callahan, to dissolve their venture
together. (Tom has not responded to either lawsuit, publicly or in