A former federal judge who served on a secret court overseeing the National Security Agency’s secret surveillance programs said Tuesday the panel is independent but flawed because only the government’s side is represented effectively in its deliberations.

“Anyone who has been a judge will tell you a judge needs to hear both sides of a case,” said James Robertson, a former federal district judge based in Washington who served on the secret Foreign Intelligence Surveillance Act court for three years between 2002 and 2005. Robertson spoke during a Tuesday hearing of a federal oversight board directed by President Barack Obama to scrutinize government spying.

Robertson questioned whether the secret FISA court should provide overall legal approval for the surveillance programs, saying the court “has turned into something like an administrative agency.” He is one of several judges with FISA experience who have spoken out recently to affirm the court’s independence. But Robertson is the first to publicly air concerns about how the court grapples with the government’s vast secret data collection programs.

Much of the NSA’s surveillance is overseen by the FISA court, which meets in secret and renders rulings that are classified. Some of these rulings also likely been disclosed by Edward Snowden, the NSA systems analyst who leaked significant information about the spying program.

After Snowden began exposing the NSA’s operations in June, Obama instructed the U.S. Privacy and Civil Liberties Board to lead a “national conversation” about the secret programs. The board, which took testimony Tuesday on the secret surveillance programs, has been given several secret briefings by national security officials and plans a comprehensive inquiry and a public report on the matter.

Robertson said he asked to join the FISA court “to see what it was up to,” had previously played a central role in national security law. He was the judge who ruled against the Bush administration in the landmark Hamdan vs. Rumsfeld case, which granted inmates at the U.S. naval prison at Guantanamo Bay, Cuba, the right to challenge their detentions. That ruling was upheld by the Supreme Court in 2006.

Robertson quit the FISA court in 2005, days after the New York Times revealed widespread NSA warrantless wiretapping under President George W. Bush’s administration. Robertson had previously refused to explain his decision. But during a break in the hearing Tuesday he confirmed for the first time that he had “resigned in protest because the Bush administration was bypassing the court on warrantless wiretaps.”

Robertson said that FISA court judges have been scrupulous in pushing back at times against the government, repeatedly sending back flawed warrants. He also said he came away from his FISA experience “deeply impressed by the careful, scrupulous and fastidious work by the Justice Department” in obtaining secret surveillance warrants.

But he warned that Congress’ 2008 reform of the FISA system expanded the government’s authority by forcing the court to approve entire surveillance systems, not just surveillance warrants, as it previously handled. Under the FISA changes, “the court is now approving programmatic surveillance. I don’t think that is a judicial function,” he said.

Robertson said he was “frankly stunned” by a recent Times report that FISA court rulings had created a new body of law broadening the ability of the NSA to use its surveillance programs to target not only terrorists but suspects in cases involving espionage, cyberattacks and weapons of mass destruction.