All of the swimmers are members of a relay team that included Ryan Lochte, a figure central to growing controversy. Lochte has left Brazil and is back in the United States. But the situation deteriorated for three of his teammates — Gunnar Bentz, Jack Conger and Jimmy Feigen.

>> Judge orders Ryan Lochte, James Feigen to stay in Brazil as robbery investigation continues

According to The Associated Press, Conger and Bentz were detained at the airport Wednesday night, then released after they agreed to comply with authorities. All three swimmers reportedly will talk to police Thursday.

Lochte — along with the rest of his relay team — said they were robbed at gunpoint in Rio early Sunday.

From NBC News:

"We got pulled over, in the taxi, and these guys came out with a badge, a police badge, no lights, no nothing just a police badge and they pulled us over," Lochte said. "They pulled out their guns, they told the other swimmers to get down on the ground — they got down on the ground. I refused, I was like we didn't do anything wrong, so — I'm not getting down on the ground."

According to Daily Mail, authorities are investigating a possible "false communication of a crime," which carries a maximum three-year jail sentence in Brazil.

While Lochte is already out of the country, Feigen remains in Brazil. He told USA Today Sports that he's safe, is cooperating with authorities and just wants to get out of Brazil.

>> Read more trending stories

As NBC News pointed out on its Wednesday evening broadcast, this is as much a public relations as it is a potential criminal issue. Rio de Janeiro is very sensitive to its crime issues, and Olympic athletes being robbed by fake cops — and gaining tremendous media attention — would be an embarrassment to the country.

Reports originally conflicted on whether Lochte was robbed, and later reports show that the Rio police seriously doubt whether the crime happened. According to Rio police, Lochte and his teammates never reported the crime, and the unit only began investigating after a report from Lochte's mother indicated that he had been robbed.

The Daily Mail also obtained video of Lochte arriving at the athletes' village Sunday, with at least some of his belongings with him.

A judge in Brazil ordered the swimmers to surrender their passports and cooperate with authorities. So far, Lochte is the only one who is back on U.S. soil.