A lawyer for an American who has spent more than four years imprisoned in Cuba said Monday that his client cannot take life in prison much longer and has said his goodbyes to his wife and a daughter.

Alan Gross was arrested in Cuba in 2009 while working covertly to set up Internet access in the Communist-run country. His attorney, Scott Gilbert, said in a statement Monday that his client “has withdrawn” and told him “life in prison is not a life worth living.” Gross has previously said through his lawyer that his 65th birthday, which took place in May, would be the last one that he “celebrates in Havana, one way or the other.”

Gross said goodbye to his wife and youngest daughter during a visit in July. Gross, who lived in Maryland before his arrest, had previously told his two daughters not to come see him in prison.

At the time Gross was arrested, he was working as a subcontractor for the U.S. Agency for International Development, which promotes democracy on the island. He had traveled repeatedly to Cuba on a mission to expand Internet access using sensitive technology typically available only to governments, a 2012 investigation found. Cuba considers USAID’s programs illegal attempts by the U.S. to undermine its government, and Gross was tried and sentenced to 15 years in prison.

Now, Gross has stopped exercising and his health is not good, said his attorney, who plans to visit his client this week. Gross’ hips are failing and he has lost most of the vision in his right eye. Gross’ “emotional deterioration has been severe,” Gilbert said, particularly following the death in June of his 92-year-old mother.

Gross had asked the Cuban government to be able to return to the United States for her funeral, but the request was denied.

Gross’ wife, Judy, said in Monday’s statement that she has never seen her husband in such bad shape during the time he has spent in prison. She wrote that “his decision to say goodbye to us was wrenching.”

Gross is also now refusing to see officials from the U.S. Interests Section in Havana, which Washington maintains instead of an embassy since the two countries do not have formal diplomatic relations.

The statement from Gross’ lawyer came as an investigation found that the Obama administration deployed young people to the island to stir political dissent under the guise of civic and health programs, putting those foreigners in danger.

Cuban officials did not immediately respond to a request for comment on the statement from Gross’ lawyer.