The chief minister's reaction came a day after a section of the media in Assam reported that the governor had said "Hindustan is for Hindus".
However, Acharya on Sunday clarified that the media only published half of his statement.
Acharya said he had said persecuted Hindus in Bangladesh or in any part of the world have the right to come to India and live. It was the duty of the government to welcome them, he added.
Gogoi, a Congress leader, said: "By saying that Hindustan is for Hindus, the governor has not only tarnished the dignity of the gubernatorial post but also breached the pledge he took to ensure unity and integrity of the country."
Gogoi, in a statement, said: "Acharya has also lost his ability to serve as a governor. He has also been working more like a BJP leader and a RSS pracharak."
He also appealed to the central government to appoint "a better person with broader outlook and mindset" as governor in the state.(IANS)