Sultan Muhammad Shah Aga Khan III built Aga Khan Palace in 1892 and is one of the important landmarks in the history of India. This is a stately building that has close relationship with the Indian freedom struggle. This was where once Mahatma Gandhi and his wife Kasturba Gandhi and Sarojini Naidu were imprisoned. In 2003 it came under the control of Archaeological Survey of India and was declared as a monument.

The Aga Khan Palace spreading over an area of about 19 acres of land is noted for its Italian arches and spacious lawns and the building has five halls. Gandhi Smarak Samittee got this palace in 1972 from Prince Karim Aga Khan as a donation. Gandhi National Memorial Society has its headquarters at this place.