The building's floors are subtly sited on the hillside in Santa Teresa. Social areas are clustered on the ground floor and give onto the Roberto Burle Marx-landscaped gardens by way of large picture windows and sliding doors. The openings, while characteristic of modern architecture, seem oversized in comparison to some of the 18th and 19th-century art objects inside. The second floor was originally bedrooms.
Taunay was one of the main artists involved in the French Artistic Mission, a group of French artists and architects who came to Rio de Janeiro in 1816. At that time, Rio was the capital city of the Kingdom of Portugal, Brazil, and the Algarves because the royal court of Portugal had been relocated to Rio in 1808 due to the invasion of Portugal by Napoleon Bonaparte. Their mission was to establish the Escola Real de Ciências, Artes e Ofícios (Royal School of Sciences, Arts and Crafts), which would later become the Escola Nacional de Belas Artes (National School of Fine Arts).
The Castro Maya collection also houses a series of watercolors of plants and fruit by Jean-Baptiste Debret, another artist from the French Artistic Mission Group. The watercolors date from 1818 to 1830.
The surrounding neighborhood of Santa Teresa is incredibly charming, with houses built into the hills, winding cobblestone streets, and a profusion of street art.