According to Brazilian News Weekly, Temer surprised many people when he announced that he has left the presidential palace.
Alongside his former beauty queen wife and seven-year-old son, he relocated down the road to the residence of the vice-president.
The modernist Alvorada, which means Dawn and was designed by Brazilian architect Oscar Niemeyer, would be a dream home for many.
It has a huge pool, football field, chapel, medical Centre and vast lawn.
But Temer,76, and his 33-year-old wife Marcela, find the cavernous, glass-fronted building spooky.
“I felt something strange there. I wasn’t able to sleep right from the first night,” he was quoted as saying.
“The energy wasn’t good.”
The newspaper said Marcela felt the same thing, but only Michelzinho (their son), who went running from one end to the other, liked it.
“We even started to wonder: could there be ghosts?” he reportedly quipped to a newspaper.
According to a report in Globo newspaper, Marcela brought in a priest to attempt to drive out any evil spirits, but to no avail.
The Temers then moved to the still luxurious but smaller Jaburu Palace nearby.
Temer moved into the presidential quarters after succeeding Dilma Rousseff, the immediate past president, who was impeached for breaking budget accounting laws.
Before then, he was staying in the place he has just moved into.
The house moving comes in the middle of a severe political crisis for Brazil, with many of Temer’s allies likely to face corruption probes.
The president himself is battling a case in the electoral court where he is accused of having benefited from illegal donations when he and Rousseff ran together in 2014.