On Saturdays , however , when only the responsible people , the presidential appointees , came to work in the White House and the Executive Office Building , casual dress was the uniform .
No one is allowed to work in the White House during weekends .