It's been on coins such as the silver and gold dollars since the mid 1800's. I don't think anyone was much worried about godless communists around at that time.
The picture below is taken from the east entrance to the Supreme Court. It depicts Moses (as the central figure), Confucius, Solon and additional allegorical characters.For the religious symbols in federal buildings, I'd need more concrete
examples... along with a chance to understand if they were religious symbols or
symbols that had been usurped by a religion.
![]()