None of the Founding Fathers were atheists. Most of the Founders were Deists, which is to say they thought the universe had a creator, but that he does not concern himself with the daily lives of humans, and does not directly communicate with humans, either by revelation or by sacred books. They spoke often of God, but this was not the God of the bible. They did not deny that there was a person called Jesus, and praised him for his benevolent teachings, but they flatly denied his divinity.
If the U.S. was founded on the Christian religion, the Constitution would clearly say so -- but it does not. Nowhere does the Constitution say: "The United States is a Christian Nation", or anything even close to that. In fact, the words "Jesus Christ, Christianity, Bible, Creator, Divine, and God" are never mentioned in the Constitution-- not even once. When the Founders wrote the nation's Constitution, they specified that "no religious test shall ever be required as a qualification to any office or public trust under the United States." This provision was radical in its day-- giving equal citizenship to believers and non-believers alike. They wanted to ensure that no religion could make the claim of being the official, national religion, such as England had.
Christian Revisionists often attempt to rewrite history, in much the same way as holocaust deniers do. The men responsible for building the foundation of the United States were men of The Enlightenment, not men of Christianity. They were Deists who did not believe the bible was necessarily true. They were Freethinkers who relied on their reason, not their faith.
And they accepted that even those who did not believe in a Supreme Being were qualified to hold office.