Closest thing I could find is some professor (Burke?) stating that he believes they smoked it, but it is "open to debate".
Did the Founding Fathers of the United States of America smoke cannabis? Some researchers think so. Dr. Burke, president of the American Historical Reference Society and a consultant for the Smithsonian Institute, counted seven early presidents as cannabis smokers: George Washington, Thomas Jefferson, James Madison, James Monroe, Andrew Jackson, Zachary Taylor and Franklin Pierce. 41 "Early letters from our founding fathers refer to the pleasures of hemp smoking," said Burke. Pierce, Taylor and Jackson, all military men, smoked it with their troops. Cannabis was twice as popular among American soldiers in the Mexican War as in Vietnam: Pierce wrote to his family that it was "about the only good thing" about that war.
Portland NORML - Hemp: Lifeline to the Future - American High Society

Since I haven't read the early letters from our founding fathers, I have no way to independently verify that. Since the source is NORML, independently verifying it would be prudent.