Story I heard was similar to what CB said but a little different.

A group of kids would all smoke pot after school at 4:20pm by a statue at/near their school and eventually it caught on to the mainstream public, as to how it applies to the actual date (April 20th) your guess is as good as mine.