Plus waiting in line,
that is. This of course depends on the point of entry and traffic of the time. It was about 15-20 minutes for me last time. I think that is the pretty much the average.
It wouldn't be a couple of hours, since a lot of Canadians and Americans have to cross the border to work or do their business everyday.