"Passing" the card off to a neighbouring player does not even have to be intentional. The card may have been dealt, but just not picked up by the player, and then it gets mixed into the muck.
I'm not making this up -- this used to be a known, documented issue.
In theory, I am not against giving him the 2nd card down, but you would have to know for sure that it really is his card, and that the card didn't somehow end up in the muck, or under the rail, or somewhere else. You have to be sure that the card on the top of the deck actually does belong to the button player. Otherwise, you'd be giving him the burn card, and you'd be throwing everything off that way anyway. Therefore, the only way to be sure of that is to count the deck, which is a huge time waster. I believe that asking the player to speak up before action (or substantial action) has occurred after the deal is a reasonable compromise.
The other issue is optics... the action goes around, there's a raise, re-raise, all-in, fold, fold, fold, and NOW the button says he wants his card. You don't think this is going to look super-fishy to the players who have already acted? There is going to be suspicion of foul play even if there was none. When so much action has occurred, if it's too late to call a misdeal, it should also be too late to "correct" a dealing error like this one.