Andy,
This is when your good judgement will be necesasry. You must consider the rules that apply, and the INTENT of the players involved. I would allow the OOT bettor to retract his bet as long as substantial action has not taken place. Were there other players involved? If there were, did a player react to the OOT bet? Based on the situation the way I understand it, (with no other players), the OOT bettor may retract his bet and his hand is DEAD because he tossed it in the direction of the muck. Whether there is a betting line or not is irrelevant. His action indicated his intention to fold, and the hand should have been "killed" by the dealer instantly.
To answer the original question, I have to agree with Chet. I think the floor was wrong. If the floor decided to force the player to keep his bet in the pot, he should have at least had the option to call. Forcing a player to put chips in the pot, when that was not his intent and then killing the hand, is a bad call, IMO.