Sheriff's investigators said Sloane Briles, 35, was on a tour boat Sunday with his girlfriend and two sons from a previous marriage.
Witnesses told them Briles got into an argument with his girlfriend, which upset the boy. Police say when the boy wouldn't calm down, Briles threw him overboard.
"The chance of him being struck by one of the boats passing by if they didn't see him, would be one of our greatest concerns," said Dep. William Nelson of the Orange County Sheriff's Department.
Tapes from 911 calls made by witnesses have been released.
"Listen, I'm on a boat tour called the Queen and there's a man that just threw his son overboard," one witness can be heard saying. "We're out in the middle of the bay. He just threw him over."
Investigators said Briles may have been drinking.
"He's drunk and violent," one woman said about Briles. "We need somebody immediately."
Briles was charged with child endangerment and resisting arrest. The boy and his brother were returned to their mother's care.