The 48-year-old man was last spotted wearing a pair of blue biker shorts, and did not return to shore when he was expected.
The Coast Guard has deployed both a ship and a helicopter from Point Mugu to help in the search.
The agency posted a picture of the man on Twitter but did not identify him by name.
Anyone who might know the man's location is urged to call authorities at (310) 521-3801.