A crowd of about 100 Libyans greeted him on a visit to the city's freedom square.
McCain is the top Republican on the Senate Armed Services Committee and has been an advocate of U.S. intervention to remove leader Moammar Gadhafi from power.
McCain said the United States and other nations should recognize the opposition's political leadership as the "legitimate voice of the Libyan people." The White House disagreed, saying it was for the Libyan people to decide who their leaders are.
McCain's trip comes as President Barack Obama authorized armed Predator drones to be used in airstrikes against forces loyal to Gadhafi. It will be the first time the drones will be used since the U.S. turned over control of the Libya operation to NATO on April 4.
The rebels have complained that NATO airstrikes since then have largely been ineffective in stopping Gadhafi forces.
In the meantime, a senior Libyan government official said the Libyan army will pull out of the besieged, rebel-held city of Misrata and be replaced by armed tribesmen.
Deputy Foreign Minister Khaled Kaim did not say when the military would withdraw and under what conditions.
The Associated Press contributed to this report.