The storm, which weakened to a Category 1 hurricane, made its official landfall just after 7:30 a.m. ET as it passed over North Carolina's Outer Banks. The center of the storm was estimated to be some 500 miles wide, tracing the East Coast from Myrtle Beach, S.C. to just below Cape Cod.
Forecasters said Irene's effects could be felt as far north as Canada even after it weakens. A tropical storm warning extended from the U.S. border to Nova Scotia's southern coast.
Two million homes and businesses were without power early Sunday.
Irene has already been linked to at least eight deaths.
In North Carolina, authorities said a large tree limb struck a man and killed him. In the second incident, a man was putting plywood over the windows of his home when he suffered a heart attack and died, according to the News & Observer. Also, an 11-year-old boy died in a crash at an intersection where traffic lights were out.
Two other deaths were reported in Virginia. In Newport News, a city spokeswoman said an 11-year-old boy was killed when Irene's 60 mph winds sent a tree crashing through his apartment's roof. In Brunswick County, a tree reportedly fell on a car and killed someone inside.
In Florida, a surfer and a beachgoer were killed in heavy waves.
A New Jersey tourist died when he drowned in the rough surf.
More than 2 million people in Maryland, New Jersey, North Carolina, Virginia and Delaware were told to move to safer places.
"Don't wait. Don't delay," said President Barack Obama, who decided to cut short his summer vacation by a day and return to Washington. "I cannot stress this highly enough: If you are in the projected path of this hurricane, you have to take precautions now."
Obama so far had declared emergencies for North Carolina, Virginia, New Jersey, New York, Connecticut and Massachusetts.
Shopping centers were deserted, streets were empty and hospitals weren't responding to emergency calls - but that's just how officials wanted it. They said it becomes too dangerous to send rescue crews out once the storm arrives, and even in a weakened state, the dangers of any hurricane are very real.
The National Weather Service said Saturday evening that winds were at 80 mph, down from 85 mph, but with higher gusts. The eye appeared to be right at the border of North Carolina and Virginia as it trekked its way to the Northeast.
The surf generated by the storm is a potential danger in and of itself. The power of the water was on display in Florida, when a crashing wave injured a group of onlookers standing on a pier.
Forecasters said Irene's center would roll up the mid-Atlantic coast Saturday night and over southern New England on Sunday.
Carolinas
Hurricane-force winds first hit Jacksonville, N.C. at about 6:15 a.m. Saturday. About an hour later, the storm's center passed near the southern tip of North Carolina's Outer Banks. At Atlantic Beach, part of a pier collapsed, falling into the ocean.
The eye of the storm is typically calm, but the storm's wind and rain are far from over. Forecasters said the landfall has little significance, as Irene remains a dangerous storm.
Eastern North Carolina got 10 to 14 inches of rain, according to the National Weather Service.
Wind and rain knocked out power to more than 210,000 customers along the North Carolina coast. More than 60 shelters have been opened in 26 counties.
In South Carolina, some viewed the hurricane conditions as an opportunity to go surfing. Several surfers were seen jumping off a pier to catch the big waves.
New York
New York Mayor Michael Bloomberg announced on Saturday night that the edge of Irene had reached the city and it was no longer safe to be outside
New York City ordered more than 370,000 people who live in flood-prone areas to leave, including Battery Park City in Lower Manhattan, Coney Island and the beachfront Rockaways. But it was not clear how many would do it, how they would get out or where they would go. Most New Yorkers don't have a car.
Officials hoped most residents would stay with family and friends, and for the rest, the city opened nearly 100 shelters with a capacity of 71,000 people. Bloomberg said at a Saturday afternoon press conference that 1,400 people are staying in those shelters, but it wasn't immediately clear how many people are defying the evacuation order.
The mayor also warned people that time was running out to follow the mandatory evacuation for Irene. He told holdouts that if they didn't leave, they're putting rescue workers at risk, too.
If the storm stays on its path, skyscraper windows could shatter, tree limbs would fall and debris could be tossed around. Streets in the southern tip of the city could be swimming under several feet of water. Police readied rescue boats, but they said they wouldn't' go out if conditions were poor.
Saturday, New York City's biggest utility Consolidated Edison announced it could cut power to the city's most vulnerable areas if Hurricane Irene brings serious flooding. A Con Ed spokesman said the utility doesn't expect to cut power before the storm, but flooding could bring a shutdown to areas including Lower Manhattan and parts of the West Village Sunday.
The city began shutting down the subways and buses around noon Saturday, and transit officials said it would take about eight hours before the entire system was shuttered.
The transit system carries about 5 million people on an average weekday, fewer on weekends. This marks the first time the city has shut down the entire subway system because of a natural disaster.
Trains and busses won't be back up and running until at least Monday, after pumps remove water from flooded subway stations. Even on a dry day, 13 million to 15 million gallons of water are removed from the tunnels deep underground.
Sporting events, concerts and even Broadway were going dark. The Braves-Mets games over the weekend were postponed, and all Broadway musicals and plays were canceled for the weekend as well.
Irene affects travel plans
The fallout from Hurricane Irene is even having an impact in Los Angeles as thousands of flights in and out of the East Coast are canceled.
At Los Angeles International Airport, passengers scrambled to get aboard the remaining flights to New York City, Washington, D.C., Boston and other hubs.
Gerhard and Liz Klarwasser were hoping to beat Irene back home to Boston, but the powerful storm stranded them at LAX.
"We thought it would come a little later so we could sneak in Sunday morning, but no such thing," Liz Klarwasser said.
Airlines have already scrapped more than 9,000 flights from North Carolina to Boston in anticipation that bad weather will shut down over two dozen East Coast airports. There were 3,900 cancellations on Saturday alone.
The five main New York-area airports - La Guardia, John F. Kennedy and Newark, plus two smaller ones - waved in their last arriving flights around noon. Many departures were also canceled.
Airlines are waiving rebooking fees for passengers who want to delay their flights. Regular schedules should resume Tuesday.
Impact on Southern California
The impact of Hurricane Irene could be felt in another way in Southern California - it could lead to higher gas prices.
Some East Coast refineries have already begun a partial shutdown of operations. If they shut down completely, it would take several days or even weeks to get up and running again.
While our gas does not come from those refineries, a shutdown would reduce overall supplies, and that could mean higher prices for everyone.
The Associated Press contributed to this report.
