The incident happened just after 11:30 a.m. near the intersection of Broadway and Prince in SoHo.
Pictures from the scene show wooden planks all over the street, and FDNY firefighters at the scene.
FDNY officials said "we're absolutely" lucky there aren't more injured in the busy neighborhood. The scaffolding collapsed over a subway entrance, but no train had let out at the time.
Investigators said strong wind is to blame for the collapse. A piece of plywood "acted like a sail" and blew the whole rig down.
Officials said all all five injuries are minor and not life-threatening.