Two were rescued and workers were in communication with two other trapped people.
Photos showed the top two stories sprawled across and onto the other side of a narrow street, with electricity wires pulled down by the building.
Police and firefighters rushed to a bridge collapse on a two-lane road in what appeared to be a rural part of Yuli where one or more vehicles may have fallen off, the Central News Agency said.
A canopy on a platform at Dongli station in Fuli town, which is between Yuli and the quake's epicenter at Chishang, hit a train, leaving three of the cars tilted at an angle, the agency said, citing the railway administration. The shaking was felt at the north end of the island in the capital, Taipei.
The Japan Meteorological Agency issued a tsunami advisory for several southern Japanese islands near Taiwan, but later lifted it.
