Keep in mind that my highway speeds were in the 80s. But I have had a few scary moments on uneven pavement at lower speeds, especially where they're repaving this one particular stretch of a two-lane road that I use frequently.
Then there's this street coming into downtown Houston that slants to the left and has a groove running parallel to the dividing line. I was on that groove running at about 40 mph when the traffic light went red on me quickly. When I hit the brakes (hard), that groove and the slant pulled me into the lane to my left. Thankfully, no one was in that lane.
Caution is key.