If you want to see how loud a pair of stereo speakers can get, you turn the volume knob up as far as you can. Similarly, if you want to see how quickly your car can speed up, you might push the gas pedal down as hard as you can. These are both examples of "testing the limits" of something.
The "limit" of something is be largest, strongest, farthest, hottest, etc. possible. When you "test the limits" of something, you're trying to find out what the limit is.
Here are some expressions with "test the limits":
test the limits of my endurance
test the limits of human potential
test the limits of the law
test the limits of credulity
When people (or things) are really annoying, you can say that they're "testing the limits of your patience".
It's like you're imagining that the annoying things are trying to be as annoying as possible, so that they can find out what will finally make you angry.