Firstly, if you aren’t sure what I am talking about, I would recommend having a read of this article by Chris Coyier.
Alright, now to the formula that I use to make sure that I get my nested border radii just right: x = z — y, where x is the inner border radius, z is the outer border radius, and y is the space between the two elements (as shown above). Keep in mind that for this formula to work, both pseudo circles formed by the border radii need to have the same centre point.
Now, unlike most mathematical formulas, this one isn’t always right. You may encounter the odd situation where the result doesn’t look quite right, so keep in mind that this is only a guide; you may need to eyeball it to get it perfect.
Here are a few examples of the formula in action (as you can see, the top right has been adjusted slightly):