How much of this is Excel programming and how much is generic math/geometry/engineering? It looks to me like you are describing basic vector addition. Excel really doesn't have any built in functions for vector addition, but, if you understand the math/geometry well enough, we should be able to come up with formulas that will perform the addition.

Sometimes, I see people use complex numbers to perform the addition. There can be advantages to this. In this particular case, once you've expressed each current as a complex number, then the addition is a simple IMSUM() function. If needed, an IMSUB() function can get the neutral leg vector. Once you have the neutral vector expressed as a complex number, then the length of that vector is an IMABS() function. The main difficulty to using complex numbers for this is taking your r,theta expressions that you currently have and converting them to x+yi values (which is mostly a trig problem).

I obviously have only thought through this superficially. Is there a specific part of this that you need programming help with? Or do you need some help understanding the geometry/trigonometry/engineering side of the problem before getting to the programming side of it?