If you want to persue with the square design, I suggest a radius of 3mm (1/8") minimum. This should be enough to relieve some of the stress encountered in the corners.
Make sure you use a tapered reinforcing stack, depending on the maximum local loads. Vang attachment and sheet are the major stress points, where you need to beef things up.
I suggest attaching these with stainless hoops (I guess Sea-Sure has the right ones) and GLUEING these on, to spread the load. Glue should be a stiff methacrylate, like Plexus MA300 or MA310 (suggesting MA310 for longer working time). Minimum glue thickness is 1mm, keep that in mind. This will spread the load evenly over the complete profile, not concentrated on just a couple of bolts / rivets.
If the boom has no track, you can attach the vang and sheet with woven band, for the sheet rivited to the top of the profile, for the vang crossed over the top, then rivited to the bottom. Topper Boss style.
I sailed one of those, and these had round carbon booms, utterly light. Breaking a boom was rare, though.
Another factor that you will encounter is that if you use vacuum or any other compression method, you will see that the fibers will “crumble up” in the corners, compromising strength. Perhaps tightly compressing the complete stack with a UD at 90 degrees every couple of layers will prevent this.
Also make sure your UDs are encapsulated in biax completely, or they will break (buckle) in compression. Preferably keep the 90degree fiber away from the UD as much as possible.
So basicly you are looking at the following schedule: (from inside out)
-woven 200gr/m2
perhaps multiple series (2 or 3) of:
-biax 45
-UD top and bottom
-biax 45
-90 UD for compression
-biax 45
-UD top and bottom
and finish with:
-biax 45
-90 UD for compression
-woven 200 gr/m2
Of course all depends on the weight of the materials used, Perhaps you can make everything from UD fibers. Quite some work to layup (if layers are different angle left/right the thing will twist during cure)
woven 200 gr is a nice finish, and will not see protruding fibers. you could also use a woven glass, which will show the UD behind it.
Heat as much as possible during cure (80 degrees or even more would be nice) and use vacuum, and you will end up with a nice lightweight boom. From experience I know that hitting a lightweight carbon boom with the head is less painful than an aluminium one. Perhaps the round section had something to do with it as well.