plaster is safe, but don’t breathe it. However, you are talking about water soluble items, that do NOT flow well into fabrics (they are solid particles, not a phase changing liquid that cures), and are VERY VERY heavy.
If you are so worried about your health, you came to the wrong field. While natural fibers are gaining popularity, you will always be stuck with resins that are not friendly. There are very few bio-resins out there, and even then…don’t drink it, and try not to get it on your skin.
You can use thermoplastics, but then you have to deal with heat (min 375f for low-grade soft PP and PE resins), and equipment that is needed to process it (hot press, ovens, etc)
If you want to cast objects, you can try chopping/milling natural fibers into, say plaster. It will give it overall breaking/cracking resistance.
here at UD, i believe we won the last SAMPE bridge building contest in natural fiber, using sheets of directional wood sheets, and glued them together. I think it was bamboo, or teek(sp?)…but you need to glue it together…with epoxy.