Programmatically dealing with embeded 3d

First time poster long time lurker,
I was hoping someone could steer me in the right direction for an upcoming upgrade I wanted to add to an app. I don’t necessarily need specific coding advice (at this time) but simply a place to start looking.
So I wanted to try my hand at 3d. I have an app that I would like to add a 3D feature to it so that I can programmatically construct a 3d image and give the user the ability to perform basic manipulations (like rotations, zooms etc…). Ideally I would love an easy library or perhaps even better a free API/SDK that allows me to embed a 3d matrix into my app. Personally I would probably fart a rainbow if I could find one that accepts a large array of 3d numbers and assigns it to a 3d matrix so something like
0,0,1,”blank”
0,0,2,”blank”
0,0,3,”red”
But I’m sure it won’t be that easy.
I know you can use directX and open GL to display pre existing models (say constructed in blender or whatever) but I need something that I can construct within my app because I will need to manipulate the array significantly.
Any suggestions?
Thanks