Spaces:
Running
on
Zero
3D scenery from video
Something I'm hoping will come to pass (and this seems like the place to start) is taking a video clip, algorithmically removing all the moving parts (actors, etc.), and then making 3D from the rest. You almost could get a sort of "poor-man's photogrammetry" if you had video of, say, a set without any people in it, which you could split to still frames and feed into a photogrammetry app, but AI could, SHOULD, be able to do this better and with less.
As a for instance, imagine taking a scene from Star Trek (any, but let's say TOS), set on the bridge. It seems to be it should be possible to feed this all into a neural network, have it figure out what's moving (instead of changing because the camera moves) and thus remove Spock, Kirk, Uhura, Checkov, Sulu, Yeoman Rand coming to get the duty roster signed off on, Bones walking in to say something random, crewman number whatever sitting at a console without defined purpose, etc. Remove those things, and then, with multiple views (as the camera moves), give us a nicer, higher res "scan" of the bridge.
Is that a pipe dream? It "feels" achievable.
it is sort of a thing already https://github.com/threestudio-project/threestudio but it takes a lot of VRAM to run it you are pretty much training a model based on the generated data regardless this Blender addon for Shap-e is really cobiniant take a look https://devbud.gumroad.com/l/Shap-e