Apr 2, 2024
3D generative models are rapidly improving but do not respect artistic and workflow constraints. These models also hallucinate unseen regions due to lack of constraints. For instance, concept artists typically produce the front, back and sides of objects before sending them to the 3D modelling team. We built a fundamentally new approach to generate 3D assets from sheet images - consisting of front and back images. This novel workflow aims to better represent a 3D character or object, as shown below. Now concept artists can rapidly visualise and generate 3D models, right from within the conception stage, before sending them to production. Creators can also generate sheets from text inputs or click 2 images to generate assets with better realism and control.
Cube's Sheet-to-3D takes two input images rather than a single image as most existing image-to-3D methods use. Both images are then fused to generate the 3D asset. This addresses the limitation of image-to-3D that it can only speculate about the geometry and appearance of an object's backside. In contrast, Sheet-to-3D ensures the back image is integrated into the final output with comparable quality as the front image. The video below highlights the advantages of the new Sheet-to-3D workflow over conventional single image-to-3D methods.
More conveniently, Cube delivers an AI-assisted text-to-sheet to automatically generate these character sheets, in addition to uploading sheets owned by the user. Cube also provides an interactive segmentation tool to label the front and back segments from the sheets (Check out the demo video).
Similar to classic animation character sheets, the front and back views do not need to be exact. This means camera parameters and even character poses can differ. As long as both views roughly correspond, Cube's Sheet-to-3D will automatically handle variations and fuse them into a consistent model as accurately as possible. Interestingly, we've found that Front & Back-to-3D is capable of producing textured meshes of often higher quality—or at least equal to—that of single image-to-3D results. We believe this could be indicative of the benefits of utilising more information.
Wondering how effective is it? Sign up and give it a try!