{"id":8174,"date":"2023-02-09T07:32:00","date_gmt":"2023-02-09T14:32:00","guid":{"rendered":"https:\/\/mattfife.com\/?p=8174"},"modified":"2023-02-09T10:49:42","modified_gmt":"2023-02-09T17:49:42","slug":"clipmatrix-and-volumeclip-ai-based-3d-model-generation","status":"publish","type":"post","link":"https:\/\/mattfife.com\/?p=8174","title":{"rendered":"CLIPMatrix and VolumeCLIP\u00a0AI based 3D model generation"},"content":{"rendered":"\n<p class=\"wp-block-paragraph\">As I mentioned in my previous article, there is a<a rel=\"noreferrer noopener\" href=\"https:\/\/arxiv.org\/abs\/2109.12922\" target=\"_blank\">&nbsp;paper by Nikolay Jetchev called ClipMatrix<\/a>&nbsp;that attempts to generate 3D models from 2D images that are generated by text-to-image diffusion models (like stable diffusion, DALL-E, MidJourney, etc). A <a rel=\"noreferrer noopener\" href=\"https:\/\/www.researchgate.net\/profile\/Nikolay-Jetchev\" data-type=\"URL\" data-id=\"https:\/\/www.researchgate.net\/profile\/Nikolay-Jetchev\" target=\"_blank\">list of his other papers can be found here<\/a>.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">He now seems to be working on auto-generated models that are animated automatically. (Content note: he seems to love to generate content based on heavy metal lyrics, demons, and other fantastical creations that I don&#8217;t think demonstrate this could work on more &#8216;normal&#8217; looking models):<\/p>\n\n\n\n<figure class=\"wp-block-video\"><video controls src=\"https:\/\/video.twimg.com\/ext_tw_video\/1501927963717083136\/pu\/vid\/720x720\/1C0gX_YH91wmvj2r.mp4?tag=12\"><\/video><\/figure>\n\n\n\n<p class=\"wp-block-paragraph\"><em>Originally tweeted by Nikolay Jetchev (<a href=\"https:\/\/twitter.com\/NJetchev\">@NJetchev<\/a>) on <a href=\"https:\/\/twitter.com\/NJetchev\/status\/1501928429607809026\">March 10, 2022<\/a>.<\/em><\/p>\n\n\n\n<p class=\"wp-block-paragraph\">In looking at his <a rel=\"noreferrer noopener\" href=\"https:\/\/twitter.com\/NJetchev\" data-type=\"URL\" data-id=\"https:\/\/twitter.com\/NJetchev\" target=\"_blank\">Twitter stream<\/a>, he also seems to be working on a version called VolumeCLIP that appears to generate voxel objects he can ray-cast into..<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">&#8220;The Fire Dwarf Blacksmith&#8221;<\/p>\n\n\n\n<figure class=\"wp-block-video\"><video controls src=\"https:\/\/video.twimg.com\/ext_tw_video\/1618726963182555136\/pu\/vid\/720x720\/bsVHyC6-4L4l0a0V.mp4?tag=12\"><\/video><\/figure>\n\n\n\n<p class=\"wp-block-paragraph\"><em>Originally tweeted by Nikolay Jetchev (<a href=\"https:\/\/twitter.com\/NJetchev\">@NJetchev<\/a>) on <a href=\"https:\/\/twitter.com\/NJetchev\/status\/1618727023006064646\">January 26, 2023<\/a>.<\/em><\/p>\n","protected":false},"excerpt":{"rendered":"<p>As I mentioned in my previous article, there is a&nbsp;paper by Nikolay Jetchev called ClipMatrix&nbsp;that attempts to generate 3D models from 2D images that are generated by text-to-image diffusion models (like stable diffusion, DALL-E, MidJourney, etc). A list of his other papers can be found here. He now seems to be working on auto-generated models that are animated automatically. (Content note: he seems to love to generate content based on heavy metal lyrics, demons, and other fantastical creations that I&#8230;<\/p>\n<p class=\"read-more\"><a class=\"btn btn-default\" href=\"https:\/\/mattfife.com\/?p=8174\"> Read More<span class=\"screen-reader-text\">  Read More<\/span><\/a><\/p>\n","protected":false},"author":2,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"_jetpack_newsletter_access":"","_jetpack_dont_email_post_to_subs":false,"_jetpack_newsletter_tier_id":0,"_jetpack_memberships_contains_paywalled_content":false,"_jetpack_memberships_contains_paid_content":false,"footnotes":"","jetpack_publicize_message":"","jetpack_publicize_feature_enabled":true,"jetpack_social_post_already_shared":true,"jetpack_social_options":{"image_generator_settings":{"template":"highway","default_image_id":0,"font":"","enabled":false},"version":2},"jetpack_post_was_ever_published":false},"categories":[28,22,9],"tags":[],"class_list":["post-8174","post","type-post","status-publish","format-standard","hentry","category-ai","category-artdesign","category-cool"],"jetpack_publicize_connections":[],"jetpack_featured_media_url":"","jetpack_sharing_enabled":true,"jetpack_shortlink":"https:\/\/wp.me\/p4WECr-27Q","jetpack-related-posts":[],"_links":{"self":[{"href":"https:\/\/mattfife.com\/index.php?rest_route=\/wp\/v2\/posts\/8174","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/mattfife.com\/index.php?rest_route=\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/mattfife.com\/index.php?rest_route=\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/mattfife.com\/index.php?rest_route=\/wp\/v2\/users\/2"}],"replies":[{"embeddable":true,"href":"https:\/\/mattfife.com\/index.php?rest_route=%2Fwp%2Fv2%2Fcomments&post=8174"}],"version-history":[{"count":3,"href":"https:\/\/mattfife.com\/index.php?rest_route=\/wp\/v2\/posts\/8174\/revisions"}],"predecessor-version":[{"id":8177,"href":"https:\/\/mattfife.com\/index.php?rest_route=\/wp\/v2\/posts\/8174\/revisions\/8177"}],"wp:attachment":[{"href":"https:\/\/mattfife.com\/index.php?rest_route=%2Fwp%2Fv2%2Fmedia&parent=8174"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/mattfife.com\/index.php?rest_route=%2Fwp%2Fv2%2Fcategories&post=8174"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/mattfife.com\/index.php?rest_route=%2Fwp%2Fv2%2Ftags&post=8174"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}