Browsed by
Category: Technical

Install Stable Dreamfusion on Windows

Install Stable Dreamfusion on Windows

Comments 5 comments

I wrote about Stable Dreamfusion previously. Dreamfusion first takes normal Stable Diffusion text prompts to generate 2D images of the desired object. Stable Dreamfusion then uses those 2D images to generate 3D meshes.

A hamburger

The authors seemed to be using A100 nVidia cards on an Ubuntu system. I wanted to see if I could get this to work locally on my home Windows PC, and found that I could do so.

System configuration I am using for this tutorial:

  • nVidia GeForce GTX 3090
  • Intel 12th gen processor
  • Windows 10

Setting Stable Dreamfusion up locally:

Step 1: Update your Windows and drivers

  1. Update Windows
  2. Ensure you have the latest nVidia driver installed.

Step 2: Install Windows Subsystem for Linux (WSL)

  1. Install Windows Subsystem for Linux (WSL). WSL install is a simple command line install. You’ll need to reboot after you install. You want to make sure you install Ubuntu 22.04, which is the default in Feb 2023 since that is what Stable Dreamfusion likes. Currently WSL installs the latest Ubuntu distro by default, so this works:
    wsl –install
    If you want to make sure you get Ubuntu 22.04, use this command line:
    wsl –install -d Ubuntu-22.04
  2. After installing WSL, Windows will ask to reboot.
  3. Upon reboot, the WSL will complete installation and ask you to create a user account.
  4. Start Ubuntu 22.04 on WSL by clicking on the Windows Start menu and typing ‘Ubuntu’ or you can type Ubuntu at a command prompt and type ‘Ubuntu’.

Step 2b (optional): Install Ubuntu wherever you want on your Windows system. By default it installs the image on your C:\Users directory – which is kind of annoying.

Step 3: Install dependent packages on Ubuntu

  1. If you don’t have Ubuntu started, go ahead and start Ubuntu 22.04 on WSL by clicking on the Windows Start menu and typing ‘Ubuntu’ (or you can type Ubuntu at a command prompt as well). A new shell terminal should appear.
  2. You need to install the nVidia CUDA SDK on Ubuntu. You’ll choose one of these two options:
    • You will then get a set of install instructions at the bottom of the page (wget, apt-get, etc). Simply copy the lines one by one and put them into your Ubuntu terminal. Ensure each step passes without errors before continuing.
    • The ‘sudo apt-get -y install cuda’ line will install a lot of packages. It can take 10-15 minutes.
  3. Install python3 pip. This is required for the Dreamfusion requirements installation script.
    • sudo apt install python3-pip

Step 4: Install Stable Dreamfusion and dependent packages

  1. You should now follow the install instructions found on the Dreamfusion page.
  2. Clone the project as directed: git clone https://github.com/ashawkey/stable-dreamfusion.git
  3. Install with PIP: Install the pre-requisites via pip as directed on the Dreamfusion github page:
    • pip install -r requirements.txt
    • I also installed both optional packages nvdiffrast and CLIP.
    • Add this export line to your .bashrc to ensure python can find libcudnn:
      export LD_LIBRARY_PATH=/usr/lib/wsl/lib:$LD_LIBRARY_PATH
  4. I did not install the build extension options
  5. Exit and restart your shell so that all path changes take effect

Step 5: Run a workload!

Follow the instructions in the USAGE section of the Dreamfusion instructions. Instead of ‘python’ use ‘python3’. They have a number of things you can specify like negative prompts, using the GUI interface (which does not work under WSL),

The very first run will take a long time. It will download several gigabytes of training data, then train 100 epoch’s, which can take up to an hour.

$> python3 main.py --text "a hamburger" --workspace trial -O 
$> python3 main.py --text "a hamburger" --workspace trial -O --sd_version 1.5 
$> python3 main.py --workspace trial -O --test 
$> python3 main.py --workspace trial -O --test --save_mesh

Check Your Output:

Look in the results directory under the workspace name:

.\stable-dreamfusion\<workspace name>\mesh\ #directory holds the .obj, .mat, and .png files
.\stable-dreamfusion\<workspace name>\results\ #directory holds a mp4 video that shows the object rotating

Copying them to Windows:
All Windows drives are pre-mounted in \mnt\<drive letter>\ for WSL.
Ex: \mnt\c\
So you can copy the output files to your windows side by doing:
cp -rP .\<workspace name> \mnt\c\workdir\

Looking at the generated meshes with materials:

  1. Install Blender
  2. File->Import->Wavefront (.obj) (legacy)
  3. Or, use 3D Viewer (though it seems to have issues with material loading at times)

Fixes:

  1. You might get an error about missing libcudnn_cnn_infer.so.8
==> Start Training trial Epoch 1, lr=0.050000 …
0% 0/100 [00:00<?, ?it/s]Could not load library libcudnn_cnn_infer.so.8. Error: libcuda.so: cannot open shared object file: No such file or directory

add this to your .bashrc to ensure it can find libcudnn:
export LD_LIBRARY_PATH=/usr/lib/wsl/lib:$LD_LIBRARY_PATH

2. If you load the object in Blender but it doesn’t load the texture maps, try Alt-Z

Links:

Comparing AI art generators for common artist workloads

Comparing AI art generators for common artist workloads

Comments 0 Comment

Gamefromscratch decides to do side-by-side results tests on DALL-E 2, Stable Diffusion, and MidJourney for a variety of art generation tasks.

His conclusion is they are not going to replace artist for all tasks, but for concept art, pixel art, and some other simple tasks these AI generators can replace artists.

Leaking corporate code via chatGPT

Leaking corporate code via chatGPT

Comments 0 Comment

After catching snippets of text generated by OpenAI’s powerful ChatGPT tool that looked a lot like company secrets, Amazon is now trying to head its employees off from leaking anything else to the algorithm.

This issue seems to have come to a head recently because Amazon staffers and other tech workers throughout the industry have begun using ChatGPT as a “coding assistant” of sorts to help them write or improve strings of code, the report notes.

While this isn’t necessarily a problem from a proprietary data perspective, it’s a different story when employees start using the AI to improve upon existing internal code — which is already happening, according to the lawyer.

Learning Quaternions

Learning Quaternions

Comments 0 Comment

3blue1brown makes lots of good videos on mathematics. One of those videos is how to visualize and understand quaternions.

Quaternions are a higher dimension system that can be used to describe 2D and 3D rotations. How they work, however, is often much harder to understand and more complex than understanding simple matrix rotations.

They made a very good video on the subject, but it required me to stop a lot and spend time thinking. These are complex concepts and almost more complex to visualize or conceptualize in your mind.

What’s nice is there is a written page that goes over these concept as well at https://eater.net/quaternions. I found this was much easier to digest than a fast-running video.

Also, if you want to play with the visualization in realtime, they even have a super-cool tool that lets you play with Quaternions in 2D, 3D, and 4D:

https://eater.net/quaternions/video/rotation

nVidia GPU’s top the Stable Diffusion performance charts

nVidia GPU’s top the Stable Diffusion performance charts

Comments 0 Comment

Toms Hardware did a great benchmarking test on which GPU’s do the best on Stable Diffusion.

They tried a number of different combinations and experiments such as changing the sampling algorithms (though they didn’t make much difference in performance), output size, etc. I wish, however, they discussed and compared the differences in memory sizes on these cards more clearly. Stable Diffusion is a memory hog, and having more memory definitely helps. They also didn’t check any of the ‘optimized models’ that allow you to run stable diffusion on as little as 4GB of VRAM.

There were some fun anomalies – like the RTX 2080 Ti often outperforming the RTX 3080 Ti.

AMD and Intel cards seem to be leaving a lot of performance on the table because their hardware should be able to do better than it is currently doing. Arc GPU’s matrix cores should provide similar performance to the RTX 3060 Ti and RX 7900 XTX, give or take, with the A380 down around the RX 6800. In practice, Arc GPUs are nowhere near those marks. This doesn’t shock me personally since nVidia has been much more invested and in the forefront of developing and optimizing AI libraries.

My heads are gone!

My heads are gone!

Comments 0 Comment

Are you losing the heads of those images you’re generating in stable diffusion?

Try adding these keywords to your prompt:

  • “A view of”
  • “A scene of”
  • “Viewed from a distance”
  • “Standing on a “
  • “longshot”, “full shot”, “wideshot”, “extreme wide shot”, “full body”
  • start the prompt with “Head, face, eyes”
  • Try adjusting the aspect ratio of the image to be taller instead of wider. Be careful not to go too tall (or two wide) or you’ll get the double-head or start generating combinations of two people.
  • The source material has been scanned in a taller aspect ratio, try adjusting the x-side of your ratio
  • Use img2img on a crop that includes part of the chest to make it match the rest of the drawing
  • Cinematography terms tend to work well. In order of close to far: Extreme close-up, close-up, medium close-up, medium shot, medium full shot, full shot, long shot, extreme long shot.

Links:

Stable diffusion high quality prompt thought process

Stable diffusion high quality prompt thought process

Comments 0 Comment

Content warning: Some of the links have some moderately NSFW pictures. There is no outright nudity, but it does deal with generating rather busty images. This article should be fine, but be aware following the links.

While this guide is translated from a Japanese source and uses the Waifu/Danbooru model to generate more anime-looking models, it works really well for generating ultra-realistic Victorian pictures using stable diffusion’s standard 1.5 model. Here’s some I made using his prompt with just 30 minutes of experimenting:

Fair warning, the original author is trying to generate more…busty women that look better as anime characters under the Waifu model. I won’t comment on his original purpose, but I thought this was an interesting description of how a ‘prompt engineer’ moved from an idea to generating a stable diffusion prompt.

First he started with a good description of what he wanted:

I want a VICTORIAN GIRL in a style of OIL PAINTING
Eye and Face are important in art so she must have PERFECT FACE, SEXY FACE and her eye have DETAILED PUPILS
I want she to have LARGE BREAST, TONED ABS and THICK THIGH.
She must look FEMININE doing EVOCATIVE POSE, SMIRK and FULL BODY wearing NIGHT GOWN
The output must be INTRICATE, HIGH DETAIL, SHARP
And in the style of {I’m not give out the artist names to avoid trouble. Apologize.}

This lead him to generate the following prompt. Note his use of parenthesize () to add emphasis, and terms inside square brackets [] to minimize the direction.

Prompt :
VICTORIAN GIRL,FEMININE,((PERFECT FACE)),((SEXY FACE)),((DETAILED PUPILS)).(ARTIST),ARTIST,ARTIST,(ARTIST). OIL PAINTING. (((LARGE BREAST)),((TONED ABS)),(THICK THIGH).EVOCATIVE POSE, SMIRK,LOOK AT VIEWER, ((BLOUSE)).(INTRICATE),(HIGH DETAIL),SHARP

Unfortunately, you don’t need to experiment for long to realize stable diffusion needs a lot of help with anatomy. It often generates nightmare fuel images that that have multiple heads, messed up arms, hands with too many fingers, eyes with terrifying pupils (or no pupils), too many limbs – well, you get the idea. So you need to make sure those things don’t show up by banning them via setting the negative prompts (again, not commenting on original purpose):

Negative Prompt :
((nipple)), ((((ugly)))), (((duplicate))), ((morbid)), ((mutilated)), (((tranny))), (((trans))), (((transsexual))), (hermaphrodite), [out of frame], extra fingers, mutated hands, ((poorly drawn hands)), ((poorly drawn face)), (((mutation))), (((deformed))), ((ugly)), blurry, ((bad anatomy)), (((bad proportions))), ((extra limbs)), cloned face, (((disfigured))). (((more than 2 nipples))). [[[adult]]], out of frame, ugly, extra limbs, (bad anatomy), gross proportions, (malformed limbs), ((missing arms)), ((missing legs)), (((extra arms))), (((extra legs))), mutated hands, (fused fingers), (too many fingers), (((long neck)))

Finally, he made these settings to the stable diffusion settings. Note you want to keep the aspect ratio in a portrait-like format (768 tall x 512 wide). Going taller can result in multiple heads, going wider can result in more than one person in the scene.

Restore Face: ON
Steps: 42
Sampler: DDIM
CFG scale: 10
Height: 768
Width: 512

Links:

Expanding and enhancing Stable Diffusion with specialized models

Expanding and enhancing Stable Diffusion with specialized models

Comments 0 Comment

Now that you have Stable Diffusion 1.5 installed on your local system, have learned how to make cool generative prompts, it might be time to take the next step of trying different latent models.

There is more than one model out there for stable diffusion, and they can generate vastly different images:

Check out this article to learn how to install and use different popular models you can use with stable diffusion:

  • F222 – People found it useful in generating beautiful female portraits with correct body part relations. It’s quite good at generating aesthetically pleasing clothing.
  • Anything V3 – a special-purpose model trained to produce high-quality anime-style images. You can use danbooru tags (like 1girl, white hair) in text prompt.
  • Open Journey – a model fine-tuned with images generated by Mid Journey v4.
  • DreamShaper – model is fine-tuned for portrait illustration style that sits between photorealistic and computer graphics
  • Waifu-diffusion – Japanese anime style
  • Arcane Diffusion – TV show Arcane style
  • Robo Diffusion – Interesting robot style model that will turn everything your subject into robot
  • Mo-di-diffusion – Generate Pixar-like style models
  • Inkpunk Diffusion – Generate images in a unique illustration style
Better stable diffusion and AI generated art prompts

Better stable diffusion and AI generated art prompts

Comments 1 comment

Now that you have stable diffusion on your system, how do you start taking advantage of it?

One way is to try some sample prompts to start with. Techspot has some good ones (halfway through the article) to whet your appetite.

You can get inspiration by looking at good examples on free public prompt marketplaces.

Then you might want to learn how to fix some common problems.

When you’re really ready to dive in, this article from Metaverse gives you a list of excellent getting started guides to help get you from beginner to proficient in generating your own awesome art.

The key to it all is learning the syntax, parameters, and art of crafting AI prompts. It’s as much art as it is science. It’s complex enough that there are everything from beginner examples, free guides, tools to help, all the way to paid marketplaces.

Learning gotten a lot better in the last 6 months since people started learning how to use AI generated prompts last year.

Installing Stable Diffusion 1.5

Installing Stable Diffusion 1.5

Comments 2 comments

To install Stable Diffusion 1.5 (released Oct 20, 2022) locally, I found this video was really excellent – except for a few points:

  1. You MUST use python 3.10.6 (I used 3.9.7 as recommended). The latest version (as of Feb 2023) is Python 3.11.1 – which stable diffusion does NOT seem to like and won’t run.

You might also want to read through this older stable diffusion 1.4 install guide, but he uses model checkpoints which haven’t been updated since version 1.4.

Gotchas and Fixes:

  • If you have an incompatible version of Python installed when you try to run webui-user.bat for the first time, stable diffusion will set itself up to point at this bad python version directory. Even if you uninstall and install the correct python version, stable diffusion will still look at the wrong python version. You can go fiddle with the different setup files – but it’s faster just to blow away the pulled git source at the top level and re-pull it to ensure you don’t have cruft laying around.

Installer Links: