How to Use DreamBooth to Fine-Tune Stable Diffusion (Colab)

With Stable Diffusion DreamBooth, you can now create art generation images using your own trained images.

For example you can generate images with yourself or a loved one as a popular video game character, as a fantastical creature, or just about anything you can think of – you can generate a sketch or a painting of your pet as a dragon, or as the Emperor of Mankind.

You can also train your own styles and aesthetics like aetherpunk/magicpunk, or maybe people’s facial expressions like Zoolander’s Magnum (I haven’t tried this yet).

In this tutorial we’ll cover the basics of fine-tuning Stable Diffusion with DreamBooth to generate your own customized images using Google Colab, for free. After we’ve fined tuned Stable Diffusion we’ll also test it out using Stable Diffusion WebUI built into the same Google Colab notebook. In machine learning, fine-tuning means adjusting a model that was trained on one dataset to work with a new, related dataset. This can make your model work better on the new dataset, or help it work better in a new situation. A dataset, in our case, is a bunch of pictures and some words that tell a machine what they should be looking for in order to generate new images.
Table of Contents

1. Quick Video Demo
2. Use DreamBooth to Fine-Tune Stable Diffusion in Google Colab
   1. Prepare Images (Resize & Crop to 512x512px)
   2. Open Fast Stable Diffusion DreamBooth Notebook in Google Colab
   3. Enable GPU
   4. Run First Cell to Connect Google Drive
   5. Run Second Cell to Set up The Environment
   6. Input Hugging Face Token and Run Third Cell to Download Stable Diffusion
   7. Setting Up Dreambooth
   8. [Optional] Upload Your Own Class Pictures
   9. Start DreamBooth
   10. Where Your New Model is Stored
3. Test the Trained Model (with Stable Diffusion WebUI by AUTOMATIC1111)
   1. Where Generated Images Are Stored
4. Conclusion

Quick Video Demo

This is a quick video of me fine-tuning Stable Diffusion with DreamBooth from start to finish. In this example I’m finetuning it using 20 images of the Sandman from The Sandman (TV Series). Ideally you should use more images to get better results.

The whole process took about an hour on Google Colab with 1500 training steps, but I have sped up the video in two of the more time consuming parts. If you leave it on default settings I estimate it would take ~40 minutes in total. And I think you can shorten this time by ~20 minutes if you provide your own class images (more on this later).
Watch this video on YouTube

Use DreamBooth to Fine-Tune Stable Diffusion in Google Colab Prepare Images (Resize & Crop to 512x512px)

Before anything else we should prepare our images. We need to resize and crop our images to be 512 x 512px. We can easily do this using the website https://birme.net.

To do this just:

1. Visit the website
2. Upload your images
3. Set your dimensions to 512 x 512px
4. Adjust the cropping area to center your subject
5. Click on Save as Zip to download the archive.
6. You can then unzip it on your computer and we’ll use them a bit later.

Resizing Images using Birme.net

It’s recommended that you provide 200 images of what you’d like to train Stable Diffusion on, to get great results.

However I’ve only used 20-30 so far, and results are pretty good. If you’re just starting out and want to test it out, I think 20-30 images should be good enough for now, and you can get 200 images later, after you’ve seen it work.

Open Fast Stable Diffusion DreamBooth Notebook in Google Colab

Next we’ll open the Fast Stable Diffusion DreamBooth colab notebook: https://colab.research.google.com/github/TheLastBen/fast-stable-diffusion/blob/main/fast-DreamBooth.ipynb

The notebook we’re using is an implementation of DreamBooth by TheLastBen. The author optimized multiple implementations of Stable Diffusion. You can check the Github repository here.
Fast DreamBooth Notebook Preview Enable GPU

Before running the notebook, we’ll first have to make sure Google Colab is using a GPU. This is because GPUs can process much more data than CPUs, and allows you to train our machine learning models faster.

To do this:

1. In the menu go to Runtime > Change runtime type. Runtime > Change runtime type
2. A small popup will appear. Under Hardware accelerator make sure you have selected GPU. Click Save when you’re done. Hardware Accelerator > GPU

Run First Cell to Connect Google Drive

By running the first cell we’ll start connecting our notebook to Google Drive so we can save our fine-tuned Stable Diffusion in it.

After running the first cell we’ll see a popup asking us if we want really want to Connect to Google Drive.

After we click it we’ll see another popup where we can select the account we want to connect with, and then to allow Google Colab some permissions to our Google Drive.

Mount Google Drive Run Second Cell to Set up The Environment

Next just run the second cell. There’s nothing for us to do there except wait for it to finish.

Input Hugging Face Token and Run Third Cell to Download Stable Diffusion

Next we’ll need to input our Hugging Face token. To do this we’ll need a huggingface.co account. If you don’t already have an account you can easily sign up at https://huggingface.co/join.

After signing up we’ll need to create a new token, which is like a password, and then input it into the Huggingface_Token input field.

1. To do this, in your Hugging Face account, click on your profile picture in the top right > Settings, and then in the left sidebar click Access Tokens, or just click here https://huggingface.co/settings/tokens.
2. Next click on New token. A popup will appear and give it any name you want, for reference,  and click Generate a token. Generate a New Token
3. If you haven’t already, we also need to accept some terms to get access to the Stable Diffusion repository.
   • To do this visit the repository URL here https://huggingface.co/CompVis/stable-diffusion-v1-4. You’ll see some terms that you have to accept and click Access repository. Those terms allow the authors access to our email and username. Access repository
4. Now we can input our token in the Huggingface_Token field and run the cell to download Stable Diffusion. Input Hugging Face Token

Setting Up Dreambooth

We can now get to setting up DreamBooth.

Before running this cell, you should know:

1. SUBJECT_NAME: This is like a base template for what we’ll be training. It’s also referred to as a class. For example I want to train Stable Diffusion to generate images of The Sandman, who looks like a guy. If I’d want to train it with pictures of my cat, I’d write cat.
   • We can either upload our own images, or let the notebook to generate its own. If we let it generate its own images and it will take ~25 minutes.
   • I’ve had great results with letting the notebook generate class pictures. If this is the first time you’re using this, I recommend you leave it generate them itself for now.
2. INSTANCE_NAME: This is the keyword you’ll use in the prompt. For example A still of Sandman2022 in The Matrix (2022).

Now you can run the cell, and the upload button will activate. You now can upload the 512 x 512px images we made earlier.

The instructions mention that you should upload ~400 images for excellent results. I’ve had great results with 20-30 images, as you can also see from the demo video. If this is your first time trying Stable Diffusion DreamBooth and want to break the ice, then I think 20-30 images will be fine for now.

Set Up DreamBooth [Optional] Upload Your Own Class Pictures

This is what we were talking about when discussing the SUBJECT_NAME. Class pictures are like the base template Stable Diffusion will use when training your images. If you’re training a dog, then you should have the class images of a dog/cat for cat/and so on. For people I’ve used either person, man, guy, or woman.

This step is optional. As we said before, if you don’t provide class images yourself, then the notebook will generate its’ own.

If this is the first time running this notebook, I recommend we skip this. If you’re skipping, then we can ignore this cell this time.

Upload Class Pictures Start DreamBooth

Finally we can run DreamBooth. This is where we’re fine-tuning

By default you’ll see the Training_Steps set to 1000. I’ve had good results with 1500 steps, which took about 40 minutes.

Depending on how long you’re willing to wait you can input more or less steps. I recommend you try 1500 steps. The entire process, along with the notebook generating class images, will take about an hour.

Start DreamBooth Where Your New Model is Stored

When it’s done you should find your model in your Google Drive. For example here’s where Sandman2022.ckpt was saved with default output folder settings.

Default Save Directory in Google Drive Test the Trained Model (with Stable Diffusion WebUI by AUTOMATIC1111)

After the training cell has finished running, we can test our new fine-tuned Stable Diffusion model.

This notebook comes with Stable Diffusion WebUI by AUTOMATIC1111, which is the most popular implementation of Stable Diffusion, and offers us a very convenient web user interface.

We have a few options:

1. If you have just fine-tuned Stable Diffusion for the first time (this is us, most likely) , and want to train your newly created model, then just run the Test the trained model cell. No need to fill out anything.
2. If you have previously fine-tuned Stable Diffusion using this notebook, then insert the INSTANCE_NAME. For example in my case it’s Sandman2022
3. If you have a model you want to load that’s in some folder in your Google Drive, then check Use_Custom_Path and after you run the cell you’ll see a field to provide the path to your model.

Test our fine-tuned Stable Diffusion model

When you run the cell it will take about 5 minutes for Stable Diffusion WebUI to be ready to use. When it’s done you’ll see some URLs like https://xyz.gradio.app.

Stable Diffusion WebUI is ready to use

Click it and then you can start using Stable Diffusion to generate our images. The user interface will open in a new tab, and you can start generating images right away.

[Prompt]: 「portrait of Sandman2022, gta san andreas cover art, borderlands style, celshading, trending on artstation, by rhads, andreas rocha, rossdraws, makoto shinkai, laurie greasley, lois van baarle, ilya kuvshinov and greg rutkowski」 Where Generated Images Are Stored

Images are stored by default in your Google Drive in My Drive > sd > stable-diffusion > outputs > txt2img-images.

Where Generated Images Are Stored Conclusion

In this tutorial we covered how to fine-tune Stable Diffusion using DreamBooth via Google Colab free, to generate our own unique image styles. We hope this tutorial helped you break the ice in fine-tuning Stable Diffusion.

If you encountered any issues or have any questions please free to leave a comment and we’ll get back to you as soon as possible.