There’s no one-size-fits-all answer to this question, but here are a few tips to help you write an awesome diffusion prompt:
- Start by thinking about the main idea of your prompt. What is it that you want your readers to think about?
- Make sure your prompt is interesting and engaging. Readers will be more likely to read and respond to your work if they’re entertained and interested in what you’re writing.
- Be sure to include a clear and concise summary of the main idea of your prompt in your opening paragraph. This will help readers understand what they need to know in order to participate in the discussion.
- Be sure to include a wide range of potential responses in your prompts, so that readers can find something that interests them both on the surface level and beneath the surface layer!
How to Write a Stable Diffusion Prompt
If you’ve spent any time at all with AI image generators, like Stable Diffusion, DALL-E, or MidJourney, you’ll have noticed that a well-worded prompt critical. A well-worded prompt is the difference between turning your idea into a great image and getting some uncanny-valley monstrosity with entirely too many fingers staring back at you.
People have been trying to suss out the best ways to get ideal results since Stable Diffusion launched in August 2022, and they’ll be “prompt smithing” or “prompt engineering” for years to come. That is especially likely since the checkpoints that inform how Stable Diffusion generates images will receive periodic updates. The ideal prompt is going to be a moving target for the foreseeable future.
RELATED: The Best AI Image Generators You Can Use Right Now
Be As Specific as You Can
Stable Diffusion tends to thrive on specific prompts, especially when compared to something like MidJourney. You need to tell it exactly what you want. Here is an example using the internet’s favorite animals: Cute cats.
Alright, they’re pretty cute, except for maybe the poor fellow in the top-middle who looks like he has spent too much time staring into the abyss. But what if you wanted grey cats, not a mix of cats? Well, you need to specify that. Use “Cute grey cats” as your prompt instead.
Now Stable Diffusion returns all grey cats. You can keep adding descriptions of what you want, including accessorizing the cats in the pictures.
This applies to anything you want Stable Diffusion to produce, including landscapes. Be descriptive, and as you try different combinations of keywords, keep a mental note of how the image changes. Some words and phrases tend to more strongly bias an image than other words, so you might need to adjust your prompt accordingly.
Name Specific Art Styles or Mediums
The need for specificity doesn’t end when you fully describe the content of the image you want. You can (and should) also specify the style you want. We’ll stick with our “Cute grey cat” prompt for this. Let’s say we want some cute grey cats, but we also want them to look like they’re an acrylic painting. The first thing you should try is adding “acrylic painting” as the next keyword in your prompt, so it would read: “Cute Grey Cat, acrylic painting.”
It is generally best to start with the fewest keywords to get close to what you want, and then add more to home in on an aesthetic you’re looking for.
Stable diffusion recognizes dozens of different styles, everything from pencil drawings to clay models to 3d rendering from Unreal Engine.
Those example prompts are extremely simple, but you can use dozens of keywords to fine-tune your results. If you want weird, psychedelic-colored clay chicken models in a Walmart, you need to specify all those terms.
Name Specific Artists to Guide Stable Diffusion
Stable Diffusion is capable of doing more than emulating specific styles or mediums; it can even mimic specific artists if you want to do that. This example used Pablo Picasso.
You should definitely try throwing “by (Artist Name)” into your prompts if you haven’t already. It tends to produce dramatic results. Don’t be afraid to combine artists that wouldn’t normally go together, either. Oftentimes Stable Diffusion will combine their respective styles in exciting and unpredictable ways.
Weight Your Keywords
Of course, just jamming keywords into your prompt will only get you so far. What happens if you’re getting all of the right things in your prompt, but they’re not properly proportioned?
Stable Diffusion supports weighting of prompt keywords. In other words, you can tell it that it really needs to pay attention to a specific keyword (or keywords) and pay less attention to others. It is handy if you’re getting results that are kinda what you’re looking for, but not quite there.
RELATED: How to Run Stable Diffusion on Your PC to Generate AI Images
In the command-line version of Stable Diffusion, you just add a full colon followed by a decimal number to the word you want to emphasize. The decimal numbers are percentages, so they must add up to 1.
Going back to our “Cute grey cat” prompt, let’s imagine that it was producing cute cats correctly, but not very many of the output images featured grey cats. You could modify the prompt to read: “cute, grey cat:0.7” instead, and it would pay the most attention to grey cat, then automatically apply the difference to “cute.” You can also assign weights to each word in the prompt manually if you want finer control, like “Cute:0.10, Grey Cat:0.60, Unreal Engine rendering:0.30” for example.
RELATED: How to Run Stable Diffusion Locally With a GUI on Windows
Most of the GUIs available for Stable Diffusion handle weighting without requiring you to enter explicit percentages. You can usually add parenthesis around a term in your prompt to emphasize it, and brackets to decrease the importance of a term. So the previous “cute, grey cat” might look like “[cute],((grey cat)).”
Find Inspiration Elsewhere
Stable Diffusion and other AI art generators have experienced an explosive popularity spike. You can find this sort of AI art all over the place. That means there are now at least a few million user-generated images floating around on the internet, and most of the time, people include the prompt they used to get their results.
RELATED: This AI Art Gallery Is Even Better Than Using a Generator
Here are a few resources to help spark your inspiration if you aren’t sure what you’re looking to create:
Lexica — a repository of images generated using Stable Diffusion and the corresponding prompt. Searchable by keyword. Stable Diffusion Artist Style Studies — A non-exhaustive list of artists Stable Diffusion might recognize, as well as general descriptions of their artistic style. There is a ranking system to describe how well Stable Diffusion responds to the artist’s name as a part of a prompt. Stable Diffusion Modifier Studies — a list of modifiers that can be used with Stable Diffusion, just like the artist page. The AI Art Modifiers List — A photo gallery showcasing some of the strongest modifiers you can use in your prompts, and what they do. They’re sorted by modifier type. Top 500 Artists Represented in Stable Diffusion — We know exactly what images were included in the Stable Diffusion training set, so it is possible to tell which artists contributed the most to training the AI. Generally speaking, the more strongly represented an artist was in the training data, the better Stable Diffusion will respond to their name as a keyword. The Stable Diffusion Subreddit — The Stable Diffusion subreddit has a constant flow of new prompts and fun discoveries. If you’re looking for inspiration or insight, you can’t go wrong.
Tweak Other Important Settings
A good prompt is the hardest part of using Stable Diffusion, but there are a few other settings that will dramatically change the results.
CFG: Determines how strongly Stable Diffusion follows your prompt. Higher numbers result in more adherence to the prompt, whereas lower numbers give the AI more freedom. Try tweaking this first. Sampling Method: How the image is refined from noise into recognizable shapes. Try out a few of these. Euler_a, k_LMS, and PLMS seem to be popular choices. Sampling Steps: The number of times an image will be sampled before you’re given a final result. Sometimes you get good results at 30 steps, sometimes you need to go to 50 or 80. You don’t usually get better results above 150 steps. Start with fewer steps and go up.
Some of Our Favorite Stylized-Prompts
These are some of our favorite prompts since they tend to work so reliably. Just drop in a subject of your choosing plus a few modifiers. All other Stable Diffusion settings were kept the same as the prompts up above.
(Subject), 35mm, sharp
Pick a subject, and this prompt reliably returns photorealistic results of people, animals, and landscapes.
(Subject), low poly 3d render, vibrant pastel colors, tilt shift, film grain
Enter a subject and you’ll get it in the “Art of Rally” aesthetic.
(Subject), (Environmental Description), cinematic, dramatic, composition, sunny sky, brutalist, hyper realistic, epic scale, sense of awe, hypermaximalist, insane level of details, artstation HQ
Just drop in a subject and an environmental description, and this prompt provides great city concept art images with an environmental effect of your choosing. Here is an example using New York City as the subject and dust storm as the environmental description:
Kermit Anything
Stable Diffusion produces great results with Kermit. Try it out anywhere.
Why is this how Stable Diffusion interprets “Kermit in Mordor,” and who is the red evil-Kermit-thing standing behind our stalwart hero? No idea.
Remember, don’t be afraid to remove what you think might be a key term from your prompt. Certain words, like “gorgeous,” will heavily influence what an image looks like, even if it doesn’t have a precise meaning. As you spend more time with Stable Diffusion, you’ll develop a feel for how it responds to certain words, and you’ll quickly find that writing a good prompt is an art unto itself.
