Blending Two Famous Faces in Stable Diffusion: Prompt Editing, Weights and My Real Settings

Short answer: Stable Diffusion 1.5 already knows what many famous people look like, so naming them in the prompt gets you a recognisable likeness. To blend two people, use AUTOMATIC1111's prompt editing syntax, [Person A:Person B:0.5], which starts with A and switches to B halfway through the sampling steps. Use (word:1.2) style weights to emphasise details, and keep weights under about 1.5.

In February 2025 I tried this with Abraham Lincoln and Henry Cavill, then with a portrait model (F222). Every prompt, seed and setting below is read straight from the PNG metadata of the images, so you can reproduce them.

One correction up front. An earlier version of this post used Abraham Lincoln | Henry Cavill :0.5 and called it prompt scheduling. That isn't valid AUTOMATIC1111 syntax. The blend I got came from simply having both names in the prompt, not from any schedule. The correct syntax is explained below.

My setup

• UI: AUTOMATIC1111 Stable Diffusion WebUI v1.9.4, running in Docker (my setup guide).
• Models: sd-v1-5-inpainting (hash c6bbc15e32) for most runs, f222 (hash 9e2c6ceff3) for the portrait model test, and HassanBlend1.4_Safe (hash b08fdba169) for the banner image.
• Size: 512x512 for every Lincoln/Cavill image.

Like in my Docker post, I used the inpainting checkpoint for text-to-image because it was the default loaded. It works, but a standard SD 1.5 checkpoint is the right choice for generating from scratch.

If you're new to the settings (steps, CFG, sampler, seed), see my Stable Diffusion parameter guide.

Every run at a glance

Step 1: just the name

The simplest possible prompt:

Copy Abraham Lincoln

The model clearly knows who Lincoln is: the beard, the hairline, the bow tie. But it gave me an old-photo look in sepia with a waxy, slightly cartoonish face. That makes sense, since nearly every photo of Lincoln in the training data is a 19th-century black and white one.

Step 2: a detailed prompt and a negative prompt

Copy A highly detailed, colorful, photorealistic portrait of Abraham Lincoln, 4K resolution, dramatic lighting, sharp facial details, deep wrinkles, strong jawline, piercing eyes, dignified expression, black suit with bow tie, cinematic composition, DSLR shot, volumetric lighting, ultra-HD, realistic skin texture., ultra quality, dof, film grain, Fujifilm XT3, crystal clear

Negative prompt:

Copy deformed face, disfigured, asymmetric, duplicate, tattoos, marks, Extra heads, extra faces, multiple faces, extra eyes, ugly, bad, immature, cartoon, anime, 3d, painting, b&w, multiple figures, multiple heads, extra hands

A big step up. "colorful" in the prompt plus "b&w" in the negative prompt pushed it out of sepia, and the skin texture and wrinkles look photographic. It didn't follow everything though: I asked for a black suit and got a grey one with a bright pink bow tie. "colorful" probably had something to do with that.

Step 3: blending two people

This is where I went wrong in the original post. I wrote the prompt like this:

Copy A highly detailed, photorealistic image of Abraham Lincoln | Henry Cavill :0.5, 4K resolution, cinematic lighting, sharp facial details, deep wrinkles, strong jawline, relaxed yet dignified expression, 19th-century bathroom with brass fixtures, warm candlelight ambiance, DSLR shot, ultra-HD, realistic skin texture, water reflections, historical accuracy.

The result is a convincing mix: Henry Cavill's face, hair and smile with Lincoln's chin-curtain beard and 19th-century collar. So it "worked". But not for the reason I thought.

Why A | B :0.5 isn't scheduling

In AUTOMATIC1111, the | and : characters only mean something inside square brackets. A bare | outside brackets makes the prompt-editing parser fail, and when that happens the WebUI quietly falls back to using the whole prompt as plain text. So the model saw both names, plus the literal text " | " and ":0.5", for every step. Both names being present is what blended the faces.

The same applied to my next attempt, where I added holding a glass:10 | candle:-4 to make him hold a glass. That isn't valid syntax either, and the output shows it: no glass, no candle, and no bathroom. I got an outdoor portrait by a lake with Cavill dominant.

The correct syntax

These are the prompt editing forms AUTOMATIC1111 actually supports:

when is a fraction of the total steps if it's below 1 (0.5 of 20 steps = switch at step 10), or an exact step number if it's 1 or more.

How to choose when:

• Early steps set the composition (pose, framing, head shape). Later steps set the fine detail (skin, eyes, texture).
• So [Abraham Lincoln:Henry Cavill:0.5] gives you roughly Lincoln's composition with Cavill's features painted in. Lower when (0.3) and Cavill takes over. Raise it (0.7) and Lincoln stays more visible.
• [Abraham Lincoln|Henry Cavill] often gives the most even 50/50 face, because the model never commits to either one.
• Fix the seed while you try different when values, so the only thing changing is the blend.

So the prompt I should have used is:

Copy A highly detailed, photorealistic portrait of [Abraham Lincoln:Henry Cavill:0.5], 19th-century bathroom with brass fixtures, warm candlelight, DSLR shot, realistic skin texture

Step 4: prompt weighting, and overdoing it

Weighting changes how much attention the model pays to a word:

Watch out for one trap: [word] reduces emphasis, but [word:0.5] is not a weight. Because it's in square brackets with a number, it's prompt editing ("add 'word' after 50% of the steps"). Use (word:0.5) for a low weight.

Weighting is handled separately from prompt editing, so it still worked in my prompt even though the | part didn't. Here I used the same seed as the previous image (1101313381) and added a chair with a very high weight:

Copy A highly detailed, photorealistic image of Abraham Lincoln | Henry Cavill :0.6 sitting on a (chair:2.3), black suit with bow tie, [history accuracy] , 4K resolution, cinematic lighting, DSLR shot, ultra-HD, realistic skin texture,  cinematic composition, DSLR shot, volumetric lighting, ultra-HD, realistic skin texture, ultra quality, dof, film grain, Canon 70D

He's sitting, so the weight did its job. But look at the background: chair legs, frames and armrests everywhere. That's what a 2.3 weight does. The word takes over the whole image. The face also drifted almost entirely to Cavill, and the "black suit" became a white jacket with burgundy trousers. Weights between 1.1 and 1.4 are usually enough. Above about 1.5, expect artefacts like this.

Step 5: a portrait-tuned model (F222)

Community models fine-tuned on photos of people usually do faces better than base SD 1.5. I used F222 (how to install it is in Mastering Human Portraits with Stable Diffusion) with a simpler prompt:

Copy A highly detailed, photorealistic image of Abraham Lincoln | Henry Cavill :0.5 in white shirt , 4K resolution, cinematic lighting, DSLR shot, ultra-HD, realistic skin texture,  cinematic composition, DSLR shot, volumetric lighting, ultra-HD, realistic skin texture, ultra quality, dof, film grain, Canon 70D

Settings: UniPC sampler, Karras schedule, 24 steps, CFG 7, same negative prompt as before.

These are the most even blends of the set. The first has Cavill's eyes and jaw with Lincoln's beard, hairline and period clothes. The second reads more as Lincoln. Both are much more natural than the inpainting-checkpoint results. Neither is in a white shirt, though. "white shirt" lost out to "Abraham Lincoln", which pulls hard towards a black coat.

To be fair, this isn't a controlled comparison. The seeds, sampler and steps differ from the SD 1.5 runs. For a proper A/B test, keep everything the same and change only the checkpoint.

A practical warning about F222: it was trained heavily on nude photos and produces nudity easily. Put nsfw, nude in the negative prompt.

Other portrait models people often use with SD 1.5 are Realistic Vision (general photoreal) and ChilloutMix (photoreal, leans towards East Asian faces; an earlier version of this post wrongly called it anime-style). I didn't test either for this post.

The banner image

The image at the top of this post came from a different model, HassanBlend1.4_Safe, with the prompt starting A highly detailed, photorealistic of natalie portman | jennifer lawrence :0.4, sitting outside restaurant, coffee on the table. Settings: Euler a, Karras, 20 steps, CFG 9, seed 1771624804. I asked for 910x512 and got 904x512, because the WebUI rounds sizes down to a multiple of 8. It uses the same invalid | syntax, so again it's both names in one prompt, not a schedule.

Tips if you're trying this

• Start with the name alone to check the model actually knows the person. Lesser-known people won't come out recognisable from a name.
• Use proper prompt editing, [A:B:when] or [A|B], and sweep when with a fixed seed.
• Keep weights at 1.5 or below.
• Name clothing and setting clearly, or accept that the person's "default look" wins. Lincoln drags in a black coat and bow tie whatever you ask for.
• Use a portrait-tuned checkpoint, not the inpainting one, for text-to-image.
• Generate several seeds. The same prompt swings from mostly Cavill to mostly Lincoln depending on the seed.

Use it responsibly

A recognisable face of a real person is a likeness, whatever tool made it.

• Don't present a generated image as a real photo, and don't use one to imply that a real person said, did or endorsed something.
• No sexual or intimate images of real people. In Australia, sharing sexually explicit deepfakes of an adult without consent is a criminal offence under the Criminal Code Amendment (Deepfake Sexual Material) Act 2024. Many other countries have similar laws.
• Commercial use is risky. Using a celebrity's likeness to sell something can amount to misleading conduct or passing off, even when the image is "only AI".
• Hosted generators often block real names. Running locally removes that filter, so the responsibility is yours.

Blending a historical figure with an actor for fun, like the "Cavilcoln" here, is about as harmless as this gets. Clearly label it as AI-generated when you share it.

References

• AUTOMATIC1111 WebUI wiki: Features (prompt editing, alternating words, attention/emphasis)
• How to generate realistic people in Stable Diffusion