Something wrong with img2img samplers

Describe your environment

mbp m1pro
Branch: development

Describe the bug
I’m trying new samplers with img2img (klms & keuler), but always get strange result.
I use a classic command from https://github.com/invoke-ai/InvokeAI/blob/main/docs/features/IMG2IMG.md, just with «-A klms» for example.

strength 0.5

strength 0.3

Originals:

I just confirmed that img2img with the plms and k* samplers is working as expected on a CUDA system. This may be an M1-specific issue, although I'd be surprised if this was the case. Could you post the original images and the exact prompts you used?

@netsvetaev

I'm getting the same weird washed out and blurry images at lower img2img strengths. Also on M1.

@netsvetaev 's original images are the last two images they posted btw.

Here's my init image:

And here's k_lms, k_euler, and ddim each at 32 steps and -f 0.25, 0.5, 0.75.

"photograph of a tree on a hill with a river" -s 32 -S 12345 -W 512 -H 512 -C 7.5 -I /Users/spencer/Documents/Code/initimage.png -A k_euler -f 0.25

"photograph of a tree on a hill with a river" -s 32 -S 12345 -W 512 -H 512 -C 7.5 -I /Users/spencer/Documents/Code/initimage.png -A k_euler -f 0.5

"photograph of a tree on a hill with a river" -s 32 -S 12345 -W 512 -H 512 -C 7.5 -I /Users/spencer/Documents/Code/initimage.png -A k_euler -f 0.75

"photograph of a tree on a hill with a river" -s 32 -S 12345 -W 512 -H 512 -C 7.5 -I /Users/spencer/Documents/Code/initimage.png -A k_lms -f 0.25

"photograph of a tree on a hill with a river" -s 32 -S 12345 -W 512 -H 512 -C 7.5 -I /Users/spencer/Documents/Code/initimage.png -A k_lms -f 0.5

"photograph of a tree on a hill with a river" -s 32 -S 12345 -W 512 -H 512 -C 7.5 -I /Users/spencer/Documents/Code/initimage.png -A k_lms -f 0.75

"photograph of a tree on a hill with a river" -s 32 -S 12345 -W 512 -H 512 -C 7.5 -I /Users/spencer/Documents/Code/initimage.png -A ddim -f 0.25

"photograph of a tree on a hill with a river" -s 32 -S 12345 -W 512 -H 512 -C 7.5 -I /Users/spencer/Documents/Code/initimage.png -A ddim -f 0.5

"photograph of a tree on a hill with a river" -s 32 -S 12345 -W 512 -H 512 -C 7.5 -I /Users/spencer/Documents/Code/initimage.png -A ddim -f 0.75

I notice that the sampler only runs for one step (regardless of input), this is probably the reason for the blurry results

I notice that the sampler only runs for one step (regardless of input), this is probably the reason for the blurry results

I'm not sure that's accurate - I get very different results when I specify 1 step vs any other number:

1 step "photograph of a tree on a hill with a river" -s 1 -S 12345 -W 512 -H 512 -C 7.5 -I /Users/spencer/Documents/Code/initimage.png -A k_euler -f 0.25

32 steps "photograph of a tree on a hill with a river" -s 32 -S 12345 -W 512 -H 512 -C 7.5 -I /Users/spencer/Documents/Code/initimage.png -A k_euler -f 0.25

It's possible you are using a low number of steps - with img2img, the number of actual steps is a function of strength. So you specify 32 steps at strength 0.25, you get floor((32 - 1) * 0.25) --> 7 steps. The result is that for certain combinatoins of low step count and strength, it only does one step.

@mh-dm

This seems to be an MPS-related bug. I don’t have a Mac to test on, but I’ll investigate any suspicious “if MPS” statements. Maybe @mh-dm or @Any-Winter-4079 could have a look?

I don't have an MPS device to test on.

@lstein

@lstein I'm not convinced it is MPS-related. I have the exact same behavior after forcing my torch device to be cpu:

in ldm/dream/devices.py:

def choose_torch_device() -> str:
    '''Convenience routine for guessing which GPU device to run model on'''
    # if torch.cuda.is_available():
    #     return 'cuda'
    # if hasattr(torch.backends, 'mps') and torch.backends.mps.is_available():
    #     return 'mps'
    return 'cpu'

in ldm/generate.py:

def fix_func(orig):
    # if hasattr(torch.backends, 'mps') and torch.backends.mps.is_available():
    #     def new_func(*args, **kw):
    #         device = kw.get("device", "mps")
    #         kw["device"]="cpu"
    #         return orig(*args, **kw).to(device)
    #     return new_func
    return orig

Each step now takes about 4x as long, and there are no messages about mismatched torch devices, so I think I've forced cpu successfully...

I have been writing some documentation for img2img, which involved visualizing the latent space, and i noticed that when the step count does not scale with strength. if you request 100 steps with a strength of 0.3, SD will only actually do 30 steps of inference. with a strength of 0.6 it will 60. not sure if this is intended behaviour, a bug, or an oversight in the UI design, but since it seems to be surprising i'd suggest adjusting the step count to compensate for strength when doing img2img.

Ok, there is something wrong on a deeper level. I have reverted the force-cpu changes to my local branch and run some tests with strength 0.75 and 0.85, each running from steps 15 to ~40. Sampler DDIM.

My REPL command:

"photograph of a tree on a hill with a river" -S 12345 -W 512 -H 512 -C 7.5 -I /Users/spencer/Documents/Code/initimage.png -A ddim -f {strength} -s {steps}

Using this same init image:

Strength 0.75 images (named the zip f7.5 but its f0.75):
f7.5.zip

Strength 0.85 images (named the zip f8.5 but its f0.85):
f8.5.zip

Images are prefixed by the steps count entered in the REPL (not the actual steps).

Have a scroll through those and you will notice that there are some interesting patterns. There seem to be 2 types of result which I'll call the "common" type - very close to the init image - and the "occasional" type - which diverges more. You get 4 or 5 "common" types, then a couple "occasional" types, then 4 or 5 "common" types, then a couple "occasional" types...

It's far more obvious on the f0.85 images which are "common" and which are "occasional".

Also, it seems like the more steps, the closer the "common" images are to the init image. I would have expected the images to converge but not on something very much like the original...

Here's f0.85 s200:

And here's f0.85 s201:

That ain't right!

@psychedelicious

@psychedelicious you might want to try my branch document-img2img which spits out pngs for each intermediate step. check the outputs/img_samples/intermediates folder, which may help understand what's going on (this is what i'm documenting!)

this is actually a subtle thing that i haven't seen any of the SD guis communicate well. because of the way the SD algorithm works (actually diffusion algorithms more generally), doing steps=50 is not equivalent to doing steps=49 then feeding the 49th image in for "one more" step. if the step count is different then the amount of denoising that happens at each step is different, all the way back to the first step. so your difference between s200 and s201 could be emerging actually from a change that happened on step 2 or 3 - two pixels get interpreted slightly differently and then 200 iterations later your spindly tree has become a bushy tree somewhere else.

the best way to get a handle on this, i found, was to try low step counts and look at the latents at each step. if there's interest i can clean up my intermediate writer and submit it as a feature..?

@damian0815

@damian0815 Thanks, I don't understand what is really going on internally - thanks for explaining that. Do I understand you correct, that the patterns I am seeing are to be expected?

I checked out your branch, but it seems to be missing from scripts.modules.preview_decoder import ApproximateDecoder.

I discovered by trial and error that, in the step_callback, calling sample_to_image(sample) doesn't give you the intermediate image. Would love to have access to your intermediate writer. Thanks.

@psychedelicious

yep that's the problem, see how we've already diverged enough to see with our eyes alone after about the 3rd or 4th step. top is 20 steps (-s 30 -f 0.7), bottom is 19 (-s 29 -f 0.7).

@psychedelicious i've fixed the issue on my branch - if you fetch again you can add --write_intermediates to the end of your dream> prompt and it will spit out all the latent steps to an intermediates folder under outputs/img_samples. please let me know how you get on with it!

& i'd love to hear your comments on my updated img2img docs! :)

@psychedelicious

@psychedelicious @lstein
I can reproduce this
Input image

"a painting of * in the style of van gogh" -s 50 -S 514644559 -W 512 -H 512 -C 7.5 -I outputs/img-samples/002109.514644559.png -A ddim -f 0.05
Before:

Now:

However, with higher strength:
"a painting of * :1.8 in the style of van gogh" -s 50 -S 514644559 -W 512 -H 512 -C 7.5 -I outputs/img-samples/002109.514644559.png -A ddim -f 0.75
Before:

Now:

A lot closer / identical

"photograph of a tree on a hill with a river" -s 40 -S 12345 -W 512 -H 512 -C 7.5 -I ./test-pictures/river-and-mountain.png -A k_euler
-f0.3

-f0.75

-f0.9

@Any-Winter-4079

I am just removing debugging code and will commit in a sec. @Any-Winter-4079 , please compare with your solution and let me know which one is working better.

UPDATE: pushed. The revised code is now in development and release-candidate-2

@Birch-san

@Birch-san - Invokes discord >> https://discord.gg/ZmtBAhwWhy

I've got discord open at this link, but I'm such a noob I don't know how to rendezvous with you folk. What are your discord names?

have made contact

@lstein

@lstein 7541c7c looks good in general. The washed out effect is completely removed.

Playing with an old commit (2c27e75) I noticed that playing with sigmas you can increase/decrease the creativity (the more sigmas from the start you remove, the less creativity is has)

But my problem was I was removing a fixed amount of sigmas, regardless of steps and strength. So, removing a dynamic amount of sigmas (like in 7541c7c) to fix this problem, we can also obtain nice results:

sigmas = self.karras_sigmas[-S-1:] from 7541c7c vs. sigmas = self.karras_sigmas[-int(1.1*S)-1:] (basically removing a few less sigmas) vs. sigmas = self.karras_sigmas[-int(1.2*S)-1:]

I'm not saying one option should be preferred over the other (how much variation we should have at -f0.6, -f0.75... I'd say is a bit of a personal choice/preference).

Also, something I've seen is increasing for example 20% the number of steps in 7541c7c won't produce the same results as
sigmas = self.karras_sigmas[-int(1.2*S)-1:] despite both doing 28 true steps, because the sigmas are not the same.

sigmas = self.karras_sigmas[-S-1:] from 7541c7c -> 29 sigmas

tensor([1.9426, 1.7265, 1.5313, 1.3554, 1.1971, 1.0549, 0.9274, 0.8134, 0.7116,
        0.6209, 0.5403, 0.4689, 0.4057, 0.3499, 0.3009, 0.2578, 0.2202, 0.1874,
        0.1588, 0.1341, 0.1127, 0.0944, 0.0786, 0.0652, 0.0538, 0.0441, 0.0360,
        0.0292, 0.0000], device='mps:0')

sigmas = self.karras_sigmas[-int(1.2*S)-1:] -> 29 sigmas

tensor([3.6092, 3.1686, 2.7749, 2.4239, 2.1117, 1.8346, 1.5892, 1.3726, 1.1817,
        1.0141, 0.8673, 0.7391, 0.6275, 0.5307, 0.4469, 0.3748, 0.3129, 0.2599,
        0.2148, 0.1767, 0.1444, 0.1174, 0.0948, 0.0761, 0.0606, 0.0479, 0.0375,
        0.0292, 0.0000], device='mps:0')

To get to the image obtained with sigmas = self.karras_sigmas[-int(1.2*S)-1:]) using 7541c7c you need to increase strength by 20% (not steps).
"photograph of a tree on a hill with a river" -S 12345 -W 512 -H 512 -C 7.5 -I 193515615-921b5f80-a2f3-4351-9459-b8e447ea765d.png -A k_euler -f0.72 -s40

I guess it's a matter of preference. For example, with 7541c7c we can't obtain this image from sigmas = self.karras_sigmas[-int(1.2*S)-1:] at -f0.9

Because we'd need 0.9 strength * 1.2 = 1.08
But for example we can obtain this image from sigmas = self.karras_sigmas[-int(1.1*S)-1:] at -f0.9

by setting strength to 0.99

So we're basically shifting the strength to start at more or less creativity, which affects how far into creativity we can go when we reach -f1

There's also a churn variable in the sampling algorithms which increases stochasticity at each step. It's been set to zero, but I think you get more "creativity" for positive values. The major question is what options do we expose to the user?

I would expose sigma_min, sigma_max and rho (mostly relevant for users trying to get good results at low number of sampler steps, by tactically choosing the range and curve of their schedule).

rho explained here:
https://twitter.com/Birchlabs/status/1576705558177935361

deliberate exclusion of sigmas from the schedule (e.g. increased sigma_min) demonstrated here:
crowsonkb/k-diffusion#23

the sigmas on which the model trained (model.sigmas) are known. not sure what UI element would be appopriate for restricting your choice to these:
https://gist.github.com/Birch-san/6cd1574e51871a5e2b88d59f0f3d4fd3

and yes, exposing churn is a good idea. I saw in k-diffusion's clip-guided diffusion example that a typical value for churn is 50.

one problem with churn at the moment is that k-diffusion doesn't yet provide any way to discretize the sigma_hats that arise after applying churn, so the model is asked to denoise sigmas on which it never trained. I think it'd probably still look relatively good though.

personally I find the idea of strength a little hard to predict; I'd prefer something like a sigma cutoff. so you say "make a 20-step noise schedule, keep only the sigmas higher than 4.0300). you'd pick sigmas from here:
https://gist.github.com/Birch-san/6cd1574e51871a5e2b88d59f0f3d4fd3

Tweaking sigma_min was interesting and definitely seemed to make a difference. https://github.com/invoke-ai/InvokeAI/discussions/914#discussioncomment-3800884
Which reminds me, I have it pending to test sigma_min for more prompts. Maybe I'll do another document similar to https://github.com/invoke-ai/InvokeAI/blob/development/docs/help/SAMPLER_CONVERGENCE.md to share the results.

fix invoke-ai#889 - fuzzy k* img2img at low strength

Navigation Menu

Repository navigation

Something wrong with img2img samplers #889

49 remaining items

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Something wrong with img2img samplers #889

Description

Activity

lstein commented on Oct 2, 2022

psychedelicious commented on Oct 3, 2022

ranhalprin commented on Oct 3, 2022

psychedelicious commented on Oct 3, 2022

lstein commented on Oct 4, 2022

mh-dm commented on Oct 4, 2022

psychedelicious commented on Oct 4, 2022

damian0815 commented on Oct 4, 2022

psychedelicious commented on Oct 4, 2022

damian0815 commented on Oct 4, 2022

damian0815 commented on Oct 4, 2022

psychedelicious commented on Oct 4, 2022

damian0815 commented on Oct 4, 2022

Any-Winter-4079 commented on Oct 4, 2022

49 remaining items

lstein commented on Oct 6, 2022

lstein commented on Oct 6, 2022

lstein commented on Oct 6, 2022

Birch-san commented on Oct 6, 2022

Any-Winter-4079 commented on Oct 7, 2022

Any-Winter-4079 commented on Oct 7, 2022

Any-Winter-4079 commented on Oct 7, 2022

lstein commented on Oct 7, 2022

Birch-san commented on Oct 8, 2022

Birch-san commented on Oct 8, 2022

Any-Winter-4079 commented on Oct 8, 2022

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions