AI video generation experiments

[Sekani]

Over the past few weeks I've expanded more into AI-generated video. While Kling is currently still the gold standard for generations relative to our interests, it's fairly restrictive as far as content goes and it's expensive. Still, can't argue with the results:

kling_20250819_Image_to_Video_The_woman__2516_1.mp4

But I've been messing around more with something that can be run locally without restrictions, as long as your machine is powerful enough (I'm cheating and renting one). Wan video only creates 3-5 second snippets, and they're not quite as polished as Kling, but I think the creative freedom allowed is worth the small tradeoff in quality (and lower expense).

Linking some samples from Deviant Art to avoid exceeding attachment limits on this post:

https://www.deviantart.com/sekani2/art/W04-1236172457

https://www.deviantart.com/sekani2/art/W03-1236172469

https://www.deviantart.com/sekani2/art/W05-1236483201

I did make one attempt to string together multiple Wan clips to make a longer one, perhaps telling a bit of a more coherent story, but it was pretty bad at keeping certain details together (like her face):

Timeline 1.mp4

I'm gonna keep exploring the medium, because I think the new tools have A LOT of potential.

[JSample]

Very nice video renderings, especially the Kling one with the audio!

[Viridian]

Well done on the experimental results. You've probably seen enough of my work on DA to know that I firmly believe that there's a lot of potential in making cohesive visual narratives. Kling is the unmatched gold standard at the moment, and it's not as restrictive as it may seem, as I've been discovering.

[BogDog]

Very cool results. I hope it doesn't take long before we can do this at home on our own PCs with quality results. A few days ago I spent $40 for points to work on videos and they were used up in only 2 days.

[dlodoski]

Very cool results. I hope it doesn't take long before we can do this at home on our own PCs with quality results. A few days ago I spent $40 for points to work on videos and they were used up in only 2 days.

Agreed.

I'm holding out for local processing as well.

[Sekani]

Well done on the experimental results. You've probably seen enough of my work on DA to know that I firmly believe that there's a lot of potential in making cohesive visual narratives. Kling is the unmatched gold standard at the moment, and it's not as restrictive as it may seem, as I've been discovering.

Yeah it's surprisingly agreeable even with the stricter attempts at censorship, but the cost for me is the main drawback. By comparison I've done hundreds of "local" generations with Wan for only the cost to rent the machine, which right now is $8/month. I still have a monthly sub to Kling for occasional experiments, but I don't think it'll be my primary driver for a while.

[mjw]

I'd experiment with Kling much more if I could run it locally (even if it took more time to process).

As for chaining multiple videos, Kling now supports start and end frame with Kling 2.1. Unfortunately this makes it a lot more expensive, but it does give you much more control over the video, so if it reduces the amount of generations you need to get what you want it may be worth doing. When using this, having a good start and end image are essential, and they must be consistent or you can get some really weird things happen.

Even so it's quite hard to make a video more than 30s long, but I've managed to turn some real life photos I took into a half decent sinking video this way.

I have also found the Hailuo 2.0 model good with sloppy mud without it all turning to water, and having the mud move in realistic ways (again starting with a good image to start the video).

[rhum]

The most important game changer with wan 2.2 is its ability to run on lower end GPUs

I have been using a 4080, but the workflows I have used can be easily adapted to lower end cards.

I have been messing a bit with wan 2.2 14B (i2v with flux and illustrious images) and had some success with chaining multiple videos by saving every video without the final frame, and saving the final frame to an image, then using the final frame image to start the next video, preventing the frame from being shown twice, this obviously does nothing for continuity of movement and you have to do many different generations to achieve anything good. That is combined with the same prompting inconsistencies that mirror Kling and other models, all are currently hit and miss when it comes to QS related material (sink? lower? descend?)

Of course, only paying electricity for these generations allow for greater experimentations. (that being said, for those who want to push further with a high end card, searching Amazon for "5090" might send you back to Kling.)

When tools or models that generate off of series of frames instead of still images, and loras or models that are either QS specific or allow more directed subject movement will be the next game changer.

I will post some later when I return to QS material, I have been using it in different genres to learn the basics.

Here was my first combined video, this was before I began using the process to prevent multiple frames, the changes between the parts are obvious and abrupt.

PA - Copy.mp4

[JSample]

Your combined video is quite impressive; regarding the abrupt change in the appearance of the quicksand, the sandy surface was obviously just a thin crust over the fluid earth that immediately liquefied as she started sinking.

[BogDog]

Your combined video is quite impressive; regarding the abrupt change in the appearance of the quicksand, the sandy surface was obviously just a thin crust over the fluid earth that immediately liquefied as she started sinking.

Very frustrating when sand or dry-looking mud turns to liquid as they sink. I've produced gritty dry sand and liquid goo but just can't produce a good thick mud or sandy muck that acts like it no matter how much I describe high viscosity or density in prompts with water-like ripples on the surface.

But my biggest issue is random inserting of things I did not specify. I made one video and when I reviewed it I watched the sky turn dark and start showing lightning flashes even though the text made no mention of anything like that or even mention the sky or the weather. Where did that come from?

Sometimes I wonder if random stuff like that is introduced intentionally to force us to do more renders and spend more money to get useful results. Especially when I see things like bubbles on the surface of a sinking medium with happy faces on them appear out of nowhere for no reason. Say WHAT?