This is an automated archive made by the Lemmit Bot.

The original was posted on /r/stablediffusion by /u/SaladChefs on 2023-09-17 13:26:37.

Stable Diffusion XL (SDXL) Benchmark

Following up from our Whisper-large-v2 benchmark, we recently benchmarked Stable Diffusion XL (SDXL) on consumer GPUs.

The result: 769 hi-res images per dollar.

The images generated were of Salads in the style of famous artists/painters.

We generated 60.6k hi-res images with randomized prompts, on 39 nodes equipped with RTX 3090 and RTX 4090 GPUs. We saw an average image generation time of 15.60s, at a per-image cost of $0.0013.

Architecture

We used an inference container based on SDNext, along with a custom worker written in Typescript that implemented the job processing pipeline. The worker used HTTP to communicate with both the SDNext container and with our batch framework.

Our simple batch processing framework comprises:

  • Storage: Image files stored in AWS S3.
  • Queue System: Jobs queued via AWS SQS, with unique identifiers and pre-signed urls to upload the generated images.
  • Result Storage: After images are generated and uploaded, download urls for each job are stored in DynamoDB.
  • Worker Coordination: We integrated HTTP handlers using AWS Lambda for easy access by workers to the queue and table.

Deployment on SaladCloud

We set up a container group targeting nodes with 4 vCPUs, 32GB of RAM, and GPUs with 24GB of VRAM, which includes the RTX 3090, 3090 ti, and 4090.

We filled a queue with randomized prompts in the following format:

`a ${adjective} ${salad} salad on a ${servingDish} in the style of ${artist}`

We used ChatGPT to generate roughly 100 options for each variable in the prompt, and queued up jobs with 4 images per prompt. SDXL is composed of two models, a base and a refiner. We generated each image at 1216 x 896 resolution, using the base model for 20 steps, and the refiner model for 15 steps. You can see the exact settings we sent to the SDNext API here.

Results – 60,600 Images for $79

For serving SDXL inference at scale, an appropriate measure of cost-efficiency is images per dollar. Popular AI image generation tools serve thousands of images every day, meaning the images per dollar on a cloud is a key to profitable growth.

Here are the images per dollar from five different tools for SDXL inference:

  1. SaladCloud (unoptimized) - 769
  2. AWS - g5.2xlarge (optimized) - 495
  3. Clipdrop API - 100
  4. Stability AI API - 50
  5. AWS - p4d.24x large (optimized) - 37

Over the benchmark period, we generated more than 60k images, uploading more than 90GB of content to our S3 bucket, incurring only $79 in charges from Salad, which is far less expensive than using an A10g on AWS, and orders of magnitude cheaper than fully managed services like the Stability API.

We did see slower image generation times on consumer GPUs than on datacenter GPUs, but the cost differences give Salad the edge. While an optimized model on an A100 did provide the best image generation time, it was by far the most expensive per image of all methods evaluated.

Future Improvements

For comparison with AWS, we gave them several advantages that we did not implement in the container we ran on Salad. In particular, torch.compile isn’t practical on Salad, because it adds 40+ minutes to the container’s start time, and Salad’s nodes are ephemeral.

However, such a long start time might be an acceptable tradeoff in a datacenter context with dedicated nodes that can be expected to stay up for a very long time, so we did use torch.compile on AWS.

Additionally, we used the default fp32 variational autoencoder (vae) in our salad worker, and an fp16 vae in our AWS worker, giving another performance edge to the legacy cloud provider.

Unlike re-compiling the model at start time, including an alternate vae is something that would be practical to do on Salad, and is an optimization we would pursue in future projects.

You can read the full benchmark here (a lot of which has already been discussed here):