Since SR needs to generate new details according to the input and sometimes it may lead to some differences. Cite as: IceClear commented Oct 12, 2023 Hi. You may adjust the pipeline following your settings. yaml at main · IceClear/StableSR May 23, 2023 · Saved searches Use saved searches to filter your results more quickly Dec 12, 2023 · I used to test on eta=0. *image - 1. With tiling, I think 24G is enough. are strided to be 15 timesteps, and the final 100 are strided to be 20. StableSR is capable of achieving arbitrary upscaling in theory, below is a 8x example with a result beyond 4K (5120x3680). per section. For tile operation, we currently use a very large tile by default, see here. process to divide up. And I run the code python scripts/sr [IJCV2024] Exploiting Diffusion Prior for Real-World Image Super-Resolution - IceClear/StableSR First, it normalizes the activations using # your favorite normalization method, such as Batch Norm or Instance Norm. This is used by SD, not StableSR. Model Description: This is the model used in Paper. Labels. Tried to allocate 20. IceClear / StableSR Public. 1 model with the SD-Turbo model， and keep the other steps unchanged. Do you mean that the tile size should be smaller than the target resolution of 1024, in my case？. Crop, you may refer to RealESRGAN to check common settings of SR. First of all, thank you for your amazing open source project I used the following image as my low resolution input, with width and height of 512 and 768 respectively Hi, thanks for the nice work! I see that we need to generate the CFW_trainingdata using the first stage model. DPED-iphone dataset. File metadata and controls. yaml at main · IceClear/StableSR I think this will be handy to use it with diffusers features such as pipeline. 0 is not as good as eta=1. You can show me the results if still does not work. It is recommended to explain which pre-models to download in advance in the Readme. And I am also not sure if other methods like Real-ESRGAN and LDM could meet your needs. May 31, 2024 · Hi! Thank you for your excellent work. ckpt. 0 because of the lack of randomness. Jul 12, 2023 · IceClear / StableSR Public. However it seems like the get_input function of LatentDiffusionSRTextWT class in ddpm. 9k. Could you please share the scripts of generating npy latents and samples? Jul 19, 2023 · I want to train StableSR on my own dataset which contains ground truth and corresponding low quality image pairs. More than 100 million people use GitHub to discover, fork, and contribute to over 420 million projects. :return: a set of diffusion steps from the original process to use. Model type: Diffusion-based image super-resolution model. ckpt and stablesr_000117. txt at main · IceClear/StableSR Hi, thanks for your work. There are no errors or meaningful warnings (see the log below) I am using a GCP notebook to test the inference, which May 11, 2023 · Looking forward to the release of the code, may I ask whether our project can achieve enlargement and add more details,What projects can be implemented If the stride is a string starting with "ddim", then the fixed striding from the DDIM paper is used, and only one section is allowed. The info of LR is introduced by the encoder of autoencoder. But I got all none values after diffusion sample. Jul 11, 2023 · But we found there is performance gap between ours and results reported in the paper. xingjunhong opened this issue on Oct 26, 2023 · 0 comments. 0 and did not see too much quality degradation. IceClear. # Second, it applies scale and bias to the normalized output, conditioned on # the segmentation map. 1base and sd2. ckpt,it would have errors of missing key. [IJCV2024] Exploiting Diffusion Prior for Real-World Image Super-Resolution - StableSR/environment. K. Jul 12, 2023 · I really like the 512 model and thought the 768 would improve things. May 28, 2023 · The math formulation is coherent with the equation we showed in the paper. txt at main · IceClear/StableSR Exploiting Diffusion Prior for Real-World Image Super-Resolution - StableSR/setup. Code; Issues 59; Sign up for a free GitHub account to open an issue and contact its maintainers and May 16, 2023 · IceClear commented on May 16, 2023. 00 GiB total capacity; 3. Second, w is set to 1 during training. Notifications You must be signed in to change notification settings; Fork 116; Star 1. I download pretrained_SDmodel v2-1_512-ema-pruned. ddpm' has no attribute 'LatentDiffusionSRTextWT' Nov 7, 2023 · Hi, I found that a model named face_stablesr_000050. DDIM paper. By clicking “Sign up for GitHub”, Jul 14, 2023 · First, CFW is used for further improve the quality of the latents from diffusion Unet. I tested the provided stablesr model by setting the dec_w to be 0. IceClear closed this as completed on Jun 6. As a special case, use "ddimN" where N. Could you please tell me where to obtain the dataset used for SFT training? Oct 12, 2023 · Saved searches Use saved searches to filter your results more quickly Exploiting Diffusion Prior for Real-World Image Super-Resolution - Issues · IceClear/StableSR [IJCV2024] Exploiting Diffusion Prior for Real-World Image Super-Resolution - IceClear/StableSR May 31, 2023 · Thank you for sharing the code. ckpt ,pretrained_CFWmodel vqgan_cfw_00011. By clicking “Sign up for GitHub”, Jul 12, 2023 · IceClear / StableSR Public. diffusion. 19 GiB total capacity; 31. infer time. Under the default setting, 17G should be enough. #93. Sign up for a free GitHub account to open an issue and contact its maintainers and the community Saved searches Use saved searches to filter your results more quickly Oct 26, 2023 · Hello First of all thanks for this outstanding work but I have some questions to ask as follows: I read in the paper that it is the LR that are converted to latent features and then they are cropped, but in the code I found that it is th Jul 31, 2023 · Following the settings in the code, same config, same dataset, same GPU, and I have carefully chosen the trained ckpt and tested all checkpoints, but it is still worse than the public stablesr_000117. Since the batch size is already 1 for the case without tiling. Open. Jun 30, 2023 · IceClear commented on Jul 3, 2023. SFT layers are inserted in each residual block of Stable Diffusion for effec- tive control. struct_cond model, cond model, diff-unet-model), and constructing an arbitrarily pipeline to make sure the training process could run with other models. Follow. Notifications. Actually, such specific scenario is very challenging for general SR. 0. However the 768 model introduces strange texture patterns such as lines and crosshatches to various surfaces when using the recommended settings such as adding prompts kia350 commented on Jun 5. Jul 7, 2023 · Saved searches Use saved searches to filter your results more quickly Exploiting Diffusion Prior for Real-World Image Super-Resolution - IceClear/StableSR Saved searches Use saved searches to filter your results more quickly [IJCV2024] Exploiting Diffusion Prior for Real-World Image Super-Resolution - StableSR/LICENSE. I just keep it as the original SD. The problem is that my fine-tuning results did not show very good generation ability. py at main · IceClear/StableSR This model card focuses on the models associated with the StableSR, available here. , 5 days to one week. I extend my utmost gratitude to the IceClear team for delivering the remarkable stableSR plugin—an innovation that has considerably amplified the efficiency of my professional Jun 12, 2023 · I'm personally get stuck on the fine-tuning stage. py that you shared in data folder. Notifications Fork 26; Star 634. I have checked "Always save all generated images" in webui, When I finished upscale an image, I can preview it in the webui windows but I couldn't find it in the outputs/img2img-images folder or outputs/extras-images folder. I downloaded it and try it with sd2. IceClear commented May 27, 2023. dataset: Using div2k (DIV2K/DIV2K_valid_HR) and degrade pipeline in your yaml file, we get 3k lr-hr images. If there exists a problem and the second training stage have a significant effect for the texture in IceClear commented Aug 25, 2023 If you would like to initialize X_T from any intermediate steps, you should comment out x_T=None and use q_sample to sample a x_T instead. author = {Wang, Jianyi and Yue, Zongsheng and Zhou, Shangchen and Chan, Kelvin C. However, StableSR uses a diffusion model as a generative prior. #107 opened on Nov 29, 2023 by yongliuy. OutOfMemoryError: CUDA out of memory. You may check the performance of different epochs on the real image. 1? Do you only replace the SD2. Thank you for your outstanding work！. I did observe that the performance of eta=0. I encountered a problem while trying to replicate your code. :param num_timesteps: the number of diffusion steps in the original process to divide up. May 17, 2023 · By default, I hope to obtain an output with a resolution of 4K, but I got: torch. The example image is taken from here. of the original process. You switched accounts on another tab or window. DIV8K), then, in the first training stage, input these LR images (without adding noise) to the VQGAN, Unet, to get the output images, which are compared with GT (HR) images to calculate the loss, and, in the second training stage, input these As a special case, use "ddimN" where N is a number of steps to use the striding from the DDIM paper. This is because you are using a tile size larger than your image size. 1v768, but the results are very bad. startswith ("ddim"): desired_count = int (section_counts [len ("ddim"):]) for i in range (1, num_timesteps): if len Sep 17, 2023 · IceClear commented on Sep 17, 2023. However, I wonder if you had compared the StableSR proposed in your paper with Stable Diffusion x4 Upscaler proposed by StabilityAI. When trying to use the Google Colab demo to infer custom images, I encountered the following error: # Inference the uploaded images #@markdown `DDPM_STEPS`: Number of DDPM steps for sampling<br> DDPM_STEPS = 200 #@param {type:"slider", m May 16, 2023 · You signed in with another tab or window. Hi, I tested the training process with some "ablation tests" by removing each model in the first stage training process (eg. 00 MiB (GPU 0; 4. Resources for more information: GitHub Repository. Reload to refresh your session. It is just for sampling. py is only written according to degradation pipeline of Real Oct 12, 2023 · By the way. Notifications You must be signed in to change notification settings; Fork 117; Star 1. Nov 30, 2023 · DPED-iphone dataset · Issue #107 · IceClear/StableSR · GitHub. 6k. IceClear mentioned this issue on Jun 7. Did you have similar problem as me? Saved searches Use saved searches to filter your results more quickly May 12, 2023 · RuntimeError: CUDA out of memory. You may try --precision full . See here. We further directly test StableSR on AIGC and compared with several diffusion-based upscalers following the suggestions. StableSR Model Card This model card focuses on the models associated with the StableSR, available here. Aug 1, 2023 · I have a few questions regarding the design of StableSR. We did not fine-tune it due to the time limitation. You may turn to text SR methods for solutions. Author. I tested on both one gpu and multiple gpus. alpha + beta Could you explain why in your paper and your code, the modified features is computed by f_hat = f . May 19, 2023 · IceClear commented on May 19, 2023. Also, it is not good at recovering Chinese characters since there is no such data during training. :param section_counts: either a list of numbers, or a string containing comma-separated numbers, indicating the step count I think StableSR without tiling may not support multiple GPUS or it maybe very difficult to achieve. Star 1. 0 and check its fine-tuning results and the results look much better than mine (as shown in issue #36). All reactions Saved searches Use saved searches to filter your results more quickly Oct 26, 2023 · infer time #93. Thanks for your excellent work, which has inprised me a lot. Please support finetune,thanks! @IceClear. No one assigned. I suspect it might be due to issues with how the dataset is constructed. [StbaleSR] in StableSR init As my proficiency in English leaves much to be desired, the following content has been translated by GPT-4. License: S-Lab License 1. If you would like to save GPU memory, you may need to reduce the tile size accordingly (larger than 512). 1 participant. Jan 17, 2024 · Saved searches Use saved searches to filter your results more quickly Exploiting Diffusion Prior for Real-World Image Super-Resolution - IceClear/StableSR IceClear / StableSR Public. [IJCV2024] Exploiting Diffusion Prior for Real-World Image Super-Resolution - StableSR/cog. The main code is as follows. :param conv_resample: if True, use learned convolutions for upsampling and downsampling. cuda. Fangzhenxuan commented on Apr 11. It seems that if I set the ckpt path with stablesr_000117-001. While the SD Upscaler requires training from scratch, once it has been trained, how does its generalization performance in IceClear / StableSR Public. Fork 86. Hi,if I want to finetune on my datasets based on your model. It is my hope that IceClear will comprehend it. No branches or pull requests. From what I understood, the training process is: first, construct LR images by degrading HR images (eg. After all, our focus is on real-world applications. 54 GiB reserved in total by PyTorch) If reserved memory is >> allocated memory try setting max_split_size_mb to avoid fragmentation. Notifications Fork 106; Star 1 Hi, this may happen since we did not specifically train StableSR on text data. And it is also possible to achieve multiple GPU support for tiling, but I am currently too busy IceClear / StableSR Public. Hi, great work! I'm wondering what's the training cost of StableSR (how many days on 8 V100 GPUs)? We find that after the first stage of training on DIV2K， the generation images will lose most texture information which is not expected. :param channel_mult: channel multiplier for each level of the UNet. The speed of A100 is more than 2x than V100, and I do not remember the exact training time of the 512 model. Notifications Fork 98; Star 1. 1 model, I can confirm that this new " Trajectory Consistency Distillation " LoRA model does. " GitHub is where people build software. 242 followers · 88 following. 3k. 62 GiB (GPU 0; 79. Hello, author. May 18, 2023 · Many pre-model models have to be downloaded , which is very unfriendly to our offline testing. I think you may need several days to obtain the stable performance, e. Ph. Sign up for a free GitHub account to open an issue and contact its maintainers and the community Jul 31, 2023 · StableSR is not good at recovering small faces with high fidelity. You may check the name of the parameters, Generally, this is expected since we only load the pre-trained model and parameters from time-aware encoder and SFT are missing. By the time I test For example, if this contains 4, then at 4x downsampling, attention will be used. It is hard to say whether there is a problem. XDDD. :param dropout: the dropout probability. Exploiting Diffusion Prior for Real-World Image Super-Resolution - StableSR/requirements. I update the codes for inferencing images with latest sd2. Sign up for a free GitHub account to open an issue and contact its maintainers and the community. IceClear commented Mar 24, 2024 FYI, while it was said the SD-Turbo model does -not- work with the 768 resolution SD2. I saw in SFT or SPADE papers, they modified the feature map by f_hat = f . StableSR sometimes may not show 100% fidelity given the powerful generation capability of the fixed Stable Diffusion. I used the dataloader named paired_image_dataset. Jul 6, 2023 · IceClear commented on Jul 6, 2023. 25 GiB already allocated; 14. Hi~ thanks for your interest of our work. Notifications Fork 74; Star 1. I may change the implementation in the future to avoid misunderstanding. And I tried training longer but it didn't work, but more blurry! Saved searches Use saved searches to filter your results more quickly Oct 24, 2023 · IceClear / StableSR Public. return 2. The tile operation may lead to border artifacts, so Jan 23, 2024 · Thank you for your excellent work, I believe I will learn a lot from it. [IJCV2024] Exploiting Diffusion Prior for Real-World Image Super-Resolution - Issues · IceClear/StableSR. And I try to train the model from scratch following your train script and config, everything is same except the DIV8k dataset(i don't have DIV8k). I want to test StableSR using your test_example, the result is a black image. :param num Jan 18, 2024 · hello ，I am encountering some issues: AttributeError: module 'ldm. Thanks for your time and help! Our time-aware encoder is similar to the contracting path of the denoising U-Net in Stable Diffusion but is much more lightweight (∼105M, including SFT layers). candidate from MMLab@NTU, mainly focusing on low-level computer vision. I've acquired the inputs and gts, and latents and samples are also saved locally. Projects. I noticed a significant discrepancy between the FID results I obtained and the FID reported in the paper. You signed out in another tab or window. By clicking “Sign up for GitHub”, Nov 8, 2023 · IceClear / StableSR Public. Model Details Developed by: Jianyi Wang. 👍 1. The example image is taken from here . I tried to run the inference by using the steps in the ReadMe, but my terminal gets stuck as if it was in an infinite loop just after the print "making attention of type 'vanilla' with 512 in_channels". None yet. 41 GiB reserved in total by PyTorch) If reserved memory is >> allocated memory try setting max_split_size_mb to Feb 21, 2024 · Saved searches Use saved searches to filter your results more quickly XuejiFang commented on Nov 8. models. Are the RealSR, DRealSR, and DPED-iphone test datasets evaluated after retraining on the corresponding original training datasets, or are they directly tested on these datasets after training on datasets such as DIV2K as mentioned in the paper to evaluate generalization of proposed method? Owner. A 4K demo is here, which is a 4x SR on the image from here. D. 4k. I am very interested in the work with SD-Turbo. g. You may try to use different seeds to check if the artifacts always exist. Based on our own visualization. You may have a try if you would like to. It seems that the design of StableSR is similar to GLEAN, which uses a generative latent bank for image super-resolution. 1. Sign up for a free GitHub account to open an issue and contact its maintainers and the community IceClear / StableSR Public. You can see the tile size is 1280 (in pixel space), and the stride is 1000. 1-768v model. . 43 GiB already allocated; 0 bytes free; 3. 34 GiB free; 39. May I ask if its training is the same as SD-2. and Loy, Chen Change}, Jianyi Wang. We may add this detail in the paper. We finetune the diffusion model of StableSR for 117 epochs Nov 10, 2023 · Development. Sign up for GitHub Nov 25, 2023 · Hello Thank you for your work. Notifications Fork 81; Star 1. from the DDIM paper is used, and only one section is allowed. To associate your repository with the stablesr topic, visit your repo's landing page and select "manage topics. By clicking “Sign up for GitHub”, Jan 15, 2024 · IceClear commented Jan 15, 2024 We observe improvements in natural images under severe degradations in such way, but not sure if it works for face SR. The performance may vary for different epochs. Moreover, our current training data does not contain Chinese characters, and I am not sure if Stable Diffusion contains such prior. I tested your command and no problem was found. """ if isinstance (section_counts, str): if section_counts. Notifications You must be signed in to change notification settings; Fork 121; Star 2k. ckpt is uploaded to huggingface model page recently, but there is no any information about how to use this model. All reactions Add this topic to your repo. :param dims: determines if the signal is 1D, 2D, or 3D. The text was updated successfully, but these errors were encountered: Assignees. We just followed RealESRGAN to generate data. Tried to allocate 55. Hello author, I'm trying to fine-tune CFW module using your guidelines to prepare CFW training data. Additionally, the design of the decoder in StableSR differs slightly from that of GLEAN. Thanks. [IJCV2024] Exploiting Diffusion Prior for Real-World Image Super-Resolution - IceClear/StableSR StableSR is a generic image SR method by efficiently leveraging prior encapsulated in Stable Diffusion. The text was updated successfully, but these errors were encountered: Development. IceClear changed the title StableSR distorts text in image StableSR distorts Chinese Jul 3, 2023 · IceClear / StableSR Public. But the performance should not be too bad. xy cf ix cm nh ub xc sp bu uq