Stylized image generation based on multi-attribute decomposition

S Yang and XY Sun and J Guo and K Wang and YZ Ke and XJ Zhang, PATTERN ANALYSIS AND APPLICATIONS, 28, 202 (2025).

DOI: 10.1007/s10044-025-01577-9

The pre-trained latent diffusion model has achieved excellent results in text-to-image generation, providing users with high-quality visual results and encouraging people to use creative text to control the generated images. In order to meet the user's demand for controlling generation details, a common practice is to employ reference images to "stylize" the generating results. Although the "text + single style image" method can help users express their generational needs, this seemingly natural combination masks many problems. The semantic information contained in the describing text and the style characteristics expressed by the reference image are not always harmonious and unified, and conflicts often break out between them. For example, the description text is "color prominence", while the reference image is a modernist concise style with medium tones. This style divergence puts the style transfer model into a dilemma. The key issue is that it is difficult to express the user's style requirements with a single style image, which limits users' control over the generation process at a fine-grained level. Therefore, we are committed to resolving the style conflict between text and style images, enabling users to provide two reference images for style control and to include control information on the attributes of these two style images within the text. Specifically, we propose a multi-attribute decomposition style transfer method, which extracts attribute features from style images and then utilizes a lightweight module to perform feature fusion fine-tuning training. Experimental results demonstrate that our method enables attribute-controllable style generation while maintaining good style alignment with the reference image. The code is available at https://gitee.com/yongzhenke/SIG-MAD.

Return to Publications page