{"id":399015,"date":"2024-06-29T14:25:02","date_gmt":"2024-06-29T14:25:02","guid":{"rendered":"http:\/\/savepearlharbor.com\/?p=399015"},"modified":"-0001-11-30T00:00:00","modified_gmt":"2024-06-29T14:25:02","slug":"","status":"publish","type":"post","link":"https:\/\/savepearlharbor.com\/?p=399015","title":{"rendered":"<span>ruDALL-E: Generating Images from Text. Facing down the biggest computational challenge in Russia<\/span>"},"content":{"rendered":"<div><!--[--><!--]--><\/div>\n<div id=\"post-content-body\">\n<div>\n<div class=\"article-formatted-body article-formatted-body article-formatted-body_version-2\">\n<div xmlns=\"http:\/\/www.w3.org\/1999\/xhtml\">\n<p>In this article we present the ruDALL-E XL model, an <a href=\"https:\/\/github.com\/sberbank-ai\/ru-dalle\">open-source<\/a> text-to-image transformer with 1.3 billion parameters as well as ruDALL-E XXL model, an text-to-image transformer with 12.0 billion parameters which is available in <a href=\"https:\/\/mlspace.aicloud.sbercloud.ru\/mlspace\/datahub\">DataHub<\/a> SberCloud, and several other satellite models.<\/p>\n<ul>\n<li>\n<p>Free demo available: <a href=\"https:\/\/rudalle.ru\/en\/\"><u>https:\/\/rudalle.ru\/en\/<\/u><\/a><\/p>\n<\/li>\n<li>\n<p>Free <a href=\"https:\/\/t.me\/sber_rudalle_xl_bot\"><u>Telegram bot<\/u><\/a><\/p>\n<\/li>\n<li>\n<p>Better and faster generation is available with the Salute App: <a href=\"https:\/\/sberdevices.ru\/app\/\"><u>https:\/\/sberdevices.ru\/app\/<\/u><\/a>\u00a0<\/p>\n<\/li>\n<li>\n<p><a href=\"https:\/\/www.instagram.com\/rudalle.official\/\"><u>Instagram<\/u><\/a> with examples of the best generations<\/p>\n<\/li>\n<li>\n<p><a href=\"https:\/\/github.com\/sberbank-ai\/ru-dalle\"><u>GitHub<\/u><\/a> with ruDALL-E XL<\/p>\n<\/li>\n<\/ul>\n<p>Our demo now works with automatic translation from 107 languages:<\/p>\n<figure class=\"full-width\"><img loading=\"lazy\" decoding=\"async\" src=\"https:\/\/habrastorage.org\/r\/w1560\/getpro\/habr\/upload_files\/9f6\/875\/82e\/9f687582e53cee03380f384167daeb9e.png\" width=\"998\" height=\"821\" data-src=\"https:\/\/habrastorage.org\/getpro\/habr\/upload_files\/9f6\/875\/82e\/9f687582e53cee03380f384167daeb9e.png\"\/><figcaption><\/figcaption><\/figure>\n<h2>Motivation<\/h2>\n<p>Multimodality has led the pack in machine learning in 2021. Neural networks are wolfing down images, text, speech and music all at the same time.\u00a0 OpenAI is, as usual, top dog, but as if in defiance of their name, they are in no hurry to share their models openly.\u00a0 At the beginning of the year, the company presented the DALL-E neural network, which generates 256&#215;256 pixel images in answer to a written request.\u00a0 Descriptions of it can be found as <a href=\"https:\/\/arxiv.org\/pdf\/2102.12092.pdf\"><u>articles<\/u><\/a> on arXiv and examples on their <a href=\"https:\/\/openai.com\/blog\/dall-e\/\"><u>blog<\/u><\/a>.\u00a0\u00a0<\/p>\n<p>As soon as DALL-E flushed out of the bushes, Chinese researchers got on its tail.\u00a0 Their open-source <a href=\"https:\/\/github.com\/THUDM\/CogView\"><u>CogView<\/u><\/a> neural network does the same trick of generating images from text.\u00a0 But what about here in Russia? One might say that \u201cinvestigate, master, and train\u201d is our engineering motto.\u00a0 Well, we caught the scent, and today we can say that we created from scratch a complete pipeline for generating images from descriptive textual input written in Russian.<\/p>\n<p>Teams at Sber AI, SberDevices, Samara University, AIRI and SberCloud all actively contributed. We trained two versions of the model, each a different size:<\/p>\n<ol>\n<li>\n<p>ruDALL-E XXL, with 12.0 billion parameters<\/p>\n<\/li>\n<li>\n<p>ruDALL-E XL, having 1.3 billion parameters<\/p>\n<\/li>\n<\/ol>\n<p>Some of our models are already freely available:<\/p>\n<ol>\n<li>\n<p>ruDALL-E XL [<a href=\"https:\/\/github.com\/sberbank-ai\/ru-dalle\"><u>GitHub<\/u><\/a>, <a href=\"https:\/\/huggingface.co\/sberbank-ai\/rudalle-Malevich\"><u>HuggingFace<\/u><\/a><u>,\u00a0<\/u><a href=\"https:\/\/www.kaggle.com\/shonenkov\/rudalle-example-generation\"><u>Kaggle<\/u><\/a>]<\/p>\n<\/li>\n<li>\n<p>Sber VQ-GAN [<a href=\"https:\/\/github.com\/sberbank-ai\/sber-vq-gan\"><u>GitHub<\/u><\/a>, <a href=\"https:\/\/huggingface.co\/sberbank-ai\/Sber-VQGAN\"><u>HuggingFace<\/u><\/a>]<\/p>\n<\/li>\n<li>\n<p>ruCLIP Small [<a href=\"https:\/\/github.com\/sberbank-ai\/ru-clip\"><u>GitHub<\/u><\/a>, <a href=\"https:\/\/huggingface.co\/sberbank-ai\/ru-clip\"><u>HuggingFace<\/u><\/a>]<\/p>\n<\/li>\n<li>\n<p>Super Resolution (Real ESRGAN) [<a href=\"https:\/\/github.com\/sberbank-ai\/Real-ESRGAN\"><u>GitHub<\/u><\/a>, <a href=\"https:\/\/huggingface.co\/sberbank-ai\/Real-ESRGAN\"><u>HuggingFace<\/u><\/a>]<\/p>\n<\/li>\n<\/ol>\n<p>The latter two models are included in the pipeline for generating images from text (as you\u2019ll see later on). <\/p>\n<p>The models ruDALL-E XL, ruDALL-E XXL, and ruCLIP Small are now available on <a href=\"https:\/\/mlspace.aicloud.sbercloud.ru\/mlspace\/datahub\"><u>DataHub<\/u><\/a>. The models ruCLIP Large and Super Resolution (Real ESRGAN) will also be available soon there.<\/p>\n<p><strong>Training the ruDALL-E neural networks on the <\/strong><a href=\"https:\/\/mlspace.aicloud.sbercloud.ru\/\"><strong><u>Christofari<\/u><\/strong><\/a><strong> cluster has become the largest calculation task in Russia:<\/strong><\/p>\n<ol>\n<li>\n<p>ruDALL-E XXL was trained for 37 days on the 512 GPU TESLA V100, and then also for 11 more days on the 128 GPU TESLA V100, for a total of 20,352 GPU-days;<\/p>\n<\/li>\n<li>\n<p>ruDALL-E XL was trained for 8 days on the 128 GPU TESLA V100, and then also for 15 more days on the 192 GPU TESLA V100, for a total of 3,904 GPU-days.<\/p>\n<\/li>\n<\/ol>\n<p>Accordingly, training for both models totaled <strong>24,256 GPU-days<\/strong>.<\/p>\n<p>Now let\u2019s explore what our generative models are capable of.<\/p>\n<figure class=\"full-width\"><img loading=\"lazy\" decoding=\"async\" src=\"https:\/\/habrastorage.org\/r\/w1560\/getpro\/habr\/upload_files\/4f4\/6d5\/b47\/4f46d5b47366c7b9bfe717729bba8c75.png\" alt=\"Images generated by ruDALL-E XL from the input &quot;\u041e\u0437\u0435\u0440\u043e \u0432 \u0433\u043e\u0440\u0430\u0445, \u0430 \u0440\u044f\u0434\u043e\u043c \u043a\u0440\u0430\u0441\u0438\u0432\u044b\u0439 \u043e\u043b\u0435\u043d\u044c \u043f\u044c\u0451\u0442 \u0432\u043e\u0434\u0443&quot; (\u201cA lake in the mountains, with a beautiful deer drinking water nearby\u201d)\" title=\"Images generated by ruDALL-E XL from the input &quot;\u041e\u0437\u0435\u0440\u043e \u0432 \u0433\u043e\u0440\u0430\u0445, \u0430 \u0440\u044f\u0434\u043e\u043c \u043a\u0440\u0430\u0441\u0438\u0432\u044b\u0439 \u043e\u043b\u0435\u043d\u044c \u043f\u044c\u0451\u0442 \u0432\u043e\u0434\u0443&quot; (\u201cA lake in the mountains, with a beautiful deer drinking water nearby\u201d)\" width=\"799\" height=\"538\" data-src=\"https:\/\/habrastorage.org\/getpro\/habr\/upload_files\/4f4\/6d5\/b47\/4f46d5b47366c7b9bfe717729bba8c75.png\"\/><figcaption>Images generated by ruDALL-E XL from the input &#171;\u041e\u0437\u0435\u0440\u043e \u0432 \u0433\u043e\u0440\u0430\u0445, \u0430 \u0440\u044f\u0434\u043e\u043c \u043a\u0440\u0430\u0441\u0438\u0432\u044b\u0439 \u043e\u043b\u0435\u043d\u044c \u043f\u044c\u0451\u0442 \u0432\u043e\u0434\u0443&#187; (\u201cA lake in the mountains, with a beautiful deer drinking water nearby\u201d)<\/figcaption><\/figure>\n<h2>Why is Big Tech doing image generation?<\/h2>\n<p>The long term goal of this research is the creation of multimodal neural networks. They will be able to pull on concepts from a variety of mediums (from text and visuals at first) in order to better understand the world as a whole.<\/p>\n<p>Image generation might seem like the wrong rabbit hole in our century of big data and search engines.\u00a0 But it actually addresses two important requirements that search is currently unable to cope with:<\/p>\n<ol>\n<li>\n<p>Being able to describe in writing exactly what you\u2019re looking for and getting a completely new image created personally for you.<\/p>\n<\/li>\n<li>\n<p>Being able to create at any time as many licence-free illustrations as you could possibly want<\/p>\n<\/li>\n<\/ol>\n<p>The most obvious uses for image generation are:<\/p>\n<ul>\n<li>\n<p>Illustrative photos for journalists, copywriters and advertisers. You can create imagery for articles automatically (meaning also more cheaply and quickly), and generate concept-boards for ads simply from a written description.\u00a0Some examples:<\/p>\n<\/li>\n<\/ul>\n<figure class=\"full-width\"><img loading=\"lazy\" decoding=\"async\" src=\"https:\/\/habrastorage.org\/r\/w1560\/getpro\/habr\/upload_files\/0be\/bd5\/226\/0bebd5226374acd74d97b39cceea6827.png\" alt=\"&quot;\u041b\u0438\u0441\u0430 \u0432 \u043b\u0435\u0441\u0443&quot; (\u201cFox in the forest\u201d)\" title=\"&quot;\u041b\u0438\u0441\u0430 \u0432 \u043b\u0435\u0441\u0443&quot; (\u201cFox in the forest\u201d)\" width=\"1024\" height=\"1024\" data-src=\"https:\/\/habrastorage.org\/getpro\/habr\/upload_files\/0be\/bd5\/226\/0bebd5226374acd74d97b39cceea6827.png\"\/><figcaption>&#171;\u041b\u0438\u0441\u0430 \u0432 \u043b\u0435\u0441\u0443&#187; (\u201cFox in the forest\u201d)<\/figcaption><\/figure>\n<figure class=\"full-width\"><img loading=\"lazy\" decoding=\"async\" src=\"https:\/\/habrastorage.org\/r\/w1560\/getpro\/habr\/upload_files\/f76\/975\/ff1\/f76975ff169bd49d313ad61203f4abd3.png\" alt=\"&quot;\u041e\u0440\u0435\u043b \u0441\u0438\u0434\u0438\u0442 \u043d\u0430 \u0434\u0435\u0440\u0435\u0432\u0435, \u0432\u0438\u0434 \u0441\u0431\u043e\u043a\u0443&quot; (\u201cAn eagle sits in a tree, looking to the side\u201d)\" title=\"&quot;\u041e\u0440\u0435\u043b \u0441\u0438\u0434\u0438\u0442 \u043d\u0430 \u0434\u0435\u0440\u0435\u0432\u0435, \u0432\u0438\u0434 \u0441\u0431\u043e\u043a\u0443&quot; (\u201cAn eagle sits in a tree, looking to the side\u201d)\" width=\"1024\" height=\"1024\" data-src=\"https:\/\/habrastorage.org\/getpro\/habr\/upload_files\/f76\/975\/ff1\/f76975ff169bd49d313ad61203f4abd3.png\"\/><figcaption>&#171;\u041e\u0440\u0435\u043b \u0441\u0438\u0434\u0438\u0442 \u043d\u0430 \u0434\u0435\u0440\u0435\u0432\u0435, \u0432\u0438\u0434 \u0441\u0431\u043e\u043a\u0443&#187; (\u201cAn eagle sits in a tree, looking to the side\u201d)<\/figcaption><\/figure>\n<figure class=\"full-width\"><img loading=\"lazy\" decoding=\"async\" src=\"https:\/\/habrastorage.org\/r\/w1560\/getpro\/habr\/upload_files\/e93\/ba2\/12a\/e93ba212a962acf54aa44bdc56a7c508.png\" alt=\"&quot;\u0410\u0432\u0442\u043e\u043c\u043e\u0431\u0438\u043b\u044c \u043d\u0430 \u0434\u043e\u0440\u043e\u0433\u0435 \u0441\u0440\u0435\u0434\u0438 \u043a\u0440\u0430\u0441\u0438\u0432\u044b\u0445 \u0433\u043e\u0440&quot; (\u201cA car on a road surrounded by beautiful mountains\u201d)\" title=\"&quot;\u0410\u0432\u0442\u043e\u043c\u043e\u0431\u0438\u043b\u044c \u043d\u0430 \u0434\u043e\u0440\u043e\u0433\u0435 \u0441\u0440\u0435\u0434\u0438 \u043a\u0440\u0430\u0441\u0438\u0432\u044b\u0445 \u0433\u043e\u0440&quot; (\u201cA car on a road surrounded by beautiful mountains\u201d)\" width=\"1000\" height=\"338\" data-src=\"https:\/\/habrastorage.org\/getpro\/habr\/upload_files\/e93\/ba2\/12a\/e93ba212a962acf54aa44bdc56a7c508.png\"\/><figcaption>&#171;\u0410\u0432\u0442\u043e\u043c\u043e\u0431\u0438\u043b\u044c \u043d\u0430 \u0434\u043e\u0440\u043e\u0433\u0435 \u0441\u0440\u0435\u0434\u0438 \u043a\u0440\u0430\u0441\u0438\u0432\u044b\u0445 \u0433\u043e\u0440&#187; (\u201cA car on a road surrounded by beautiful mountains\u201d)<\/figcaption><\/figure>\n<ul>\n<li>\n<p>An artwork, completely license-free, can also be created ad infinitum:<\/p>\n<figure class=\"full-width\"><img loading=\"lazy\" decoding=\"async\" src=\"https:\/\/habrastorage.org\/r\/w1560\/getpro\/habr\/upload_files\/d76\/31d\/d62\/d7631dd623c03051d885d35df93a0deb.png\" alt=\"&quot;\u0412\u0435\u043a\u0442\u043e\u0440\u043d\u0430\u044f \u0438\u043b\u043b\u044e\u0441\u0442\u0440\u0430\u0446\u0438\u044f \u0441 \u0440\u043e\u0437\u043e\u0432\u044b\u043c\u0438 \u0446\u0432\u0435\u0442\u0430\u043c\u0438&quot; (\u201cVector artwork with pink flowers\u201d)\" title=\"&quot;\u0412\u0435\u043a\u0442\u043e\u0440\u043d\u0430\u044f \u0438\u043b\u043b\u044e\u0441\u0442\u0440\u0430\u0446\u0438\u044f \u0441 \u0440\u043e\u0437\u043e\u0432\u044b\u043c\u0438 \u0446\u0432\u0435\u0442\u0430\u043c\u0438&quot; (\u201cVector artwork with pink flowers\u201d)\" width=\"1000\" height=\"339\" data-src=\"https:\/\/habrastorage.org\/getpro\/habr\/upload_files\/d76\/31d\/d62\/d7631dd623c03051d885d35df93a0deb.png\"\/><figcaption>&#171;\u0412\u0435\u043a\u0442\u043e\u0440\u043d\u0430\u044f \u0438\u043b\u043b\u044e\u0441\u0442\u0440\u0430\u0446\u0438\u044f \u0441 \u0440\u043e\u0437\u043e\u0432\u044b\u043c\u0438 \u0446\u0432\u0435\u0442\u0430\u043c\u0438&#187; (\u201cVector artwork with pink flowers\u201d)<\/figcaption><\/figure>\n<\/li>\n<li>\n<p>Interior design mock-ups.\u00a0You can do a quick visual check of your ideas for a remodel, and play with colors, form, and light:<\/p>\n<\/li>\n<\/ul>\n<figure class=\"full-width\"><img loading=\"lazy\" decoding=\"async\" src=\"https:\/\/habrastorage.org\/r\/w1560\/getpro\/habr\/upload_files\/7ce\/b00\/da1\/7ceb00da1be3227c229b35ba2e7826bf.png\" alt=\"&quot;\u0428\u0438\u043a\u0430\u0440\u043d\u0430\u044f \u0433\u043e\u0441\u0442\u0438\u043d\u0430\u044f \u0441 \u0437\u0435\u043b\u0435\u043d\u044b\u043c\u0438 \u043a\u0440\u0435\u0441\u043b\u0430\u043c\u0438&quot; (\u201cElegant living room with green stuffed chairs\u201d)\" title=\"&quot;\u0428\u0438\u043a\u0430\u0440\u043d\u0430\u044f \u0433\u043e\u0441\u0442\u0438\u043d\u0430\u044f \u0441 \u0437\u0435\u043b\u0435\u043d\u044b\u043c\u0438 \u043a\u0440\u0435\u0441\u043b\u0430\u043c\u0438&quot; (\u201cElegant living room with green stuffed chairs\u201d)\" width=\"1000\" height=\"337\" data-src=\"https:\/\/habrastorage.org\/getpro\/habr\/upload_files\/7ce\/b00\/da1\/7ceb00da1be3227c229b35ba2e7826bf.png\"\/><figcaption>&#171;\u0428\u0438\u043a\u0430\u0440\u043d\u0430\u044f \u0433\u043e\u0441\u0442\u0438\u043d\u0430\u044f \u0441 \u0437\u0435\u043b\u0435\u043d\u044b\u043c\u0438 \u043a\u0440\u0435\u0441\u043b\u0430\u043c\u0438&#187; (\u201cElegant living room with green stuffed chairs\u201d)<\/figcaption><\/figure>\n<figure class=\"full-width\"><img loading=\"lazy\" decoding=\"async\" src=\"https:\/\/habrastorage.org\/r\/w1560\/getpro\/habr\/upload_files\/f06\/ea8\/83b\/f06ea883b90db4f90db29bbfaeac49ee.png\" alt=\"&quot;\u0421\u043e\u0432\u0440\u0435\u043c\u0435\u043d\u043d\u043e\u0435 \u043a\u0440\u0435\u0441\u043b\u043e \u0444\u0438\u043e\u043b\u0435\u0442\u043e\u0432\u043e\u0433\u043e \u0446\u0432\u0435\u0442\u0430&quot; (\u201cContemporary purple armchair\u201d)\" title=\"&quot;\u0421\u043e\u0432\u0440\u0435\u043c\u0435\u043d\u043d\u043e\u0435 \u043a\u0440\u0435\u0441\u043b\u043e \u0444\u0438\u043e\u043b\u0435\u0442\u043e\u0432\u043e\u0433\u043e \u0446\u0432\u0435\u0442\u0430&quot; (\u201cContemporary purple armchair\u201d)\" width=\"1000\" height=\"511\" data-src=\"https:\/\/habrastorage.org\/getpro\/habr\/upload_files\/f06\/ea8\/83b\/f06ea883b90db4f90db29bbfaeac49ee.png\"\/><figcaption>&#171;\u0421\u043e\u0432\u0440\u0435\u043c\u0435\u043d\u043d\u043e\u0435 \u043a\u0440\u0435\u0441\u043b\u043e \u0444\u0438\u043e\u043b\u0435\u0442\u043e\u0432\u043e\u0433\u043e \u0446\u0432\u0435\u0442\u0430&#187; (\u201cContemporary purple armchair\u201d)<\/figcaption><\/figure>\n<ul>\n<li>\n<p>Visual art.\u00a0A source of visual concepts combining various attributes with abstraction:<\/p>\n<figure class=\"full-width\"><img loading=\"lazy\" decoding=\"async\" src=\"https:\/\/habrastorage.org\/r\/w1560\/getpro\/habr\/upload_files\/a57\/5a3\/b1d\/a575a3b1d04ce6370849fa406bb0dce4.png\" alt=\"&quot;\u0422\u0435\u043c\u043d\u0430\u044f \u044d\u043d\u0435\u0440\u0433\u0438\u044f&quot; (\u201cDark energy\u201d)\" title=\"&quot;\u0422\u0435\u043c\u043d\u0430\u044f \u044d\u043d\u0435\u0440\u0433\u0438\u044f&quot; (\u201cDark energy\u201d)\" width=\"1024\" height=\"1024\" data-src=\"https:\/\/habrastorage.org\/getpro\/habr\/upload_files\/a57\/5a3\/b1d\/a575a3b1d04ce6370849fa406bb0dce4.png\"\/><figcaption>&#171;\u0422\u0435\u043c\u043d\u0430\u044f \u044d\u043d\u0435\u0440\u0433\u0438\u044f&#187; (\u201cDark energy\u201d)<\/figcaption><\/figure>\n<\/li>\n<\/ul>\n<figure class=\"full-width\"><img loading=\"lazy\" decoding=\"async\" src=\"https:\/\/habrastorage.org\/r\/w1560\/getpro\/habr\/upload_files\/0d6\/6e9\/6c7\/0d66e96c7d2aa7873c57bac5e167f090.png\" alt=\"&quot;\u041a\u043e\u0442 \u043d\u0430 \u041b\u0443\u043d\u0435&quot; (\u201cCat on the moon\u201d)\" title=\"&quot;\u041a\u043e\u0442 \u043d\u0430 \u041b\u0443\u043d\u0435&quot; (\u201cCat on the moon\u201d)\" width=\"1024\" height=\"1024\" data-src=\"https:\/\/habrastorage.org\/getpro\/habr\/upload_files\/0d6\/6e9\/6c7\/0d66e96c7d2aa7873c57bac5e167f090.png\"\/><figcaption>&#171;\u041a\u043e\u0442 \u043d\u0430 \u041b\u0443\u043d\u0435&#187; (\u201cCat on the moon\u201d)<\/figcaption><\/figure>\n<figure class=\"full-width\"><img loading=\"lazy\" decoding=\"async\" src=\"https:\/\/habrastorage.org\/r\/w1560\/getpro\/habr\/upload_files\/8fb\/5e7\/b61\/8fb5e7b6191d19eefc43ab42858b9bf0.png\" alt=\"&quot;\u041a\u043e\u0448\u043a\u0430 \u043a\u043e\u0442\u043e\u0440\u0430\u044f \u0441\u0434\u0435\u043b\u0430\u043d\u0430 \u0438\u0437 \u0431\u0435\u043b\u043e\u0433\u043e \u043e\u0431\u043b\u0430\u043a\u0430&quot; (\u201cA cat made out of white cloud\u201d)\" title=\"&quot;\u041a\u043e\u0448\u043a\u0430 \u043a\u043e\u0442\u043e\u0440\u0430\u044f \u0441\u0434\u0435\u043b\u0430\u043d\u0430 \u0438\u0437 \u0431\u0435\u043b\u043e\u0433\u043e \u043e\u0431\u043b\u0430\u043a\u0430&quot; (\u201cA cat made out of white cloud\u201d)\" width=\"1000\" height=\"498\" data-src=\"https:\/\/habrastorage.org\/getpro\/habr\/upload_files\/8fb\/5e7\/b61\/8fb5e7b6191d19eefc43ab42858b9bf0.png\"\/><figcaption>&#171;\u041a\u043e\u0448\u043a\u0430 \u043a\u043e\u0442\u043e\u0440\u0430\u044f \u0441\u0434\u0435\u043b\u0430\u043d\u0430 \u0438\u0437 \u0431\u0435\u043b\u043e\u0433\u043e \u043e\u0431\u043b\u0430\u043a\u0430&#187; (\u201cA cat made out of white cloud\u201d)<\/figcaption><\/figure>\n<figure class=\"full-width\"><img loading=\"lazy\" decoding=\"async\" src=\"https:\/\/habrastorage.org\/r\/w1560\/getpro\/habr\/upload_files\/9d3\/285\/866\/9d3285866f0a287e613968c2bdcb9e2e.png\" alt=\"&quot;\u0415\u043d\u043e\u0442 \u0441 \u043f\u0443\u0448\u043a\u043e\u0439&quot; (\u201cRaccoon with gun\u201d)\" title=\"&quot;\u0415\u043d\u043e\u0442 \u0441 \u043f\u0443\u0448\u043a\u043e\u0439&quot; (\u201cRaccoon with gun\u201d)\" width=\"1000\" height=\"502\" data-src=\"https:\/\/habrastorage.org\/getpro\/habr\/upload_files\/9d3\/285\/866\/9d3285866f0a287e613968c2bdcb9e2e.png\"\/><figcaption>&#171;\u0415\u043d\u043e\u0442 \u0441 \u043f\u0443\u0448\u043a\u043e\u0439&#187; (\u201cRaccoon with gun\u201d)<\/figcaption><\/figure>\n<figure class=\"full-width\"><img loading=\"lazy\" decoding=\"async\" src=\"https:\/\/habrastorage.org\/r\/w1560\/getpro\/habr\/upload_files\/637\/679\/882\/637679882bb72bb5b809a1c20db0c1a4.png\" alt=\"&quot;\u041a\u0440\u0430\u0441\u0438\u0432\u043e\u0435 \u043e\u0437\u0435\u0440\u043e \u043d\u0430 \u0437\u0430\u043a\u0430\u0442\u0435&quot; (\u201cPretty lake at sunset\u201d)\" title=\"&quot;\u041a\u0440\u0430\u0441\u0438\u0432\u043e\u0435 \u043e\u0437\u0435\u0440\u043e \u043d\u0430 \u0437\u0430\u043a\u0430\u0442\u0435&quot; (\u201cPretty lake at sunset\u201d)\" width=\"1000\" height=\"500\" data-src=\"https:\/\/habrastorage.org\/getpro\/habr\/upload_files\/637\/679\/882\/637679882bb72bb5b809a1c20db0c1a4.png\"\/><figcaption>&#171;\u041a\u0440\u0430\u0441\u0438\u0432\u043e\u0435 \u043e\u0437\u0435\u0440\u043e \u043d\u0430 \u0437\u0430\u043a\u0430\u0442\u0435&#187; (\u201cPretty lake at sunset\u201d)<\/figcaption><\/figure>\n<figure class=\"full-width\"><img loading=\"lazy\" decoding=\"async\" src=\"https:\/\/habrastorage.org\/r\/w780q1\/getpro\/habr\/upload_files\/217\/c2c\/b24\/217c2cb24ae7411fe457070b47c8fbe8.jpg\" alt=\"&quot;\u0420\u0430\u0434\u0443\u0436\u043d\u0430\u044f \u0441\u043e\u0432\u0430&quot; (&quot;Rainbow owl&quot;)\" title=\"&quot;\u0420\u0430\u0434\u0443\u0436\u043d\u0430\u044f \u0441\u043e\u0432\u0430&quot; (&quot;Rainbow owl&quot;)\" width=\"640\" height=\"640\" data-src=\"https:\/\/habrastorage.org\/getpro\/habr\/upload_files\/217\/c2c\/b24\/217c2cb24ae7411fe457070b47c8fbe8.jpg\" data-blurred=\"true\"\/><figcaption>&#171;\u0420\u0430\u0434\u0443\u0436\u043d\u0430\u044f \u0441\u043e\u0432\u0430&#187; (&#171;Rainbow owl&#187;)<\/figcaption><\/figure>\n<figure class=\"full-width\"><img loading=\"lazy\" decoding=\"async\" src=\"https:\/\/habrastorage.org\/r\/w780q1\/getpro\/habr\/upload_files\/435\/3a7\/490\/4353a749064d5237b0396fbf99c1cda3.jpg\" alt=\"&quot;\u0416\u0434\u0443\u043d \u0441 \u0430\u0432\u043e\u043a\u0430\u0434\u043e&quot; (&quot;Zhdun with avocado&quot;)\" title=\"&quot;\u0416\u0434\u0443\u043d \u0441 \u0430\u0432\u043e\u043a\u0430\u0434\u043e&quot; (&quot;Zhdun with avocado&quot;)\" width=\"1024\" height=\"1024\" data-src=\"https:\/\/habrastorage.org\/getpro\/habr\/upload_files\/435\/3a7\/490\/4353a749064d5237b0396fbf99c1cda3.jpg\" data-blurred=\"true\"\/><figcaption>&#171;\u0416\u0434\u0443\u043d \u0441 \u0430\u0432\u043e\u043a\u0430\u0434\u043e&#187; (&#171;Zhdun with avocado&#187;)<\/figcaption><\/figure>\n<h2>Model Architecture<\/h2>\n<p>The architectural foundation of DALL-E, the so-called <em>transformer<\/em>, is an encoder-decoder model.\u00a0It works in two steps.\u00a0First, the encoder needs to calculate an embedding from the input.\u00a0Then the decoder deciphers the resulting vector in view of a known target so that the output matches expectations.\u00a0 <\/p>\n<figure class=\"full-width\"><img loading=\"lazy\" decoding=\"async\" src=\"https:\/\/habrastorage.org\/r\/w1560\/getpro\/habr\/upload_files\/5b0\/025\/710\/5b00257106f0fd747bee5e4b765fe0fd.png\" alt=\"Top-level scheme of the vanilla transformer\" title=\"Top-level scheme of the vanilla transformer\" width=\"756\" height=\"474\" data-src=\"https:\/\/habrastorage.org\/getpro\/habr\/upload_files\/5b0\/025\/710\/5b00257106f0fd747bee5e4b765fe0fd.png\"\/><figcaption>Top-level scheme of the vanilla transformer<\/figcaption><\/figure>\n<figure class=\"full-width\"><img loading=\"lazy\" decoding=\"async\" src=\"https:\/\/habrastorage.org\/r\/w1560\/getpro\/habr\/upload_files\/b9b\/60a\/04a\/b9b60a04a3be8009bda85b055dd7ea55.png\" alt=\"A slightly more detailed scheme of the &quot;vanilla&quot; transformer\" title=\"A slightly more detailed scheme of the &quot;vanilla&quot; transformer\" width=\"1218\" height=\"793\" data-src=\"https:\/\/habrastorage.org\/getpro\/habr\/upload_files\/b9b\/60a\/04a\/b9b60a04a3be8009bda85b055dd7ea55.png\"\/><figcaption>A slightly more detailed scheme of the &#171;vanilla&#187; transformer<\/figcaption><\/figure>\n<p>The self-attention is the most important component of the transformer.\u00a0It helps the model determine which parts of the input are important and how important each part is to other parts of the input.\u00a0Conceptually, the self-attention allows the model to refine the input series via insights generated from global information about all the elements in the series.\u00a0Similar to LSTM models, this makes the transformer naturally good with long series data.\u00a0However, in contrast with LSTM models, transformers are also good at parallelization, and, by turn, its effective inference.<\/p>\n<p>The transformer also creates a dictionary.\u00a0Each element in the dictionary is a <em>token<\/em>.\u00a0The size of the dictionary depends on the model.\u00a0Thus, input first is converted into a series of tokens that then is converted into an embedding with the help of the encoder.\u00a0For the text, its own tokenizer is used (in our case YTTM tokenizer); for the image, low-level tokens are first calculated, and then visual tokens are calculated in a sliding window.\u00a0Using a self-attention allows the model to extract context from input token series during learning.\u00a0It\u2019s worth noting that training the model requires a large volume of (ideally \u201cclean\u201d) data, which we\u2019ll discuss presently.<\/p>\n<h3>How ruDALL-E is constructed<\/h3>\n<p>ruDALL-E architecture is designed so that the transformer will autoregressively learn to model textual and visual tokens as a unified flow.\u00a0However, directly using pixels as a representation of the image requires extraordinary amounts of memory.\u00a0To avoid training only short term dependencies between pixels and text (meaning to attain a higher level of execution overall) the model should undergo training in two steps:\u00a0\u00a0<\/p>\n<ol>\n<li>\n<p>Images having undergone compression to a resolution of 256&#215;256 enter the autoencoder (we used our own SBER VQ-GAN, having earlier improved the metrics for generation on some domains, which we described here, and even shared our code), which learns to compress them into a 32&#215;32 matrix of tokens, totalling 1024.\u00a0The 8 times compression factor ensures the image can be reconstructed without much loss of quality, as can be seen in the cat picture below.<\/p>\n<\/li>\n<li>\n<p>The transformer learns to combine the 1024 image tokens with the textual tokens.\u00a0 We had meanwhile used the YTTM tokenizer to prepare 128 tokens from the text input.\u00a0 The image and the textual tokens come into contact one after the next in series.\u00a0<\/p>\n<\/li>\n<\/ol>\n<figure class=\"full-width\"><img loading=\"lazy\" decoding=\"async\" src=\"https:\/\/habrastorage.org\/r\/w1560\/getpro\/habr\/upload_files\/209\/093\/c0a\/209093c0ae17569a0f6646a42ba87672.png\" alt=\"The initial cat and the reconstructed cat\" title=\"The initial cat and the reconstructed cat\" width=\"944\" height=\"470\" data-src=\"https:\/\/habrastorage.org\/getpro\/habr\/upload_files\/209\/093\/c0a\/209093c0ae17569a0f6646a42ba87672.png\"\/><figcaption>The initial cat and the reconstructed cat<\/figcaption><\/figure>\n<h3>Important aspects of training<\/h3>\n<ol>\n<li>\n<p>At the current time, OpenAI\u2019s code for DALL-E is not publicly available.\u00a0Their publication describes the model in general terms, but it does attend to a few important subtleties in implementation.\u00a0We started with our own code for training ruGPT models and, after studying the documentation, other attempts from open-sourcers, and Chinese efforts, we coded our own DALL-E model.\u00a0It includes features like positional coding of image blocks, convolutional and coordinated masked attention layers, a general representation of text and image embeddings, weighted losses in the text and image sections, and a dropout-tokenizer.<\/p>\n<\/li>\n<li>\n<p>Because of the enormous calculation requirements to effectively train a model, one must use the fp16 precision mode.\u00a0It\u2019s 5-7 times faster than in the classic fp32, and it takes up less space.\u00a0But barriers to precise numerical representations created a mass of difficulties for such a deep architecture:<\/p>\n<p>a) Sometimes really large values found in the network can cause a Nan loss and halt learning;<\/p>\n<p>b) When the learning rate is set at low values in order to avoid problem a), the network stops improving and falls apart because of a large number of zeros appearing in gradients.\u00a0 <\/p>\n<p>To address this issue, we implemented some ideas from the <a href=\"https:\/\/github.com\/THUDM\/CogView\">CogView<\/a> project at China\u2019s Tsinghua University.\u00a0We also carried out our own research into stability, which gave us a number of architectural insights that then helped us stabilize learning.\u00a0Since we found ourselves training the model and researching it simultaneously, the road to success turned out to be long and bumpy.\u00a0 For distributed learning across a couple DGX, we used <a href=\"https:\/\/github.com\/microsoft\/DeepSpeed\">DeepSpeed<\/a>, just as we did for <a href=\"https:\/\/habr.com\/ru\/company\/sberbank\/blog\/524522\/\">ruGPT-3<\/a>.<\/p>\n<\/li>\n<li>\n<p>Collecting and filtering data. As everybody knows, a lot of data is needed to train a transformer, and it should be \u201cclean.\u201d\u00a0We defined \u201cclean\u201d as having two properties. First and foremost, the data must contain good verbal descriptions (which we then translated into Russian). And secondly, images must have an aspect ratio no worse than 1:2 or 2:1, so that cropping doesn\u2019t exclude the main content of the image.<\/p>\n<p>Our first move was to snare the data that OpenAI proffered in their publication (over 250 million pairs) along with the data used by Cogview (30 million pairs).\u00a0Together this included: Conceptual Captions, YFCC100m, Wikipedia data, and ImageNet. Then we added the OpenImages, LAION-400m, WIT, Web2M and HowTo datasets as sources of data about human activities.\u00a0 We also included other datasets that covered domains of interest to us.\u00a0Key domains were people, animals, famous figures, interiors, landmarks and landscapes, various types of technology, human activities, and emotions.<\/p>\n<p>After collecting the data, filtering out overly short descriptions, overly small pictures, pictures with unexaptable aspect ratios, and pictures poorly matching their description (we used the English CLIP model for this), after translation of all English captions into Russian, we were able to create a broad learning dataset of over 120 million pairs image\u2014text caption.<\/p>\n<\/li>\n<li>\n<p>Below is the sloping learning trajectory of ruDALL-E XXL. As you can see, learning needed to be restarted a few times after mistakes and getting off track as a result of Nan.<\/p>\n<figure class=\"full-width\"><img loading=\"lazy\" decoding=\"async\" src=\"https:\/\/habrastorage.org\/r\/w1560\/getpro\/habr\/upload_files\/b28\/ccc\/68e\/b28ccc68e42e416e8eca00cec0809bd3.png\" width=\"1280\" height=\"617\" data-src=\"https:\/\/habrastorage.org\/getpro\/habr\/upload_files\/b28\/ccc\/68e\/b28ccc68e42e416e8eca00cec0809bd3.png\"\/><figcaption><\/figcaption><\/figure>\n<p>The ruDALL-E XXL model was trained in 2 phases: 37 days on 512 TESLA V100 GPUs, and then another 11 days on 128 TESLA V100 GPUs.<\/p>\n<\/li>\n<li>\n<p>Details of training ruDALL-E XL:<\/p>\n<figure class=\"full-width\"><img loading=\"lazy\" decoding=\"async\" src=\"https:\/\/habrastorage.org\/r\/w780q1\/getpro\/habr\/upload_files\/fbe\/75a\/2c6\/fbe75a2c662d0b456e09e12cfbe22a62.jpeg\" alt=\"Loss dynamics on a train sample\" title=\"Loss dynamics on a train sample\" width=\"1280\" height=\"640\" data-src=\"https:\/\/habrastorage.org\/getpro\/habr\/upload_files\/fbe\/75a\/2c6\/fbe75a2c662d0b456e09e12cfbe22a62.jpeg\" data-blurred=\"true\"\/><figcaption>Loss dynamics on a train sample<\/figcaption><\/figure>\n<figure class=\"full-width\"><img loading=\"lazy\" decoding=\"async\" src=\"https:\/\/habrastorage.org\/r\/w780q1\/getpro\/habr\/upload_files\/233\/b8f\/4a6\/233b8f4a6fd92222d963fa1592412da2.jpeg\" alt=\"Loss dynamics on a valid sample\" title=\"Loss dynamics on a valid sample\" width=\"1280\" height=\"640\" data-src=\"https:\/\/habrastorage.org\/getpro\/habr\/upload_files\/233\/b8f\/4a6\/233b8f4a6fd92222d963fa1592412da2.jpeg\" data-blurred=\"true\"\/><figcaption>Loss dynamics on a valid sample<\/figcaption><\/figure>\n<figure class=\"full-width\"><img loading=\"lazy\" decoding=\"async\" src=\"https:\/\/habrastorage.org\/r\/w780q1\/getpro\/habr\/upload_files\/fee\/b31\/3c5\/feeb313c50c4b705a63c66db8868c453.jpeg\" alt=\"Learning rate dynamics\" title=\"Learning rate dynamics\" width=\"1280\" height=\"640\" data-src=\"https:\/\/habrastorage.org\/getpro\/habr\/upload_files\/fee\/b31\/3c5\/feeb313c50c4b705a63c66db8868c453.jpeg\" data-blurred=\"true\"\/><figcaption>Learning rate dynamics<\/figcaption><\/figure>\n<p>The ruDALL-E XL model was trained in 3 phases: 8 days on 128 TESLA V100 GPUs, and then another 6.5 and 8.5 days on 192 TESLA V100 GPUs, but with slightly different training samples.<\/p>\n<\/li>\n<li>\n<p>Choosing optimal generation parameters for different objects and domains is not easy.\u00a0 When it came time to use the model generatively, we began with Nucleus Sample and Top-K sampling. These have already proven their usefulness on NLP tasks and limit the token space accessible for generation.\u00a0Although this topic is well researched for textual creation, the standard setup turned out not to work well for images.\u00a0A series of experiments helped us determine appropriate parameter ranges, but also indicated to us that for different types of outputs, the ranges might be drastically different.\u00a0A mistake might cause significant degradation of quality in the resulting images. Whether parameter ranges can be determined automatically for generation is a topic for future investigations.<\/p>\n<\/li>\n<\/ol>\n<p>Here are some not-so-great examples of output generated in response to the request, <em>\u00ab\u043a\u043e\u0442\u0438\u043a \u0441 \u043a\u0440\u0430\u0441\u043d\u043e\u0439 \u043b\u0435\u043d\u0442\u043e\u0439\u00bb <\/em>(\u201c<em>cat with a red ribbon<\/em>\u201d):<\/p>\n<figure class=\"full-width\"><img loading=\"lazy\" decoding=\"async\" src=\"https:\/\/habrastorage.org\/r\/w1560\/getpro\/habr\/upload_files\/fc7\/590\/1d2\/fc75901d282af2d76d08676f574893cb.png\" alt=\"In the first picture, the cat has three ears, in the second it\u2019s an odd shape, and in the third it\u2019s a little out of focus\" title=\"In the first picture, the cat has three ears, in the second it\u2019s an odd shape, and in the third it\u2019s a little out of focus\" width=\"1000\" height=\"367\" data-src=\"https:\/\/habrastorage.org\/getpro\/habr\/upload_files\/fc7\/590\/1d2\/fc75901d282af2d76d08676f574893cb.png\"\/><figcaption>In the first picture, the cat has three ears, in the second it\u2019s an odd shape, and in the third it\u2019s a little out of focus<\/figcaption><\/figure>\n<p>And here are some results for the request, &#171;<em>\u0410\u0432\u0442\u043e\u043c\u043e\u0431\u0438\u043b\u044c \u043d\u0430 \u0434\u043e\u0440\u043e\u0433\u0435 \u0441\u0440\u0435\u0434\u0438 \u043a\u0440\u0430\u0441\u0438\u0432\u044b\u0445 \u0433\u043e\u0440<\/em>&#187; (\u201c<em>A car on a road surrounded by beautiful mountains<\/em>\u201d).\u00a0In the one on the left, the car encountered some sort of pole, and on the right, it is misshapen.\u00a0<\/p>\n<figure class=\"full-width\"><img loading=\"lazy\" decoding=\"async\" src=\"https:\/\/habrastorage.org\/r\/w1560\/getpro\/habr\/upload_files\/c5d\/f78\/f91\/c5df78f91c06922da48098256d0249aa.png\" alt=\"&quot;\u0410\u0432\u0442\u043e\u043c\u043e\u0431\u0438\u043b\u044c \u043d\u0430 \u0434\u043e\u0440\u043e\u0433\u0435 \u0441\u0440\u0435\u0434\u0438 \u043a\u0440\u0430\u0441\u0438\u0432\u044b\u0445 \u0433\u043e\u0440&quot; (\u201cA car on a road surrounded by beautiful mountains\u201d)\" title=\"&quot;\u0410\u0432\u0442\u043e\u043c\u043e\u0431\u0438\u043b\u044c \u043d\u0430 \u0434\u043e\u0440\u043e\u0433\u0435 \u0441\u0440\u0435\u0434\u0438 \u043a\u0440\u0430\u0441\u0438\u0432\u044b\u0445 \u0433\u043e\u0440&quot; (\u201cA car on a road surrounded by beautiful mountains\u201d)\" width=\"1000\" height=\"497\" data-src=\"https:\/\/habrastorage.org\/getpro\/habr\/upload_files\/c5d\/f78\/f91\/c5df78f91c06922da48098256d0249aa.png\"\/><figcaption>&#171;\u0410\u0432\u0442\u043e\u043c\u043e\u0431\u0438\u043b\u044c \u043d\u0430 \u0434\u043e\u0440\u043e\u0433\u0435 \u0441\u0440\u0435\u0434\u0438 \u043a\u0440\u0430\u0441\u0438\u0432\u044b\u0445 \u0433\u043e\u0440&#187; (\u201cA car on a road surrounded by beautiful mountains\u201d)<\/figcaption><\/figure>\n<h2>Pipeline for image generation<\/h2>\n<p>Image generation today involves using a pipeline consisting of three parts: 1) generation using ruDALL-E, 2) ranking the results with ruCLIP, and 3) improving the quality and resolution of the images using SuperResolution.<\/p>\n<p>During generation and ranking, it\u2019s possible to play with the parameters that affect the quantity of examples generated, their selection, and their level of abstraction.<\/p>\n<p>Pipeline for generating images from text:<\/p>\n<figure class=\"full-width\"><img loading=\"lazy\" decoding=\"async\" src=\"https:\/\/habrastorage.org\/r\/w1560\/getpro\/habr\/upload_files\/d07\/8a9\/798\/d078a97981a47becc053e529cdc00832.png\" alt=\"Pipeline\" title=\"Pipeline\" width=\"1174\" height=\"590\" data-src=\"https:\/\/habrastorage.org\/getpro\/habr\/upload_files\/d07\/8a9\/798\/d078a97981a47becc053e529cdc00832.png\"\/><figcaption>Pipeline<\/figcaption><\/figure>\n<p>In <a href=\"https:\/\/colab.research.google.com\/drive\/1wGE-046et27oHvNlBNPH07qrEQNE04PQ?usp=sharing\">Colab<\/a> one can run the inference model <a href=\"https:\/\/github.com\/sberbank-ai\/ru-dalle\">ruDALL-E XL<\/a> using the whole pipeline (image generation, automatic ranking, and enlargement). Let\u2019s review it via the example with the deer from earlier.<\/p>\n<p><strong>Step 1.<\/strong>\u00a0 Set up and import libraries<\/p>\n<pre><code class=\"python\">git clone https:\/\/github.com\/sberbank-ai\/ru-dalle pip install -r ru-dalle\/requirements.txt > \/dev\/null  from rudalle import get_rudalle_model, get_tokenizer, get_vae,  get_realesrgan, get_ruclip  from rudalle.pipelines import generate_images, show, super_resolution,  cherry_pick_by_clip  device = 'cuda' <\/code><\/pre>\n<p><strong>Step 2.<\/strong> \u00a0 Generate images based on the textual input.\u00a0<\/p>\n<pre><code>text = '\u043e\u0437\u0435\u0440\u043e \u0432 \u0433\u043e\u0440\u0430\u0445, \u0430 \u0440\u044f\u0434\u043e\u043c \u043a\u0440\u0430\u0441\u0438\u0432\u044b\u0439 \u043e\u043b\u0435\u043d\u044c \u043f\u044c\u0435\u0442 \u0432\u043e\u0434\u0443' tokenizer = get_tokenizer() dalle = get_rudalle_model('Malevich', pretrained=True, fp16=True, device=device)  vae = get_vae().to(device)  pil_images, _ = generate_images(text, tokenizer, dalle, vae, top_k=1024, top_p=0.99, images_num=24) show(pil_images, 24)<\/code><\/pre>\n<p>The result looks like this:<\/p>\n<figure class=\"full-width\"><img loading=\"lazy\" decoding=\"async\" src=\"https:\/\/habrastorage.org\/r\/w1560\/getpro\/habr\/upload_files\/e0b\/00c\/bfe\/e0b00cbfebd87a947a85929caa88ee2e.png\" alt=\"Generation of images from text\" title=\"Generation of images from text\" width=\"799\" height=\"539\" data-src=\"https:\/\/habrastorage.org\/getpro\/habr\/upload_files\/e0b\/00c\/bfe\/e0b00cbfebd87a947a85929caa88ee2e.png\"\/><figcaption>Generation of images from text<\/figcaption><\/figure>\n<p><strong>Step 3.<\/strong>\u00a0 Automatic ranking of images and selection of the best<\/p>\n<pre><code class=\"python\">ruclip, ruclip_processor = get_ruclip('ruclip-vit-base-patch32-v5') ruclip = ruclip.to(device)  top_images, _ = cherry_pick_by_clip(pil_images, text, ruclip, ruclip_processor, device=device, count=24) show(top_images, 6)<\/code><\/pre>\n<figure class=\"full-width\"><img loading=\"lazy\" decoding=\"async\" src=\"https:\/\/habrastorage.org\/r\/w1560\/getpro\/habr\/upload_files\/a15\/634\/753\/a15634753fbf1a3ae8f600f61da80a18.png\" alt=\"The outcome after using ruCLIP (top6)\" title=\"The outcome after using ruCLIP (top6)\" width=\"799\" height=\"539\" data-src=\"https:\/\/habrastorage.org\/getpro\/habr\/upload_files\/a15\/634\/753\/a15634753fbf1a3ae8f600f61da80a18.png\"\/><figcaption>The outcome after using ruCLIP (top6)<\/figcaption><\/figure>\n<p>It\u2019s worth noticing that one deer looks fairly slug-like.\u00a0During generation, one can tweak hyperparameters that control how well the output matches your domain.\u00a0After experimenting a bit, we learned that the parameters top_p and top_k control the level of abstraction in the image.\u00a0We generally recommend assigning their values this way:<\/p>\n<ul>\n<li>\n<p>top_k=2048, top_p=0.995<\/p>\n<\/li>\n<li>\n<p> top_k=1536, top_p=0.99<\/p>\n<\/li>\n<li>\n<p>top_k=1024, top_p=0.99  <\/p>\n<\/li>\n<\/ul>\n<p><strong>Step 4.<\/strong>\u00a0 Apply Super Resolution<\/p>\n<pre><code class=\"python\">realesrgan = get_realesrgan('x4', device=device)  sr_images = super_resolution(top_images, realesrgan)  show(sr_images, 6)<\/code><\/pre>\n<figure class=\"full-width\"><img loading=\"lazy\" decoding=\"async\" src=\"https:\/\/habrastorage.org\/r\/w1560\/getpro\/habr\/upload_files\/803\/72f\/d62\/80372fd62e84cdd3447c3d30cb32c2be.png\" alt=\"Output after applying Super Resolution\" title=\"Output after applying Super Resolution\" width=\"799\" height=\"538\" data-src=\"https:\/\/habrastorage.org\/getpro\/habr\/upload_files\/803\/72f\/d62\/80372fd62e84cdd3447c3d30cb32c2be.png\"\/><figcaption>Output after applying Super Resolution<\/figcaption><\/figure>\n<p>The model catalog <a href=\"https:\/\/mlspace.aicloud.sbercloud.ru\/mlspace\/datahub\"><u>DataHub<\/u><\/a> (<a href=\"https:\/\/mlspace.aicloud.sbercloud.ru\/\"><u>ML Space Christofari<\/u><\/a>) can also be used to initialize the pipeline for ruDALL-E XXL or XL.<\/p>\n<h2>The future for multimodal models<\/h2>\n<p>Multimodal research is becoming more and more popular for the wide variety of tasks. Taking center stage are CV + NLP (the first such model for Russian, called ruCLIP, we presented <a href=\"https:\/\/habr.com\/ru\/company\/sberdevices\/blog\/564440\/\">here<\/a>).\u00a0Just behind them are NLP + programming code.\u00a0And then there are models being trained on many source media simultaneously, for example, <a href=\"https:\/\/github.com\/AndreyGuzhov\/AudioCLIP\">AudioCLIP<\/a>. Of particular interest is the <a href=\"https:\/\/arxiv.org\/pdf\/2108.07258.pdf\">Foundation Model<\/a>, which was recently announced by researchers at Stanford University.<\/p>\n<p>Along this line, the <a href=\"https:\/\/ai-journey.ru\">AI Journey<\/a> conference was holding the <a href=\"https:\/\/dsworks.ru\/champ\/fb5778a8-94e9-46de-8bad-aa2c83a755fb\">Fusion Brain Challenge<\/a> competition as we speak.\u00a0In it a unified model has to solve the following four tasks:<\/p>\n<ul>\n<li>\n<p>\u04212\u0421 \u2014 translation from Java to Python;\u00a0<\/p>\n<\/li>\n<li>\n<p>HTR \u2014 recognition of handwriting found in photographs;<\/p>\n<\/li>\n<li>\n<p>Zero-shot Object Detection \u2014 detection of objects described in natural language in images;\u00a0<\/p>\n<\/li>\n<li>\n<p>VQA \u2014 answers to questions about images.\u00a0<\/p>\n<\/li>\n<\/ul>\n<p>According to the competition\u2019s rules, the neural network should dedicate at least 25% of its parameters to general weights!\u00a0 Weights shared among tasks makes models more economical compared to their monomodal analogs.\u00a0The organizers also provided a baseline solution, which can be found officially on <a href=\"https:\/\/github.com\/sberbank-ai\/fusion_brain_aij2021\">GitHub<\/a>, and the teams provided very good concepts of the Fusion Brain.\u00a0Despite the fact that the competition has been completed, the organizers promise to develop this idea and launch a new competition for a longer period.<\/p>\n<p>While specialists are pumping their skills, and while companies will cultivate their calculating prowess for training private models, we remain dedicated to open source and the community at large.\u00a0We\u2019d be happy to see your prototypes, unexpected finds, tests, and suggestions for model improvement!<\/p>\n<p>The most important links:<\/p>\n<ul>\n<li>\n<p>Free demo available: <a href=\"https:\/\/rudalle.ru\/en\/\"><u>https:\/\/rudalle.ru\/en\/<\/u><\/a><\/p>\n<\/li>\n<li>\n<p>Free <a href=\"https:\/\/t.me\/sber_rudalle_xl_bot\"><u>Telegram bot<\/u><\/a><\/p>\n<\/li>\n<li>\n<p>Better and faster generation is available with the Salute App: <a href=\"https:\/\/sberdevices.ru\/app\/\"><u>https:\/\/sberdevices.ru\/app\/<\/u><\/a>\u00a0<\/p>\n<\/li>\n<li>\n<p><a href=\"https:\/\/www.instagram.com\/rudalle.official\/\">Instagram<\/a> with examples of the best generations<\/p>\n<\/li>\n<li>\n<p><a href=\"https:\/\/github.com\/sberbank-ai\/ru-dalle\">GitHub<\/a> with ruDALL-E XL<\/p>\n<\/li>\n<\/ul>\n<p>Team of authors: <a href=\"https:\/\/habr.com\/ru\/users\/rybolos\/\"><u>@rybolos<\/u><\/a><u>,<\/u>\u00a0<a href=\"https:\/\/habr.com\/ru\/users\/shonenkov\/\"><u>@shonenkov<\/u><\/a><u>,\u00a0<\/u><a href=\"https:\/\/habr.com\/ru\/users\/Alter_Ego\/\"><u>@ollmer<\/u><\/a>,\u00a0<a href=\"https:\/\/habr.com\/ru\/users\/kuznetsoff87\/\"><u>@kuznetsoff87<\/u><\/a>,\u00a0<a href=\"\/ru\/users\/alexander-shustanov\/\"><u>@alexander-shustanov<\/u><\/a>,\u00a0<a href=\"https:\/\/habr.com\/ru\/users\/oulenspiegel\/\"><u>@oulenspeigel<\/u><\/a><u>,\u00a0<\/u><a href=\"https:\/\/habr.com\/ru\/users\/mboyarkin\/\"><u>@mboyarkin<\/u><\/a><u>,\u00a0<\/u><a href=\"https:\/\/habr.com\/ru\/users\/achertok\/\"><u>@achertok<\/u><\/a><u>,\u00a0<\/u><a href=\"https:\/\/habr.com\/ru\/users\/da0c\/\"><u>@da0c<\/u><\/a><u>,\u00a0<\/u><a href=\"https:\/\/habr.com\/ru\/users\/boomb0om\/\"><u>@boomb0om<\/u><\/a><\/p>\n<\/div>\n<\/div>\n<\/div>\n<p><!----><!----><\/div>\n<p><!----><!----><br \/> \u0441\u0441\u044b\u043b\u043a\u0430 \u043d\u0430 \u043e\u0440\u0438\u0433\u0438\u043d\u0430\u043b \u0441\u0442\u0430\u0442\u044c\u0438 <a href=\"https:\/\/habr.com\/ru\/articles\/589673\/\"> https:\/\/habr.com\/ru\/articles\/589673\/<\/a><\/p>\n","protected":false},"excerpt":{"rendered":"<div><!--[--><!--]--><\/div>\n<div id=\"post-content-body\">\n<div>\n<div class=\"article-formatted-body article-formatted-body article-formatted-body_version-2\">\n<div xmlns=\"http:\/\/www.w3.org\/1999\/xhtml\">\n<p>In this article we present the ruDALL-E XL model, an <a href=\"https:\/\/github.com\/sberbank-ai\/ru-dalle\">open-source<\/a> text-to-image transformer with 1.3 billion parameters as well as ruDALL-E XXL model, an text-to-image transformer with 12.0 billion parameters which is available in <a href=\"https:\/\/mlspace.aicloud.sbercloud.ru\/mlspace\/datahub\">DataHub<\/a> SberCloud, and several other satellite models.<\/p>\n<ul>\n<li>\n<p>Free demo available: <a href=\"https:\/\/rudalle.ru\/en\/\"><u>https:\/\/rudalle.ru\/en\/<\/u><\/a><\/p>\n<\/li>\n<li>\n<p>Free <a href=\"https:\/\/t.me\/sber_rudalle_xl_bot\"><u>Telegram bot<\/u><\/a><\/p>\n<\/li>\n<li>\n<p>Better and faster generation is available with the Salute App: <a href=\"https:\/\/sberdevices.ru\/app\/\"><u>https:\/\/sberdevices.ru\/app\/<\/u><\/a>\u00a0<\/p>\n<\/li>\n<li>\n<p><a href=\"https:\/\/www.instagram.com\/rudalle.official\/\"><u>Instagram<\/u><\/a> with examples of the best generations<\/p>\n<\/li>\n<li>\n<p><a href=\"https:\/\/github.com\/sberbank-ai\/ru-dalle\"><u>GitHub<\/u><\/a> with ruDALL-E XL<\/p>\n<\/li>\n<\/ul>\n<p>Our demo now works with automatic translation from 107 languages:<\/p>\n<figure class=\"full-width\"><figcaption><\/figcaption><\/figure>\n<h2>Motivation<\/h2>\n<p>Multimodality has led the pack in machine learning in 2021. Neural networks are wolfing down images, text, speech and music all at the same time.\u00a0 OpenAI is, as usual, top dog, but as if in defiance of their name, they are in no hurry to share their models openly.\u00a0 At the beginning of the year, the company presented the DALL-E neural network, which generates 256&#215;256 pixel images in answer to a written request.\u00a0 Descriptions of it can be found as <a href=\"https:\/\/arxiv.org\/pdf\/2102.12092.pdf\"><u>articles<\/u><\/a> on arXiv and examples on their <a href=\"https:\/\/openai.com\/blog\/dall-e\/\"><u>blog<\/u><\/a>.\u00a0\u00a0<\/p>\n<p>As soon as DALL-E flushed out of the bushes, Chinese researchers got on its tail.\u00a0 Their open-source <a href=\"https:\/\/github.com\/THUDM\/CogView\"><u>CogView<\/u><\/a> neural network does the same trick of generating images from text.\u00a0 But what about here in Russia? One might say that \u201cinvestigate, master, and train\u201d is our engineering motto.\u00a0 Well, we caught the scent, and today we can say that we created from scratch a complete pipeline for generating images from descriptive textual input written in Russian.<\/p>\n<p>Teams at Sber AI, SberDevices, Samara University, AIRI and SberCloud all actively contributed. We trained two versions of the model, each a different size:<\/p>\n<ol>\n<li>\n<p>ruDALL-E XXL, with 12.0 billion parameters<\/p>\n<\/li>\n<li>\n<p>ruDALL-E XL, having 1.3 billion parameters<\/p>\n<\/li>\n<\/ol>\n<p>Some of our models are already freely available:<\/p>\n<ol>\n<li>\n<p>ruDALL-E XL [<a href=\"https:\/\/github.com\/sberbank-ai\/ru-dalle\"><u>GitHub<\/u><\/a>, <a href=\"https:\/\/huggingface.co\/sberbank-ai\/rudalle-Malevich\"><u>HuggingFace<\/u><\/a><u>,\u00a0<\/u><a href=\"https:\/\/www.kaggle.com\/shonenkov\/rudalle-example-generation\"><u>Kaggle<\/u><\/a>]<\/p>\n<\/li>\n<li>\n<p>Sber VQ-GAN [<a href=\"https:\/\/github.com\/sberbank-ai\/sber-vq-gan\"><u>GitHub<\/u><\/a>, <a href=\"https:\/\/huggingface.co\/sberbank-ai\/Sber-VQGAN\"><u>HuggingFace<\/u><\/a>]<\/p>\n<\/li>\n<li>\n<p>ruCLIP Small [<a href=\"https:\/\/github.com\/sberbank-ai\/ru-clip\"><u>GitHub<\/u><\/a>, <a href=\"https:\/\/huggingface.co\/sberbank-ai\/ru-clip\"><u>HuggingFace<\/u><\/a>]<\/p>\n<\/li>\n<li>\n<p>Super Resolution (Real ESRGAN) [<a href=\"https:\/\/github.com\/sberbank-ai\/Real-ESRGAN\"><u>GitHub<\/u><\/a>, <a href=\"https:\/\/huggingface.co\/sberbank-ai\/Real-ESRGAN\"><u>HuggingFace<\/u><\/a>]<\/p>\n<\/li>\n<\/ol>\n<p>The latter two models are included in the pipeline for generating images from text (as you\u2019ll see later on). <\/p>\n<p>The models ruDALL-E XL, ruDALL-E XXL, and ruCLIP Small are now available on <a href=\"https:\/\/mlspace.aicloud.sbercloud.ru\/mlspace\/datahub\"><u>DataHub<\/u><\/a>. The models ruCLIP Large and Super Resolution (Real ESRGAN) will also be available soon there.<\/p>\n<p><strong>Training the ruDALL-E neural networks on the <\/strong><a href=\"https:\/\/mlspace.aicloud.sbercloud.ru\/\"><strong><u>Christofari<\/u><\/strong><\/a><strong> cluster has become the largest calculation task in Russia:<\/strong><\/p>\n<ol>\n<li>\n<p>ruDALL-E XXL was trained for 37 days on the 512 GPU TESLA V100, and then also for 11 more days on the 128 GPU TESLA V100, for a total of 20,352 GPU-days;<\/p>\n<\/li>\n<li>\n<p>ruDALL-E XL was trained for 8 days on the 128 GPU TESLA V100, and then also for 15 more days on the 192 GPU TESLA V100, for a total of 3,904 GPU-days.<\/p>\n<\/li>\n<\/ol>\n<p>Accordingly, training for both models totaled <strong>24,256 GPU-days<\/strong>.<\/p>\n<p>Now let\u2019s explore what our generative models are capable of.<\/p>\n<figure class=\"full-width\"><figcaption>Images generated by ruDALL-E XL from the input &#171;\u041e\u0437\u0435\u0440\u043e \u0432 \u0433\u043e\u0440\u0430\u0445, \u0430 \u0440\u044f\u0434\u043e\u043c \u043a\u0440\u0430\u0441\u0438\u0432\u044b\u0439 \u043e\u043b\u0435\u043d\u044c \u043f\u044c\u0451\u0442 \u0432\u043e\u0434\u0443&#187; (\u201cA lake in the mountains, with a beautiful deer drinking water nearby\u201d)<\/figcaption><\/figure>\n<h2>Why is Big Tech doing image generation?<\/h2>\n<p>The long term goal of this research is the creation of multimodal neural networks. They will be able to pull on concepts from a variety of mediums (from text and visuals at first) in order to better understand the world as a whole.<\/p>\n<p>Image generation might seem like the wrong rabbit hole in our century of big data and search engines.\u00a0 But it actually addresses two important requirements that search is currently unable to cope with:<\/p>\n<ol>\n<li>\n<p>Being able to describe in writing exactly what you\u2019re looking for and getting a completely new image created personally for you.<\/p>\n<\/li>\n<li>\n<p>Being able to create at any time as many licence-free illustrations as you could possibly want<\/p>\n<\/li>\n<\/ol>\n<p>The most obvious uses for image generation are:<\/p>\n<ul>\n<li>\n<p>Illustrative photos for journalists, copywriters and advertisers. You can create imagery for articles automatically (meaning also more cheaply and quickly), and generate concept-boards for ads simply from a written description.\u00a0Some examples:<\/p>\n<\/li>\n<\/ul>\n<figure class=\"full-width\"><figcaption>&#171;\u041b\u0438\u0441\u0430 \u0432 \u043b\u0435\u0441\u0443&#187; (\u201cFox in the forest\u201d)<\/figcaption><\/figure>\n<figure class=\"full-width\"><figcaption>&#171;\u041e\u0440\u0435\u043b \u0441\u0438\u0434\u0438\u0442 \u043d\u0430 \u0434\u0435\u0440\u0435\u0432\u0435, \u0432\u0438\u0434 \u0441\u0431\u043e\u043a\u0443&#187; (\u201cAn eagle sits in a tree, looking to the side\u201d)<\/figcaption><\/figure>\n<figure class=\"full-width\"><figcaption>&#171;\u0410\u0432\u0442\u043e\u043c\u043e\u0431\u0438\u043b\u044c \u043d\u0430 \u0434\u043e\u0440\u043e\u0433\u0435 \u0441\u0440\u0435\u0434\u0438 \u043a\u0440\u0430\u0441\u0438\u0432\u044b\u0445 \u0433\u043e\u0440&#187; (\u201cA car on a road surrounded by beautiful mountains\u201d)<\/figcaption><\/figure>\n<ul>\n<li>\n<p>An artwork, completely license-free, can also be created ad infinitum:<\/p>\n<figure class=\"full-width\"><figcaption>&#171;\u0412\u0435\u043a\u0442\u043e\u0440\u043d\u0430\u044f \u0438\u043b\u043b\u044e\u0441\u0442\u0440\u0430\u0446\u0438\u044f \u0441 \u0440\u043e\u0437\u043e\u0432\u044b\u043c\u0438 \u0446\u0432\u0435\u0442\u0430\u043c\u0438&#187; (\u201cVector artwork with pink flowers\u201d)<\/figcaption><\/figure>\n<\/li>\n<li>\n<p>Interior design mock-ups.\u00a0You can do a quick visual check of your ideas for a remodel, and play with colors, form, and light:<\/p>\n<\/li>\n<\/ul>\n<figure class=\"full-width\"><figcaption>&#171;\u0428\u0438\u043a\u0430\u0440\u043d\u0430\u044f \u0433\u043e\u0441\u0442\u0438\u043d\u0430\u044f \u0441 \u0437\u0435\u043b\u0435\u043d\u044b\u043c\u0438 \u043a\u0440\u0435\u0441\u043b\u0430\u043c\u0438&#187; (\u201cElegant living room with green stuffed chairs\u201d)<\/figcaption><\/figure>\n<figure class=\"full-width\"><figcaption>&#171;\u0421\u043e\u0432\u0440\u0435\u043c\u0435\u043d\u043d\u043e\u0435 \u043a\u0440\u0435\u0441\u043b\u043e \u0444\u0438\u043e\u043b\u0435\u0442\u043e\u0432\u043e\u0433\u043e \u0446\u0432\u0435\u0442\u0430&#187; (\u201cContemporary purple armchair\u201d)<\/figcaption><\/figure>\n<ul>\n<li>\n<p>Visual art.\u00a0A source of visual concepts combining various attributes with abstraction:<\/p>\n<figure class=\"full-width\"><figcaption>&#171;\u0422\u0435\u043c\u043d\u0430\u044f \u044d\u043d\u0435\u0440\u0433\u0438\u044f&#187; (\u201cDark energy\u201d)<\/figcaption><\/figure>\n<\/li>\n<\/ul>\n<figure class=\"full-width\"><figcaption>&#171;\u041a\u043e\u0442 \u043d\u0430 \u041b\u0443\u043d\u0435&#187; (\u201cCat on the moon\u201d)<\/figcaption><\/figure>\n<figure class=\"full-width\"><figcaption>&#171;\u041a\u043e\u0448\u043a\u0430 \u043a\u043e\u0442\u043e\u0440\u0430\u044f \u0441\u0434\u0435\u043b\u0430\u043d\u0430 \u0438\u0437 \u0431\u0435\u043b\u043e\u0433\u043e \u043e\u0431\u043b\u0430\u043a\u0430&#187; (\u201cA cat made out of white cloud\u201d)<\/figcaption><\/figure>\n<figure class=\"full-width\"><figcaption>&#171;\u0415\u043d\u043e\u0442 \u0441 \u043f\u0443\u0448\u043a\u043e\u0439&#187; (\u201cRaccoon with gun\u201d)<\/figcaption><\/figure>\n<figure class=\"full-width\"><figcaption>&#171;\u041a\u0440\u0430\u0441\u0438\u0432\u043e\u0435 \u043e\u0437\u0435\u0440\u043e \u043d\u0430 \u0437\u0430\u043a\u0430\u0442\u0435&#187; (\u201cPretty lake at sunset\u201d)<\/figcaption><\/figure>\n<figure class=\"full-width\"><figcaption>&#171;\u0420\u0430\u0434\u0443\u0436\u043d\u0430\u044f \u0441\u043e\u0432\u0430&#187; (&#171;Rainbow owl&#187;)<\/figcaption><\/figure>\n<figure class=\"full-width\"><figcaption>&#171;\u0416\u0434\u0443\u043d \u0441 \u0430\u0432\u043e\u043a\u0430\u0434\u043e&#187; (&#171;Zhdun with avocado&#187;)<\/figcaption><\/figure>\n<h2>Model Architecture<\/h2>\n<p>The architectural foundation of DALL-E, the so-called <em>transformer<\/em>, is an encoder-decoder model.\u00a0It works in two steps.\u00a0First, the encoder needs to calculate an embedding from the input.\u00a0Then the decoder deciphers the resulting vector in view of a known target so that the output matches expectations.\u00a0 <\/p>\n<figure class=\"full-width\"><figcaption>Top-level scheme of the vanilla transformer<\/figcaption><\/figure>\n<figure class=\"full-width\"><figcaption>A slightly more detailed scheme of the &#171;vanilla&#187; transformer<\/figcaption><\/figure>\n<p>The self-attention is the most important component of the transformer.\u00a0It helps the model determine which parts of the input are important and how important each part is to other parts of the input.\u00a0Conceptually, the self-attention allows the model to refine the input series via insights generated from global information about all the elements in the series.\u00a0Similar to LSTM models, this makes the transformer naturally good with long series data.\u00a0However, in contrast with LSTM models, transformers are also good at parallelization, and, by turn, its effective inference.<\/p>\n<p>The transformer also creates a dictionary.\u00a0Each element in the dictionary is a <em>token<\/em>.\u00a0The size of the dictionary depends on the model.\u00a0Thus, input first is converted into a series of tokens that then is converted into an embedding with the help of the encoder.\u00a0For the text, its own tokenizer is used (in our case YTTM tokenizer); for the image, low-level tokens are first calculated, and then visual tokens are calculated in a sliding window.\u00a0Using a self-attention allows the model to extract context from input token series during learning.\u00a0It\u2019s worth noting that training the model requires a large volume of (ideally \u201cclean\u201d) data, which we\u2019ll discuss presently.<\/p>\n<h3>How ruDALL-E is constructed<\/h3>\n<p>ruDALL-E architecture is designed so that the transformer will autoregressively learn to model textual and visual tokens as a unified flow.\u00a0However, directly using pixels as a representation of the image requires extraordinary amounts of memory.\u00a0To avoid training only short term dependencies between pixels and text (meaning to attain a higher level of execution overall) the model should undergo training in two steps:\u00a0\u00a0<\/p>\n<ol>\n<li>\n<p>Images having undergone compression to a resolution of 256&#215;256 enter the autoencoder (we used our own SBER VQ-GAN, having earlier improved the metrics for generation on some domains, which we described here, and even shared our code), which learns to compress them into a 32&#215;32 matrix of tokens, totalling 1024.\u00a0The 8 times compression factor ensures the image can be reconstructed without much loss of quality, as can be seen in the cat picture below.<\/p>\n<\/li>\n<li>\n<p>The transformer learns to combine the 1024 image tokens with the textual tokens.\u00a0 We had meanwhile used the YTTM tokenizer to prepare 128 tokens from the text input.\u00a0 The image and the textual tokens come into contact one after the next in series.\u00a0<\/p>\n<\/li>\n<\/ol>\n<figure class=\"full-width\"><figcaption>The initial cat and the reconstructed cat<\/figcaption><\/figure>\n<h3>Important aspects of training<\/h3>\n<ol>\n<li>\n<p>At the current time, OpenAI\u2019s code for DALL-E is not publicly available.\u00a0Their publication describes the model in general terms, but it does attend to a few important subtleties in implementation.\u00a0We started with our own code for training ruGPT models and, after studying the documentation, other attempts from open-sourcers, and Chinese efforts, we coded our own DALL-E model.\u00a0It includes features like positional coding of image blocks, convolutional and coordinated masked attention layers, a general representation of text and image embeddings, weighted losses in the text and image sections, and a dropout-tokenizer.<\/p>\n<\/li>\n<li>\n<p>Because of the enormous calculation requirements to effectively train a model, one must use the fp16 precision mode.\u00a0It\u2019s 5-7 times faster than in the classic fp32, and it takes up less space.\u00a0But barriers to precise numerical representations created a mass of difficulties for such a deep architecture:<\/p>\n<p>a) Sometimes really large values found in the network can cause a Nan loss and halt learning;<\/p>\n<p>b) When the learning rate is set at low values in order to avoid problem a), the network stops improving and falls apart because of a large number of zeros appearing in gradients.\u00a0 <\/p>\n<p>To address this issue, we implemented some ideas from the <a href=\"https:\/\/github.com\/THUDM\/CogView\">CogView<\/a> project at China\u2019s Tsinghua University.\u00a0We also carried out our own research into stability, which gave us a number of architectural insights that then helped us stabilize learning.\u00a0Since we found ourselves training the model and researching it simultaneously, the road to success turned out to be long and bumpy.\u00a0 For distributed learning across a couple DGX, we used <a href=\"https:\/\/github.com\/microsoft\/DeepSpeed\">DeepSpeed<\/a>, just as we did for <a href=\"https:\/\/habr.com\/ru\/company\/sberbank\/blog\/524522\/\">ruGPT-3<\/a>.<\/p>\n<\/li>\n<li>\n<p>Collecting and filtering data. As everybody knows, a lot of data is needed to train a transformer, and it should be \u201cclean.\u201d\u00a0We defined \u201cclean\u201d as having two properties. First and foremost, the data must contain good verbal descriptions (which we then translated into Russian). And secondly, images must have an aspect ratio no worse than 1:2 or 2:1, so that cropping doesn\u2019t exclude the main content of the image.<\/p>\n<p>Our first move was to snare<\/p>\n<\/li>\n<\/ol>\n<\/div>\n<\/div>\n<\/div>\n<\/div>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[],"tags":[],"class_list":["post-399015","post","type-post","status-publish","format-standard","hentry"],"_links":{"self":[{"href":"https:\/\/savepearlharbor.com\/index.php?rest_route=\/wp\/v2\/posts\/399015","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/savepearlharbor.com\/index.php?rest_route=\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/savepearlharbor.com\/index.php?rest_route=\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/savepearlharbor.com\/index.php?rest_route=\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/savepearlharbor.com\/index.php?rest_route=%2Fwp%2Fv2%2Fcomments&post=399015"}],"version-history":[{"count":0,"href":"https:\/\/savepearlharbor.com\/index.php?rest_route=\/wp\/v2\/posts\/399015\/revisions"}],"wp:attachment":[{"href":"https:\/\/savepearlharbor.com\/index.php?rest_route=%2Fwp%2Fv2%2Fmedia&parent=399015"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/savepearlharbor.com\/index.php?rest_route=%2Fwp%2Fv2%2Fcategories&post=399015"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/savepearlharbor.com\/index.php?rest_route=%2Fwp%2Fv2%2Ftags&post=399015"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}