{"id":373549,"date":"2024-05-21T05:38:48","date_gmt":"2024-05-21T05:38:48","guid":{"rendered":"http:\/\/savepearlharbor.com\/?p=373549"},"modified":"-0001-11-30T00:00:00","modified_gmt":"-0001-11-29T21:00:00","slug":"","status":"publish","type":"post","link":"https:\/\/savepearlharbor.com\/?p=373549","title":{"rendered":"<span>\u0417\u0430\u043f\u0443\u0441\u043a\u0430\u0435\u043c PrivateGPT \u043d\u0430 GPU AMD Radeon \u0432 Docker<\/span>"},"content":{"rendered":"<div><!--[--><!--]--><\/div>\n<div id=\"post-content-body\">\n<div>\n<div class=\"article-formatted-body article-formatted-body article-formatted-body_version-2\">\n<div xmlns=\"http:\/\/www.w3.org\/1999\/xhtml\">\n<p><a href=\"https:\/\/docs.privategpt.dev\/overview\/welcome\/introduction\" rel=\"noopener noreferrer nofollow\">PrivateGPT<\/a> \u2014 \u044d\u0442\u043e \u043f\u0440\u043e\u0435\u043a\u0442, \u043a\u043e\u0442\u043e\u0440\u044b\u0439 \u0440\u0430\u0441\u0448\u0438\u0440\u044f\u0435\u0442 \u0432\u043e\u0437\u043c\u043e\u0436\u043d\u043e\u0441\u0442\u0438 \u0440\u0430\u0431\u043e\u0442\u044b LLM-\u043c\u043e\u0434\u0435\u043b\u0435\u0439, \u043f\u043e\u0437\u0432\u043e\u043b\u044f\u044f \u0434\u043e\u0431\u0430\u0432\u043b\u044f\u0442\u044c \u043d\u0435\u043e\u0433\u0440\u0430\u043d\u0438\u0447\u0435\u043d\u043d\u043e\u0435 \u043a\u043e\u043b\u0438\u0447\u0435\u0441\u0442\u0432\u043e \u043b\u0438\u0447\u043d\u044b\u0445 \u0434\u0430\u043d\u043d\u044b\u0445.<\/p>\n<p>31 \u043e\u043a\u0442\u044f\u0431\u0440\u044f 2023 <a href=\"https:\/\/pytorch.org\/blog\/amd-extends-support-for-pt-ml\/\" rel=\"noopener noreferrer nofollow\">AMD Radeon \u043f\u0440\u0435\u0434\u043e\u0441\u0442\u0430\u0432\u0438\u043b\u0430 \u043f\u043e\u0434\u0434\u0435\u0440\u0436\u043a\u0443 PyTorch<\/a> \u0434\u043b\u044f \u043b\u044e\u0431\u0438\u0442\u0435\u043b\u044c\u0441\u043a\u0438\u0445 \u0432\u0438\u0434\u0435\u043e\u043a\u0430\u0440\u0442. \u041f\u043e\u043b\u043d\u044b\u0439 \u0441\u043f\u0438\u0441\u043e\u043a \u0432\u0438\u0434\u0435\u043e\u043a\u0430\u0440\u0442 \u0438 \u041e\u0421 \u043c\u043e\u0436\u043d\u043e \u043f\u043e\u0441\u043c\u043e\u0442\u0440\u0435\u0442\u044c <a href=\"https:\/\/rocm.docs.amd.com\/projects\/install-on-linux\/en\/latest\/reference\/system-requirements.html\" rel=\"noopener noreferrer nofollow\">\u0437\u0434\u0435\u0441\u044c<\/a>. \u041e\u043f\u0438\u0441\u0430\u043d\u043d\u0430\u044f \u0438\u043d\u0441\u0442\u0440\u0443\u043a\u0446\u0438\u044f \u043f\u0440\u043e\u0442\u0435\u0441\u0442\u0438\u0440\u043e\u0432\u0430\u043d\u0430 \u043d\u0430 AMD Radeon RX 7900 XTX.<\/p>\n<p>\u0414\u043b\u044f \u0437\u0430\u043f\u0443\u0441\u043a\u0430 \u043d\u0430\u043c \u043f\u043e\u043d\u0430\u0434\u043e\u0431\u0438\u0442\u0441\u044f Ubuntu \u0441 \u0443\u0441\u0442\u0430\u043d\u043e\u0432\u043b\u0435\u043d\u043d\u044b\u043c\u0438: git, make, docker \u0438 ROCm.<\/p>\n<p>ROCm \u0443\u0441\u0442\u0430\u043d\u043e\u0432\u0438\u043c \u043f\u043e <a href=\"https:\/\/rocm.docs.amd.com\/projects\/install-on-linux\/en\/latest\/tutorial\/quick-start.html\" rel=\"noopener noreferrer nofollow\">\u0438\u043d\u0441\u0442\u0440\u0443\u043a\u0446\u0438\u0438<\/a>.<\/p>\n<details class=\"spoiler\">\n<summary>\u041a\u0440\u0430\u0442\u043a\u0430\u044f \u0438\u043d\u0441\u0442\u0440\u0443\u043a\u0446\u0438\u044f \u0443\u0441\u0442\u0430\u043d\u043e\u0432\u043a\u0438 ROCm<\/summary>\n<div class=\"spoiler__content\">\n<pre><code class=\"bash\">sudo apt install \"linux-headers-$(uname -r)\" \"linux-modules-extra-$(uname -r)\" sudo usermod -a -G render,video $LOGNAME wget https:\/\/repo.radeon.com\/amdgpu-install\/6.0.2\/ubuntu\/jammy\/amdgpu-install_6.0.60002-1_all.deb sudo apt install .\/amdgpu-install_6.0.60002-1_all.deb  # \u041f\u0440\u0438 \u043e\u0448\u0438\u0431\u043a\u0435 \u043d\u0435\u043e\u0431\u0445\u043e\u0434\u0438\u043c\u043e \u0432\u044b\u043f\u043e\u043b\u043d\u0438\u0442\u044c: sudo chown -Rv _apt:root \/var\/cache\/apt\/archives\/partial\/ sudo chmod -Rv 700 \/var\/cache\/apt\/archives\/partial\/  sudo apt update sudo apt install amdgpu-dkms sudo apt install rocm-hip-libraries sudo reboot<\/code><\/pre>\n<\/p>\n<\/div>\n<\/details>\n<p>\u0421\u043a\u043e\u043f\u0438\u0440\u0443\u0435\u043c \u043f\u0440\u043e\u0435\u043a\u0442 \u043f\u043e\u0437\u0432\u043e\u043b\u044f\u044e\u0449\u0438\u0439 \u043d\u0430\u0441\u0442\u0440\u043e\u0438\u0442\u044c \u0438 \u0437\u0430\u043f\u0443\u0441\u0442\u0438\u0442\u044c PrivateGPT \u0432 docker \u043a\u043e\u043d\u0442\u0435\u0439\u043d\u0435\u0440\u0435:<\/p>\n<pre><code class=\"bash\">git clone https:\/\/github.com\/HardAndHeavy\/private-gpt-rocm-docker cd private-gpt-rocm-docker<\/code><\/pre>\n<p>\u0421\u043e\u0437\u0434\u0430\u0434\u0438\u043c \u0444\u0430\u0439\u043b \u043d\u0430\u0441\u0442\u0440\u043e\u0435\u043a PrivateGPT (<a href=\"https:\/\/github.com\/zylon-ai\/private-gpt\/blob\/main\/settings.yaml\" rel=\"noopener noreferrer nofollow\">settings.yaml<\/a>) \u0441 \u043f\u043e\u043c\u043e\u0449\u044c\u044e \u043a\u043e\u043c\u0430\u043d\u0434\u044b <code>make gen<\/code>. \u041d\u0430 \u0432\u0441\u0435 \u0437\u0430\u0434\u0430\u043d\u043d\u044b\u0435 \u0432\u043e\u043f\u0440\u043e\u0441\u044b \u043c\u043e\u0436\u043d\u043e \u043f\u0440\u043e\u0441\u0442\u043e \u043d\u0430\u0436\u0430\u0442\u044c \u043a\u043d\u043e\u043f\u043a\u0443 <code>\u0412\u0432\u043e\u0434<\/code>. \u0422\u043e\u0433\u0434\u0430 \u0444\u0430\u0439\u043b \u043d\u0430\u0441\u0442\u0440\u043e\u0435\u043a \u043f\u0440\u0438\u043c\u0435\u0442 \u0437\u043d\u0430\u0447\u0435\u043d\u0438\u0435 \u043f\u043e \u0443\u043c\u043e\u043b\u0447\u0430\u043d\u0438\u044e.<\/p>\n<details class=\"spoiler\">\n<summary>\u0413\u0435\u043d\u0435\u0440\u0430\u0446\u0438\u044f \u0444\u0430\u0439\u043b\u0430 \u043d\u0430\u0441\u0442\u0440\u043e\u0439\u043a\u0438<\/summary>\n<div class=\"spoiler__content\">\n<pre><code class=\"bash\">\u042f\u0437\u044b\u043a\u043e\u0432\u0430\u044f \u043c\u043e\u0434\u0435\u043b\u044c [IlyaGusev\/saiga_mistral_7b_gguf]:  \u0424\u0430\u0439\u043b \u044f\u0437\u044b\u043a\u043e\u0432\u043e\u0439 \u043c\u043e\u0434\u0435\u043b\u0438 [model-q8_0.gguf]:  \u042f\u0437\u044b\u043a\u043e\u0432\u0430\u044f \u043c\u043e\u0434\u0435\u043b\u044c \u0434\u043b\u044f \u0431\u043b\u043e\u043a\u043e\u0432 [mistralai\/Mistral-7B-Instruct-v0.2]:  \u041c\u043e\u0434\u0435\u043b\u044c \u0432\u0441\u0442\u0440\u0430\u0438\u0432\u0430\u043d\u0438\u044f [intfloat\/multilingual-e5-large]:<\/code><\/pre>\n<\/p>\n<\/div>\n<\/details>\n<p>\u0412 \u0440\u0435\u0437\u0443\u043b\u044c\u0442\u0430\u0442\u0435 \u043f\u043e\u044f\u0432\u0438\u0442\u0441\u044f \u0444\u0430\u0439\u043b <code>settings-gpt.yaml<\/code>, \u043a\u043e\u0442\u043e\u0440\u044b\u0439 \u0432 \u043f\u043e\u0441\u043b\u0435\u0434\u0443\u044e\u0449\u0435\u043c \u043c\u043e\u0436\u043d\u043e \u0434\u043e\u043d\u0430\u0441\u0442\u0440\u043e\u0438\u0442\u044c \u043f\u0440\u0438 \u0433\u043b\u0443\u0431\u043e\u043a\u043e\u043c \u0438\u0437\u0443\u0447\u0435\u043d\u0438\u0438 PrivateGPT.<\/p>\n<p>\u0417\u0430\u043f\u0443\u0441\u0442\u0438\u043c PrivateGPT \u0441 \u043f\u043e\u043c\u043e\u0449\u044c\u044e \u043a\u043e\u043c\u0430\u043d\u0434\u044b <code>make run<\/code>. \u041f\u0440\u0438 \u043f\u0435\u0440\u0432\u043e\u043c \u0437\u0430\u043f\u0443\u0441\u043a\u0435 \u0431\u0443\u0434\u0435\u0442 \u043f\u0440\u043e\u0438\u0441\u0445\u043e\u0434\u0438\u0442\u044c \u0434\u043b\u0438\u0442\u0435\u043b\u044c\u043d\u044b\u0439 \u043f\u0440\u043e\u0446\u0435\u0441\u0441 \u0441\u043a\u0430\u0447\u0438\u0432\u0430\u043d\u0438\u044f \u043c\u043e\u0434\u0435\u043b\u0438. \u041a\u043e\u0433\u0434\u0430 \u044d\u0442\u043e\u0442 \u043f\u0440\u043e\u0446\u0435\u0441\u0441 \u0437\u0430\u0432\u0435\u0440\u0448\u0438\u0442\u0441\u044f, PrivateGPT \u0441\u0442\u0430\u043d\u0435\u0442 \u0434\u043e\u0441\u0442\u0443\u043f\u0435\u043d \u043f\u043e \u0430\u0434\u0440\u0435\u0441\u0443 <a href=\"http:\/\/localhost\" rel=\"noopener noreferrer nofollow\">http:\/\/localhost<\/a>.<\/p>\n<details class=\"spoiler\">\n<summary>\u041b\u043e\u0433 \u043f\u0440\u043e\u0446\u0435\u0441\u0441\u0430 \u0437\u0430\u043f\u0443\u0441\u043a\u0430<\/summary>\n<div class=\"spoiler__content\">\n<pre><code class=\"bash\">docker run -it --rm \\                                                                                                                                                                                                -p 80:8080 \\                                                                                                                                                                                                 -v .\/local_data\/:\/app\/local_data \\                                                                                                                                                                           -v .\/models\/:\/app\/models \\                                                                                                                                                                                   -e PGPT_PROFILES=gpt \\                                                                                                                                                                                       -v .\/settings-gpt.yaml:\/app\/settings-gpt.yaml \\                                                                                                                                                              --device=\/dev\/kfd \\                                                                                                                                                                                          --device=\/dev\/dri \\                                                                                                                                                                                          hardandheavy\/private-gpt-rocm:latest                                                                                                                                                                 10:47:50.938 [INFO    ] private_gpt.settings.settings_loader - Starting application with profiles=['default', 'gpt']                                                                                         Downloading embedding intfloat\/multilingual-e5-large                                                                                                                                                         Fetching 19 files: 100%|\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588| 19\/19 [00:01&lt;00:00, 17.22it\/s] Embedding model downloaded!                                                                                                                                                                                  Downloading LLM model-q8_0.gguf                                                                                                                                                                              LLM model downloaded!                                                                                                                                                                                        Downloading tokenizer mistralai\/Mistral-7B-Instruct-v0.2                                                                                                                                                     Tokenizer downloaded!                                                                                                                                                                                        Setup done                                                                                                                                                                                                   poetry run python -m private_gpt                                                                                                                                                                             10:47:54.978 [INFO    ] private_gpt.settings.settings_loader - Starting application with profiles=['default', 'gpt']                                                                                         10:48:00.396 [INFO    ]   matplotlib.font_manager - generated new fontManager                                                                                                                                10:48:01.914 [INFO    ] private_gpt.components.llm.llm_component - Initializing the LLM in mode=llamacpp                                                                                                     ggml_init_cublas: GGML_CUDA_FORCE_MMQ:   no                                                                                                                                                                  ggml_init_cublas: CUDA_USE_TENSOR_CORES: yes                                                                                                                                                                 ggml_init_cublas: found 1 ROCm devices:                                                                                                                                                                        Device 0: Radeon RX 7900 XTX, compute capability 11.0, VMM: no                                                                                                                                             llama_model_loader: loaded meta data with 21 key-value pairs and 291 tensors from \/app\/models\/model-q8_0.gguf (version GGUF V2)                                                                              llama_model_loader: Dumping metadata keys\/values. Note: KV overrides do not apply in this output.                                                                                                            llama_model_loader: - kv   0:                       general.architecture str              = llama                                                                                                            llama_model_loader: - kv   1:                               general.name str              = models                                                                                                           llama_model_loader: - kv   2:                       llama.context_length u32              = 32768                                                                                                            llama_model_loader: - kv   3:                     llama.embedding_length u32              = 4096                                                                                                             llama_model_loader: - kv   4:                          llama.block_count u32              = 32                                                                                                               llama_model_loader: - kv   5:                  llama.feed_forward_length u32              = 14336                                                                                                            llama_model_loader: - kv   6:                 llama.rope.dimension_count u32              = 128                                                                                                              llama_model_loader: - kv   7:                 llama.attention.head_count u32              = 32                                                                                                               llama_model_loader: - kv   8:              llama.attention.head_count_kv u32              = 8                                                                                                                llama_model_loader: - kv   9:     llama.attention.layer_norm_rms_epsilon f32              = 0.000010                                                                                                         llama_model_loader: - kv  10:                       llama.rope.freq_base f32              = 10000.000000                                                                                                     llama_model_loader: - kv  11:                          general.file_type u32              = 7                                                                                                                llama_model_loader: - kv  12:                       tokenizer.ggml.model str              = llama                                                                                                            llama_model_loader: - kv  13:                      tokenizer.ggml.tokens arr[str,32002]   = [\"&lt;unk>\", \"&lt;s>\", \"&lt;\/s>\", \"&lt;0x00>\", \"&lt;...                                                                         llama_model_loader: - kv  14:                      tokenizer.ggml.scores arr[f32,32002]   = [0.000000, 0.000000, 0.000000, 0.0000... llama_model_loader: - kv  llama_model_loader: - kv  16:                tokenizer.ggml.bos_token_id u32              = 1                                                                                                                llama_model_loader: - kv  17:                tokenizer.ggml.eos_token_id u32              = 2                                                                                                                llama_model_loader: - kv  18:            tokenizer.ggml.unknown_token_id u32              = 0                                                                                                                llama_model_loader: - kv  19:            tokenizer.ggml.padding_token_id u32              = 0                                                                                                                llama_model_loader: - kv  20:               general.quantization_version u32              = 2                                                                                                                llama_model_loader: - type  f32:   65 tensors                                                                                                                                                                llama_model_loader: - type q8_0:  226 tensors                                                                                                                                                                llm_load_vocab: special tokens definition check successful ( 261\/32002 ).                                                                                                                                    llm_load_print_meta: format           = GGUF V2                                                                                                                                                              llm_load_print_meta: arch             = llama                                                                                                                                                                llm_load_print_meta: vocab type       = SPM                                                                                                                                                                  llm_load_print_meta: n_vocab          = 32002                                                                                                                                                                llm_load_print_meta: n_merges         = 0                                                                                                                                                                    llm_load_print_meta: n_ctx_train      = 32768                                                                                                                                                                llm_load_print_meta: n_embd           = 4096                                                                                                                                                                 llm_load_print_meta: n_head           = 32                                                                                                                                                                   llm_load_print_meta: n_head_kv        = 8                                                                                                                                                                    llm_load_print_meta: n_layer          = 32                                                                                                                                                                   llm_load_print_meta: n_rot            = 128                                                                                                                                                                  llm_load_print_meta: n_embd_head_k    = 128                                                                                                                                                                  llm_load_print_meta: n_embd_head_v    = 128                                                                                                                                                                  llm_load_print_meta: n_gqa            = 4                                                                                                                                                                    llm_load_print_meta: n_embd_k_gqa     = 1024                                                                                                                                                                 llm_load_print_meta: n_embd_v_gqa     = 1024                                                                                                                                                                 llm_load_print_meta: f_norm_eps       = 0.0e+00                                                                                                                                                              llm_load_print_meta: f_norm_rms_eps   = 1.0e-05                                                                                                                                                              llm_load_print_meta: f_clamp_kqv      = 0.0e+00                                                                                                                                                              llm_load_print_meta: f_max_alibi_bias = 0.0e+00                                                                                                                                                              llm_load_print_meta: n_ff             = 14336                                                                                                                                                                llm_load_print_meta: n_expert         = 0                                                                                                                                                                    llm_load_print_meta: n_expert_used    = 0 llm_load_print_meta: pooling type     = 0 llm_load_print_meta: rope type        = 0 llm_load_print_meta: rope scaling     = linear llm_load_print_meta: freq_base_train  = 10000.0 llm_load_print_meta: freq_scale_train = 1 llm_load_print_meta: n_yarn_orig_ctx  = 32768 llm_load_print_meta: rope_finetuned   = unknown llm_load_print_meta: ssm_d_conv       = 0 llm_load_print_meta: ssm_d_inner      = 0 llm_load_print_meta: ssm_d_state      = 0 llm_load_print_meta: ssm_dt_rank      = 0 llm_load_print_meta: model type       = 7B llm_load_print_meta: model ftype      = Q8_0 llm_load_print_meta: model params     = 7.24 B llm_load_print_meta: model size       = 7.17 GiB (8.50 BPW)  llm_load_print_meta: general.name     = models llm_load_print_meta: BOS token        = 1 '&lt;s>' llm_load_print_meta: EOS token        = 2 '&lt;\/s>' llm_load_print_meta: UNK token        = 0 '&lt;unk>' llm_load_print_meta: PAD token        = 0 '&lt;unk>' llm_load_print_meta: LF token         = 13 '&lt;0x0A>' llm_load_tensors: ggml ctx size =    0.22 MiB llm_load_tensors: offloading 32 repeating layers to GPU llm_load_tensors: offloading non-repeating layers to GPU llm_load_tensors: offloaded 33\/33 layers to GPU llm_load_tensors:      ROCm0 buffer size =  7205.84 MiB llm_load_tensors:        CPU buffer size =   132.82 MiB ................................................................................................... llama_new_context_with_model: n_ctx      = 3900 llama_new_context_with_model: freq_base  = 10000.0 llama_new_context_with_model: freq_scale = 1 llama_kv_cache_init:      ROCm0 KV buffer size =   487.50 MiB llama_new_context_with_model: KV self size  =  487.50 MiB, K (f16):  243.75 MiB, V (f16):  243.75 MiB llama_new_context_with_model:  ROCm_Host input buffer size   =    16.65 MiB llama_new_context_with_model:      ROCm0 compute buffer size =   283.37 MiB llama_new_context_with_model:  ROCm_Host compute buffer size =     8.00 MiB llama_new_context_with_model: graph splits (measure): 2 AVX = 1 | AVX_VNNI = 0 | AVX2 = 1 | AVX512 = 0 | AVX512_VBMI = 0 | AVX512_VNNI = 0 | FMA = 1 | NEON = 0 | ARM_FMA = 0 | F16C = 1 | FP16_VA = 0 | WASM_SIMD = 0 | BLAS = 1 | SSE3 = 1 | SSSE3 = 1 | VSX = 0 | MATMUL_INT8 = 0 |  Model metadata: {'tokenizer.ggml.padding_token_id': '0', 'tokenizer.ggml.unknown_token_id': '0', 'tokenizer.ggml.eos_token_id': '2', 'general.architecture': 'llama', 'llama.rope.freq_base': '10000.000000', 'llama.context_length': '32768', 'general.name': 'models', 'llama.embedding_length': '4096', 'llama.feed_forward_length': '14336', 'llama.attention.layer_norm_rms_epsilon': '0.000010', 'llama.rope.dimension_count': '128', 'tokenizer.ggml.bos_token_id': '1', 'llama.attention.head_count': '32', 'llama.block_count': '32', 'llama.attention.head_count_kv': '8', 'general.quantization_version': '2', 'tokenizer.ggml.model': 'llama', 'general.file_type': '7'} Using fallback chat format: None 10:50:22.172 [INFO    ] private_gpt.components.embedding.embedding_component - Initializing the embedding model in mode=huggingface 10:51:15.008 [INFO    ] llama_index.core.indices.loading - Loading all indices. 10:51:15.301 [INFO    ]         private_gpt.ui.ui - Mounting the gradio UI, at path=\/ 10:51:15.437 [INFO    ]             uvicorn.error - Started server process [50] 10:51:15.437 [INFO    ]             uvicorn.error - Waiting for application startup. 10:51:15.438 [INFO    ]             uvicorn.error - Application startup complete. 10:51:15.441 [INFO    ]             uvicorn.error - Uvicorn running on http:\/\/0.0.0.0:8080 (Press CTRL+C to quit)<\/code><\/pre>\n<\/p>\n<\/div>\n<\/details>\n<figure class=\"full-width\"><img loading=\"lazy\" decoding=\"async\" src=\"https:\/\/habrastorage.org\/r\/w1560\/getpro\/habr\/upload_files\/8f1\/30e\/6f8\/8f130e6f80d3145bea6d41ce5faf39dc.png\" alt=\"\u0412\u0435\u0431-\u0438\u043d\u0442\u0435\u0440\u0444\u0435\u0439\u0441 PrivateGPT\" title=\"\u0412\u0435\u0431-\u0438\u043d\u0442\u0435\u0440\u0444\u0435\u0439\u0441 PrivateGPT\" width=\"1238\" height=\"930\" data-src=\"https:\/\/habrastorage.org\/getpro\/habr\/upload_files\/8f1\/30e\/6f8\/8f130e6f80d3145bea6d41ce5faf39dc.png\"\/><\/p>\n<div><figcaption>\u0412\u0435\u0431-\u0438\u043d\u0442\u0435\u0440\u0444\u0435\u0439\u0441 PrivateGPT<\/figcaption><\/div>\n<\/figure>\n<h4>\u041f\u0440\u0438\u043c\u0435\u0440\u044b \u043d\u0430\u0441\u0442\u0440\u043e\u0435\u043a<\/h4>\n<p>Mistral<\/p>\n<pre><code class=\"yaml\">tokenizer: mistralai\/Mistral-7B-Instruct-v0.2 llm_hf_repo_id: IlyaGusev\/saiga_mistral_7b_gguf llm_hf_model_file: model-q8_0.gguf<\/code><\/pre>\n<p>LLaMA<\/p>\n<pre><code class=\"yaml\">tokenizer: TheBloke\/Llama-2-13B-fp16 llm_hf_repo_id: IlyaGusev\/saiga2_13b_gguf llm_hf_model_file: model-q8_0.gguf<\/code><\/pre>\n<\/p>\n<\/div>\n<\/div>\n<\/div>\n<p><!----><!----><\/div>\n<p><!----><!----><br \/> \u0441\u0441\u044b\u043b\u043a\u0430 \u043d\u0430 \u043e\u0440\u0438\u0433\u0438\u043d\u0430\u043b \u0441\u0442\u0430\u0442\u044c\u0438 <a href=\"https:\/\/habr.com\/ru\/articles\/807469\/\"> https:\/\/habr.com\/ru\/articles\/807469\/<\/a><\/p>\n","protected":false},"excerpt":{"rendered":"<div><!--[--><!--]--><\/div>\n<div id=\"post-content-body\">\n<div>\n<div class=\"article-formatted-body article-formatted-body article-formatted-body_version-2\">\n<div xmlns=\"http:\/\/www.w3.org\/1999\/xhtml\">\n<p><a href=\"https:\/\/docs.privategpt.dev\/overview\/welcome\/introduction\" rel=\"noopener noreferrer nofollow\">PrivateGPT<\/a> \u2014 \u044d\u0442\u043e \u043f\u0440\u043e\u0435\u043a\u0442, \u043a\u043e\u0442\u043e\u0440\u044b\u0439 \u0440\u0430\u0441\u0448\u0438\u0440\u044f\u0435\u0442 \u0432\u043e\u0437\u043c\u043e\u0436\u043d\u043e\u0441\u0442\u0438 \u0440\u0430\u0431\u043e\u0442\u044b LLM-\u043c\u043e\u0434\u0435\u043b\u0435\u0439, \u043f\u043e\u0437\u0432\u043e\u043b\u044f\u044f \u0434\u043e\u0431\u0430\u0432\u043b\u044f\u0442\u044c \u043d\u0435\u043e\u0433\u0440\u0430\u043d\u0438\u0447\u0435\u043d\u043d\u043e\u0435 \u043a\u043e\u043b\u0438\u0447\u0435\u0441\u0442\u0432\u043e \u043b\u0438\u0447\u043d\u044b\u0445 \u0434\u0430\u043d\u043d\u044b\u0445.<\/p>\n<p>31 \u043e\u043a\u0442\u044f\u0431\u0440\u044f 2023 <a href=\"https:\/\/pytorch.org\/blog\/amd-extends-support-for-pt-ml\/\" rel=\"noopener noreferrer nofollow\">AMD Radeon \u043f\u0440\u0435\u0434\u043e\u0441\u0442\u0430\u0432\u0438\u043b\u0430 \u043f\u043e\u0434\u0434\u0435\u0440\u0436\u043a\u0443 PyTorch<\/a> \u0434\u043b\u044f \u043b\u044e\u0431\u0438\u0442\u0435\u043b\u044c\u0441\u043a\u0438\u0445 \u0432\u0438\u0434\u0435\u043e\u043a\u0430\u0440\u0442. \u041f\u043e\u043b\u043d\u044b\u0439 \u0441\u043f\u0438\u0441\u043e\u043a \u0432\u0438\u0434\u0435\u043e\u043a\u0430\u0440\u0442 \u0438 \u041e\u0421 \u043c\u043e\u0436\u043d\u043e \u043f\u043e\u0441\u043c\u043e\u0442\u0440\u0435\u0442\u044c <a href=\"https:\/\/rocm.docs.amd.com\/projects\/install-on-linux\/en\/latest\/reference\/system-requirements.html\" rel=\"noopener noreferrer nofollow\">\u0437\u0434\u0435\u0441\u044c<\/a>. \u041e\u043f\u0438\u0441\u0430\u043d\u043d\u0430\u044f \u0438\u043d\u0441\u0442\u0440\u0443\u043a\u0446\u0438\u044f \u043f\u0440\u043e\u0442\u0435\u0441\u0442\u0438\u0440\u043e\u0432\u0430\u043d\u0430 \u043d\u0430 AMD Radeon RX 7900 XTX.<\/p>\n<p>\u0414\u043b\u044f \u0437\u0430\u043f\u0443\u0441\u043a\u0430 \u043d\u0430\u043c \u043f\u043e\u043d\u0430\u0434\u043e\u0431\u0438\u0442\u0441\u044f Ubuntu \u0441 \u0443\u0441\u0442\u0430\u043d\u043e\u0432\u043b\u0435\u043d\u043d\u044b\u043c\u0438: git, make, docker \u0438 ROCm.<\/p>\n<p>ROCm \u0443\u0441\u0442\u0430\u043d\u043e\u0432\u0438\u043c \u043f\u043e <a href=\"https:\/\/rocm.docs.amd.com\/projects\/install-on-linux\/en\/latest\/tutorial\/quick-start.html\" rel=\"noopener noreferrer nofollow\">\u0438\u043d\u0441\u0442\u0440\u0443\u043a\u0446\u0438\u0438<\/a>.<\/p>\n<details class=\"spoiler\">\n<summary>\u041a\u0440\u0430\u0442\u043a\u0430\u044f \u0438\u043d\u0441\u0442\u0440\u0443\u043a\u0446\u0438\u044f \u0443\u0441\u0442\u0430\u043d\u043e\u0432\u043a\u0438 ROCm<\/summary>\n<div class=\"spoiler__content\">\n<pre><code class=\"bash\">sudo apt install \"linux-headers-$(uname -r)\" \"linux-modules-extra-$(uname -r)\" sudo usermod -a -G render,video $LOGNAME wget https:\/\/repo.radeon.com\/amdgpu-install\/6.0.2\/ubuntu\/jammy\/amdgpu-install_6.0.60002-1_all.deb sudo apt install .\/amdgpu-install_6.0.60002-1_all.deb  # \u041f\u0440\u0438 \u043e\u0448\u0438\u0431\u043a\u0435 \u043d\u0435\u043e\u0431\u0445\u043e\u0434\u0438\u043c\u043e \u0432\u044b\u043f\u043e\u043b\u043d\u0438\u0442\u044c: sudo chown -Rv _apt:root \/var\/cache\/apt\/archives\/partial\/ sudo chmod -Rv 700 \/var\/cache\/apt\/archives\/partial\/  sudo apt update sudo apt install amdgpu-dkms sudo apt install rocm-hip-libraries sudo reboot<\/code><\/pre>\n<\/p>\n<\/div>\n<\/details>\n<p>\u0421\u043a\u043e\u043f\u0438\u0440\u0443\u0435\u043c \u043f\u0440\u043e\u0435\u043a\u0442 \u043f\u043e\u0437\u0432\u043e\u043b\u044f\u044e\u0449\u0438\u0439 \u043d\u0430\u0441\u0442\u0440\u043e\u0438\u0442\u044c \u0438 \u0437\u0430\u043f\u0443\u0441\u0442\u0438\u0442\u044c PrivateGPT \u0432 docker \u043a\u043e\u043d\u0442\u0435\u0439\u043d\u0435\u0440\u0435:<\/p>\n<pre><code class=\"bash\">git clone https:\/\/github.com\/HardAndHeavy\/private-gpt-rocm-docker cd private-gpt-rocm-docker<\/code><\/pre>\n<p>\u0421\u043e\u0437\u0434\u0430\u0434\u0438\u043c \u0444\u0430\u0439\u043b \u043d\u0430\u0441\u0442\u0440\u043e\u0435\u043a PrivateGPT (<a href=\"https:\/\/github.com\/zylon-ai\/private-gpt\/blob\/main\/settings.yaml\" rel=\"noopener noreferrer nofollow\">settings.yaml<\/a>) \u0441 \u043f\u043e\u043c\u043e\u0449\u044c\u044e \u043a\u043e\u043c\u0430\u043d\u0434\u044b <code>make gen<\/code>. \u041d\u0430 \u0432\u0441\u0435 \u0437\u0430\u0434\u0430\u043d\u043d\u044b\u0435 \u0432\u043e\u043f\u0440\u043e\u0441\u044b \u043c\u043e\u0436\u043d\u043e \u043f\u0440\u043e\u0441\u0442\u043e \u043d\u0430\u0436\u0430\u0442\u044c \u043a\u043d\u043e\u043f\u043a\u0443 <code>\u0412\u0432\u043e\u0434<\/code>. \u0422\u043e\u0433\u0434\u0430 \u0444\u0430\u0439\u043b \u043d\u0430\u0441\u0442\u0440\u043e\u0435\u043a \u043f\u0440\u0438\u043c\u0435\u0442 \u0437\u043d\u0430\u0447\u0435\u043d\u0438\u0435 \u043f\u043e \u0443\u043c\u043e\u043b\u0447\u0430\u043d\u0438\u044e.<\/p>\n<details class=\"spoiler\">\n<summary>\u0413\u0435\u043d\u0435\u0440\u0430\u0446\u0438\u044f \u0444\u0430\u0439\u043b\u0430 \u043d\u0430\u0441\u0442\u0440\u043e\u0439\u043a\u0438<\/summary>\n<div class=\"spoiler__content\">\n<pre><code class=\"bash\">\u042f\u0437\u044b\u043a\u043e\u0432\u0430\u044f \u043c\u043e\u0434\u0435\u043b\u044c [IlyaGusev\/saiga_mistral_7b_gguf]:  \u0424\u0430\u0439\u043b \u044f\u0437\u044b\u043a\u043e\u0432\u043e\u0439 \u043c\u043e\u0434\u0435\u043b\u0438 [model-q8_0.gguf]:  \u042f\u0437\u044b\u043a\u043e\u0432\u0430\u044f \u043c\u043e\u0434\u0435\u043b\u044c \u0434\u043b\u044f \u0431\u043b\u043e\u043a\u043e\u0432 [mistralai\/Mistral-7B-Instruct-v0.2]:  \u041c\u043e\u0434\u0435\u043b\u044c \u0432\u0441\u0442\u0440\u0430\u0438\u0432\u0430\u043d\u0438\u044f [intfloat\/multilingual-e5-large]:<\/code><\/pre>\n<\/p>\n<\/div>\n<\/details>\n<p>\u0412 \u0440\u0435\u0437\u0443\u043b\u044c\u0442\u0430\u0442\u0435 \u043f\u043e\u044f\u0432\u0438\u0442\u0441\u044f \u0444\u0430\u0439\u043b <code>settings-gpt.yaml<\/code>, \u043a\u043e\u0442\u043e\u0440\u044b\u0439 \u0432 \u043f\u043e\u0441\u043b\u0435\u0434\u0443\u044e\u0449\u0435\u043c \u043c\u043e\u0436\u043d\u043e \u0434\u043e\u043d\u0430\u0441\u0442\u0440\u043e\u0438\u0442\u044c \u043f\u0440\u0438 \u0433\u043b\u0443\u0431\u043e\u043a\u043e\u043c \u0438\u0437\u0443\u0447\u0435\u043d\u0438\u0438 PrivateGPT.<\/p>\n<p>\u0417\u0430\u043f\u0443\u0441\u0442\u0438\u043c PrivateGPT \u0441 \u043f\u043e\u043c\u043e\u0449\u044c\u044e \u043a\u043e\u043c\u0430\u043d\u0434\u044b <code>make run<\/code>. \u041f\u0440\u0438 \u043f\u0435\u0440\u0432\u043e\u043c \u0437\u0430\u043f\u0443\u0441\u043a\u0435 \u0431\u0443\u0434\u0435\u0442 \u043f\u0440\u043e\u0438\u0441\u0445\u043e\u0434\u0438\u0442\u044c \u0434\u043b\u0438\u0442\u0435\u043b\u044c\u043d\u044b\u0439 \u043f\u0440\u043e\u0446\u0435\u0441\u0441 \u0441\u043a\u0430\u0447\u0438\u0432\u0430\u043d\u0438\u044f \u043c\u043e\u0434\u0435\u043b\u0438. \u041a\u043e\u0433\u0434\u0430 \u044d\u0442\u043e\u0442 \u043f\u0440\u043e\u0446\u0435\u0441\u0441 \u0437\u0430\u0432\u0435\u0440\u0448\u0438\u0442\u0441\u044f, PrivateGPT \u0441\u0442\u0430\u043d\u0435\u0442 \u0434\u043e\u0441\u0442\u0443\u043f\u0435\u043d \u043f\u043e \u0430\u0434\u0440\u0435\u0441\u0443 <a href=\"http:\/\/localhost\" rel=\"noopener noreferrer nofollow\">http:\/\/localhost<\/a>.<\/p>\n<details class=\"spoiler\">\n<summary>\u041b\u043e\u0433 \u043f\u0440\u043e\u0446\u0435\u0441\u0441\u0430 \u0437\u0430\u043f\u0443\u0441\u043a\u0430<\/summary>\n<div class=\"spoiler__content\">\n<pre><code class=\"bash\">docker run -it --rm \\                                                                                                                                                                                                -p 80:8080 \\                                                                                                                                                                                                 -v .\/local_data\/:\/app\/local_data \\                                                                                                                                                                           -v .\/models\/:\/app\/models \\                                                                                                                                                                                   -e PGPT_PROFILES=gpt \\                                                                                                                                                                                       -v .\/settings-gpt.yaml:\/app\/settings-gpt.yaml \\                                                                                                                                                              --device=\/dev\/kfd \\                                                                                                                                                                                          --device=\/dev\/dri \\                                                                                                                                                                                          hardandheavy\/private-gpt-rocm:latest                                                                                                                                                                 10:47:50.938 [INFO    ] private_gpt.settings.settings_loader - Starting application with profiles=['default', 'gpt']                                                                                         Downloading embedding intfloat\/multilingual-e5-large                                                                                                                                                         Fetching 19 files: 100%|\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588| 19\/19 [00:01&lt;00:00, 17.22it\/s] Embedding model downloaded!                                                                                                                                                                                  Downloading LLM model-q8_0.gguf                                                                                                                                                                              LLM model downloaded!                                                                                                                                                                                        Downloading tokenizer mistralai\/Mistral-7B-Instruct-v0.2                                                                                                                                                     Tokenizer downloaded!                                                                                                                                                                                        Setup done                                                                                                                                                                                                   poetry run python -m private_gpt                                                                                                                                                                             10:47:54.978 [INFO    ] private_gpt.settings.settings_loader - Starting application with profiles=['default', 'gpt']                                                                                         10:48:00.396 [INFO    ]   matplotlib.font_manager - generated new fontManager                                                                                                                                10:48:01.914 [INFO    ] private_gpt.components.llm.llm_component - Initializing the LLM in mode=llamacpp                                                                                                     ggml_init_cublas: GGML_CUDA_FORCE_MMQ:   no                                                                                                                                                                  ggml_init_cublas: CUDA_USE_TENSOR_CORES: yes                                                                                                                                                                 ggml_init_cublas: found 1 ROCm devices:                                                                                                                                                                        Device 0: Radeon RX 7900 XTX, compute capability 11.0, VMM: no                                                                                                                                             llama_model_loader: loaded meta data with 21 key-value pairs and 291 tensors from \/app\/models\/model-q8_0.gguf (version GGUF V2)                                                                              llama_model_loader: Dumping metadata keys\/values. Note: KV overrides do not apply in this output.                                                                                                            llama_model_loader: - kv   0:                       general.architecture str              = llama                                                                                                            llama_model_loader: - kv   1:                               general.name str              = models                                                                                                           llama_model_loader: - kv   2:                       llama.context_length u32              = 32768                                                                                                            llama_model_loader: - kv   3:                     llama.embedding_length u32              = 4096                                                                                                             llama_model_loader: - kv   4:                          llama.block_count u32              = 32                                                                                                               llama_model_loader: - kv   5:                  llama.feed_forward_length u32              = 14336                                                                                                            llama_model_loader: - kv   6:                 llama.rope.dimension_count u32              = 128                                                                                                              llama_model_loader: - kv   7:                 llama.attention.head_count u32              = 32                                                                                                               llama_model_loader: - kv   8:              llama.attention.head_count_kv u32              = 8                                                                                                                llama_model_loader: - kv   9:     llama.attention.layer_norm_rms_epsilon f32              = 0.000010                                                                                                         llama_model_loader: - kv  10:                       llama.rope.freq_base f32              = 10000.000000                                                                                                     llama_model_loader: - kv  11:<\/code><\/pre>\n<\/div>\n<\/details>\n<\/div>\n<\/div>\n<\/div>\n<\/div>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[],"tags":[],"class_list":["post-373549","post","type-post","status-publish","format-standard","hentry"],"_links":{"self":[{"href":"https:\/\/savepearlharbor.com\/index.php?rest_route=\/wp\/v2\/posts\/373549","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/savepearlharbor.com\/index.php?rest_route=\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/savepearlharbor.com\/index.php?rest_route=\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/savepearlharbor.com\/index.php?rest_route=\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/savepearlharbor.com\/index.php?rest_route=%2Fwp%2Fv2%2Fcomments&post=373549"}],"version-history":[{"count":0,"href":"https:\/\/savepearlharbor.com\/index.php?rest_route=\/wp\/v2\/posts\/373549\/revisions"}],"wp:attachment":[{"href":"https:\/\/savepearlharbor.com\/index.php?rest_route=%2Fwp%2Fv2%2Fmedia&parent=373549"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/savepearlharbor.com\/index.php?rest_route=%2Fwp%2Fv2%2Fcategories&post=373549"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/savepearlharbor.com\/index.php?rest_route=%2Fwp%2Fv2%2Ftags&post=373549"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}