{"id":362495,"date":"2024-05-21T01:36:15","date_gmt":"2024-05-21T01:36:15","guid":{"rendered":"http:\/\/savepearlharbor.com\/?p=362495"},"modified":"-0001-11-30T00:00:00","modified_gmt":"2024-05-21T01:36:15","slug":"","status":"publish","type":"post","link":"https:\/\/savepearlharbor.com\/?p=362495","title":{"rendered":"<span>Memory consumption of .NET applications on Linux<\/span>"},"content":{"rendered":"<div><!--[--><!--]--><\/div>\n<div id=\"post-content-body\">\n<div>\n<div class=\"article-formatted-body article-formatted-body article-formatted-body_version-2\">\n<div xmlns=\"http:\/\/www.w3.org\/1999\/xhtml\">\n<h2>Introduction<\/h2>\n<p>In this article, I will cover the memory consumption of .NET applications on Linux. Firstly, we will try to understand the idea of virtual memory. Then, we will examine the memory statistics that Linux provides, such as RSS (Resident Set Size), VmData, RES (Resident Memory Size), and SWAP. Following that, we will delve into specifics related to the .NET.<\/p>\n<h2>Basic memory stats in Linux<\/h2>\n<p>Let\u2019s create a basic ASP.NET application from the default template, publish it in framework-dependent mode and run it on Linux.<\/p>\n<p>I\u2019ve started the application on Linux and executed the top command.<\/p>\n<figure class=\"full-width\"><img loading=\"lazy\" decoding=\"async\" src=\"https:\/\/habrastorage.org\/r\/w1560\/getpro\/habr\/upload_files\/474\/bc3\/d6f\/474bc3d6f0539fec0c1905350eae6e9e.png\" width=\"1438\" height=\"631\" data-src=\"https:\/\/habrastorage.org\/getpro\/habr\/upload_files\/474\/bc3\/d6f\/474bc3d6f0539fec0c1905350eae6e9e.png\"\/><\/figure>\n<p>We have three memory-related statistics here: 262Gb, 66Mb and 57Mb, but how do we find the \u201ctrue\u201d memory consumption? Unfortunately, it\u2019s impossible unless we understand the idea of virtual memory and the principles of memory allocation.<\/p>\n<h2>Virtual memory<\/h2>\n<p>In older operation systems and on devices like Arduino, you have direct access to physical memory. It means that operations like mov 0, 42 would actually load \u201c42\u201d to the very beginning of the CPU\u2019s physical memory. But modern OSs like Linux and Windows, with help from CPU don\u2019t give you access to physical memory. A mechanism called virtualisation is used instead.<\/p>\n<p>Let\u2019s say we have two processes which load some files from the disk and allocate some memory to store data.<\/p>\n<p>Each of the processes has so-called virtual memory. Only virtual memory can be accessed inside the process. When you try to put 42 to a memory address like 7f849c021000 it doesn\u2019t mean that this very address of the physical memory will be accessed. Moreover, it doesn\u2019t mean any physical storage is allocated before you access it.<\/p>\n<figure class=\"full-width\"><img loading=\"lazy\" decoding=\"async\" src=\"https:\/\/habrastorage.org\/r\/w1560\/getpro\/habr\/upload_files\/fac\/cf5\/d4b\/faccf5d4b9d318d049b26bbe071d4d98.png\" width=\"811\" height=\"672\" data-src=\"https:\/\/habrastorage.org\/getpro\/habr\/upload_files\/fac\/cf5\/d4b\/faccf5d4b9d318d049b26bbe071d4d98.png\"\/><\/figure>\n<p>Let\u2019s look at the diagram showing two processes and go through all the memory segments these processes have in the virtual memory space.<\/p>\n<h4>First process<\/h4>\n<ul>\n<li>\n<p>Segment 1.1 &#8212; it\u2019s a so-called anonymous memory. Anonymous memory is a memory that doesn\u2019t correspond to any file on a disk. It\u2019s a type of memory you get when calling malloc in c or new in C#.<\/p>\n<\/li>\n<li>\n<p>Segment 1.2 It\u2019s a <a href=\"https:\/\/en.wikipedia.org\/wiki\/Memory-mapped_file\" rel=\"noopener noreferrer nofollow\"><u>memory-mapped file<\/u><\/a>. When you read this memory at the beginning of the segment, you\u2019ll get bytes from the beginning of the file. Memory-mapped files are a completely different approach than .NET File.Open or fopen in C. It is mainly used to load dynamic libraries (DLL in .NET and Windows and so in Linux). In this example, no physical memory is allocated. There can be two reasons for that. The first reason is this segment was never accessed, and the Linux kernel was too lazy to load it in advance. The second reason is that this file is rarely accessed, and the kernel removed it from physical memory. Nevertheless, the kernel will load the file or, most likely, only a small part of it into physical memory once the application tries to access memory from this segment.\u00a0<\/p>\n<\/li>\n<li>\n<p>Segment 1.3 is a memory-mapped file as well. It\u2019s actively used, so the kernel loads all the content into the memory. This file and the physical memory range are shared with the second process.<\/p>\n<\/li>\n<li>\n<p>Segment 1.4 is an anonymous memory-mapped file. This functionality allows for data exchange between processes. In .NET, the MemoryMappedFile class enables access to this feature.<\/p>\n<\/li>\n<li>\n<p>This segment is shared between processes but not mapped to any disk file.<\/p>\n<\/li>\n<\/ul>\n<h4>Second process<\/h4>\n<ul>\n<li>\n<p>Segment 2.1 is the same as segment 1.3 of the first file<\/p>\n<\/li>\n<li>\n<p>Segment 2.2 is the same as segment 1.4 of the first file<\/p>\n<\/li>\n<li>\n<p>Segment 2.3 is an ordinary private anonymous memory like segment 1.1 from the first process.<\/p>\n<\/li>\n<li>\n<p>Segment 2.4 is a memory-mapped file. The part of the file is frequently accessed, so the kernel allocated a physical memory for it, but some parts of this file are accessed very seldom or not accessed at all. That\u2019s why the kernel hasn\u2019t yet allocated physical memory for this file part.<\/p>\n<\/li>\n<li>\n<p>Segment 2.5 is an anonymous segment like 2.2, and the process writes to all the parts of the segments, so it used to be fully allocated. Then, this memory was not accessed for a long time, and there was a shortage of physical memory, so the kernel decided to swap this segment to the disk swap segment.<\/p>\n<\/li>\n<\/ul>\n<p>Let\u2019s try to generalise all these cases:<\/p>\n<ul>\n<li>\n<p>Anonymous vs. file-backed<\/p>\n<ul>\n<li>\n<p>We use an anonymous memory when we allocate some virtual memory via malloc or new.<\/p>\n<\/li>\n<li>\n<p>When we want to map some disk file to a memory, we use a memory-mapped file.<\/p>\n<\/li>\n<\/ul>\n<\/li>\n<li>\n<p>There is a difference between virtual memory allocation by malloc and physical memory allocation. <strong>Unaccessed memory, in most cases, is not physically allocated<\/strong>.<\/p>\n<\/li>\n<li>\n<p>The kernel tries not to hold unused data in physical memory. There can be two different cases:<\/p>\n<ul>\n<li>\n<p>For anonymous blocks, memory would be swapped, meaning that the kernel would write it to a swap segment<\/p>\n<\/li>\n<li>\n<p>For a not modified file-backed memory, the kernel can throw away data from physical memory and read it once again on demand if a process doesn\u2019t change it.<\/p>\n<\/li>\n<\/ul>\n<\/li>\n<\/ul>\n<h2>Linux memory stat<\/h2>\n<p>In Linux, there are several sources of memory-related information:<\/p>\n<ul>\n<li>\n<p>\/proc\/{proc id}\/smaps most accurate and detailed information. Divided by individual segments.<\/p>\n<\/li>\n<li>\n<p>\/proc\/{proc id}\/smaps_rollup the same as previous but summed up.<\/p>\n<\/li>\n<li>\n<p>\/proc\/{proc id}\/status much faster but less accurate<\/p>\n<\/li>\n<li>\n<p>top and ps same figures as above but less detailed<\/p>\n<\/li>\n<\/ul>\n<p>To get all the data at the same moment, I execute the following:<\/p>\n<pre><code>$ cat \/proc\/$(pgrep Memory)\/smaps > mem.txt;cat \/proc\/$(pgrep Memory)\/smaps_rollup;cat \/proc\/$(pgrep Memory)\/status;pmap $(pgrep Memory) -x > smap.txt;top  559468e18000-7ffc8b1ef000 ---p 00000000 00:00 0                      [rollup] Rss:           87812 kB Pss:           78760 kB Pss_Anon:      23416 kB Pss_File:      43144 kB Pss_Shmem:     12200 kB Shared_Clean:  16184 kB Shared_Dirty:      8 kB Private_Clean: 36052 kB Private_Dirty: 35568 kB Referenced:    84432 kB Anonymous:     23416 kB LazyFree:          0 kB AnonHugePages: 12288 kB ShmemPmdMapped:    0 kB FilePmdMapped:     0 kB Shared_Hugetlb:    0 kB Private_Hugetlb:   0 kB Swap:              0 kB SwapPss:           0 kB Locked:            0 kB    Name:   Memory Umask:  0022 State:  S (sleeping) Tgid:   10000 Ngid:   0 Pid:10000 PPid:   169 TracerPid:  0 Uid:1000100010001000 Gid:1000100010001000 FDSize: 256 Groups: 4 20 24 25 27 29 30 44 46 116 1000 1001 NStgid: 10000 NSpid:  10000 NSpgid: 10000 NSsid:  169 VmPeak: 275330800 kB VmSize: 275248656 kB VmLck:     0 kB VmPin:     0 kB VmHWM: 87360 kB VmRSS: 87360 kB RssAnon:       23096 kB RssFile:       52128 kB RssShmem:      12136 kB VmData:  1244932 kB VmStk:   132 kB VmExe:    92 kB VmLib: 42836 kB VmPTE:   700 kB VmSwap:    0 kB HugetlbPages:      0 kB CoreDumping:0 THP_enabled:1 Threads:    30 SigQ:   3\/61471 SigPnd: 0000000000000000 ShdPnd: 0000000000000000 SigBlk: 0000000000000000 SigIgn: 0000000000001000 SigCgt: 00000003080344fe CapInh: 0000000000000000 CapPrm: 0000000000000000 CapEff: 0000000000000000 CapBnd: 000001ffffffffff CapAmb: 0000000000000000 NoNewPrivs: 0 Seccomp:    0 Seccomp_filters:    0 Speculation_Store_Bypass:   thread vulnerable SpeculationIndirectBranch:  conditional enabled Cpus_allowed:   ffff Cpus_allowed_list:  0-15 Mems_allowed:   1 Mems_allowed_list:  0 voluntary_ctxt_switches:    70 nonvoluntary_ctxt_switches: 2 top - 20:15:05 up 15:44,  0 users,  load average: 0.00, 0.01, 0.01 Tasks:  25 total,   1 running,  24 sleeping,   0 stopped,   0 zombie %Cpu(s):  0.0 us,  0.0 sy,  0.0 ni,100.0 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st MiB Mem :  15374.3 total,459.8 free,   1654.7 used,  13259.8 buff\/cache MiB Swap:   4096.0 total,   4095.2 free,  0.8 used.  13380.6 avail Mem    PID USER  PR  NIVIRTRESSHR S  %CPU  %MEM TIME+ COMMAND   259 root  20   02340120  0 S   0.0   0.0   0:00.60 Relay(260)   260 rumata20   06208   5196   3408 S   0.0   0.0   0:00.64 bash   483 root  20   02340112  0 S   0.0   0.0   0:00.00 SessionLeader   484 root  20   02340120  0 S   0.0   0.0   0:00.19 Relay(485)   485 rumata20   06340   5312   3416 S   0.0   0.0   0:00.17 bash   882 root  20   02340112  0 S   0.0   0.0   0:00.00 SessionLeader   883 root  20   02340120  0 S   0.0   0.0   0:00.03 Relay(884)   884 rumata20   06340   5272   3416 S   0.0   0.0   0:00.10 bash   998 root  20   02340112  0 S   0.0   0.0   0:00.00 SessionLeader   999 root  20   02340120  0 S   0.0   0.0   0:00.02 Relay(1000)  1000 rumata20   06324   5404   3560 S   0.0   0.0   0:00.04 bash 10000 rumata20   0  262.5g  87360  64264 S   0.0   0.6   0:01.56 Memory 10289 rumata20   02772952864 S   0.0   0.0   0:00.00 memtest_sf<\/code><\/pre>\n<p>Let\u2019s take the most important values for each of these outputs and try to understand what do they mean and how to calculate them using smap.<\/p>\n<p>There are two primary data sources I use to understand the meaning of these outputs deeply:<\/p>\n<p><a href=\"https:\/\/www.kernel.org\/doc\/Documentation\/filesystems\/proc.txt\" rel=\"noopener noreferrer nofollow\"><u>https:\/\/www.kernel.org\/doc\/Documentation\/filesystems\/proc.txt<\/u><\/a><\/p>\n<p>https:\/\/man7.org\/linux\/man-pages\/man5\/proc.5.html<\/p>\n<h4>smaps_rollup<\/h4>\n<p><code>Rss: \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 87812 kB<\/code>\u00a0<\/p>\n<p>RSS is a \u201cthe amount of the mapping that is currently resident in RAM\u201d or, in other words, the physical memory consumption.<\/p>\n<pre><code>Pss: \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 78760 kB Pss_Anon:\u00a0 \u00a0 \u00a0 23416 kB Pss_File:\u00a0 \u00a0 \u00a0 43144 kB Pss_Shmem: \u00a0 \u00a0 12200 kB<\/code><\/pre>\n<p>\u201cproportional share of this mapping.\u201d I\u2019ll cover this in more detail a little bit later.<\/p>\n<pre><code>Shared_Clean:\u00a0 16184 kB Shared_Dirty:\u00a0 \u00a0 \u00a0 8 kB Private_Clean: 36052 kB Private_Dirty: 35568 kB<\/code><\/pre>\n<p>Each memory segment can be mapped by multiple processes. In this statistic, memory accessed by one process with sharable permissions will go to private. It will become shared only when the second process maps it.<\/p>\n<p>Clean memory is a memory we haven\u2019t modified. It can be either a memory-mapped file that we\u2019ve loaded from the disk and haven\u2019t modified or an anonymous memory we\u2019ve mapped but haven\u2019t written. Clean means that the kernel can swipe out this allocation from physical memory without writing it to the disk swap segment.<\/p>\n<p><code>Referenced:\u00a0 \u00a0 84432 kB<\/code><\/p>\n<p>\u201cthe amount of memory currently marked as referenced or accessed.\u201d<\/p>\n<p><code>Anonymous: \u00a0 \u00a0 23416 kB<\/code><\/p>\n<p>\u201cthe amount of memory that does not belong to any file\u201d. It\u2019s a memory allocated when we use malloc in c or new in C#.<\/p>\n<p>Anonymous memory can appear even in the file-backed segment when an application writes to this memory.<\/p>\n<h4>smap<\/h4>\n<p>smaps_rollup is a rollup of the very detailed smap which gives us details about each individual segment. It\u2019s too large to put it here, but it has the same fields as the rollup version.<\/p>\n<h4>status<\/h4>\n<p>Before we go to the figures, it\u2019s important to mention that all these values are inaccurate. Here is the quote from Linux documentation:<\/p>\n<blockquote>\n<p>Some of these values are inaccurate because of a kernel-internal scalability optimization.\u00a0 If accurate values are\u00a0required, use <em>\/proc\/<\/em>pid<em>\/smaps<\/em> or <em>\/proc\/<\/em>pid<em>\/smaps_rollup<\/em>\u00a0instead, which are much slower but provide accurate,\u00a0detailed information.<\/p>\n<\/blockquote>\n<p>So, there always be some difference between process\/ps\/top and smap.<\/p>\n<p><code>VmSize: 275248656 kB<\/code><\/p>\n<p>Virtual memory size. It\u2019s about 256Gb, which is an enormous amount of memory. I\u2019ll explain why we see such a huge amount here a little bit later.<\/p>\n<p><code>VmRSS: 87360 kB<\/code><\/p>\n<p>Current virtual memory consumption.<\/p>\n<p>VmRSS = RssAnon + RssFile + RssShmem<\/p>\n<p><code>RssAnon: \u00a0 \u00a0 \u00a0 23096 kB<\/code><\/p>\n<p>It&#8217;s the same as Anonymous from smap.<\/p>\n<p><code>RssFile: \u00a0 \u00a0 \u00a0 52128 kB<\/code><\/p>\n<p>Memory-mapped files or, in other words, segments with non-empty file fields excluding the following file names:<\/p>\n<ul>\n<li>\n<p>\u00a0[stack] and [heap] which go to anonymous<\/p>\n<\/li>\n<li>\n<p>\/memfd\/* and \/dev\/shm\/* which go to RssShmem<\/p>\n<\/li>\n<\/ul>\n<p><code>RssShmem:\u00a0 \u00a0 \u00a0 12136 kB<\/code><\/p>\n<p>Non-file-backed shared memory or, in other words, segments with \/memfd\/* and \/dev\/shm\/* file names<\/p>\n<p><code>VmSwap:\u00a0 \u00a0 0 kB<\/code><\/p>\n<p>Swapped memory<\/p>\n<h4>top<\/h4>\n<pre><code>PID   USER\u00a0 PR\u00a0 NIVIRTRES   SHR   S\u00a0 %CPU\u00a0  %MEM   TIME+ COMMAND 10000 rumata20 \u00a0 0\u00a0 262.5g\u00a0 87360\u00a0 64264 S \u00a0 0.0 \u00a0 0.6 \u00a0 0:01.56 Memory<\/code><\/pre>\n<p>Here we have the following figures<\/p>\n<ul>\n<li>\n<p>Virt is the same as VmSize. Don\u2019t forget that here we have Gb, so we need to multiply it by 1024*1024, not 1000000, to get kb as in proc output.<\/p>\n<\/li>\n<li>\n<p>Res is the same as VmRSS<\/p>\n<\/li>\n<li>\n<p>Shr equals VmRSS &#8212; RssAnon<\/p>\n<\/li>\n<\/ul>\n<h4>Pss<\/h4>\n<p>Here is the quote from the Linux documentation:\u00a0<\/p>\n<blockquote>\n<p>The following lines show the size of the mapping, the amount of the mapping that is currently resident in RAM (&#171;Rss&#187;), the process&#8217;s proportional share of this mapping (&#171;Pss&#187;)<\/p>\n<\/blockquote>\n<p>So, RSS is an actual RAM consumption, but it includes all the shred segments. In our example in the \u201cVirtual memory\u201d paragraph, segments 1.3 and 1.4 will be counted in full for both processes. When I sum up RSS for both processes, segments 1.3 and 1.4 will be counted twice, which doesn\u2019t correctly represent physical memory consumption.<\/p>\n<p>Pss tries to mitigate this problem by measuring shared pages proportionally, so if 5 processes would use one shared file, pss would count only \u2155 of the consumption for each process.<\/p>\n<p>Let\u2019s try to start two instances of the application and look at the smaps_rollup of the <strong>same <\/strong>app instance when only one instance is running and when two instances are running.<\/p>\n<figure class=\"\"><img loading=\"lazy\" decoding=\"async\" src=\"https:\/\/habrastorage.org\/r\/w1560\/getpro\/habr\/upload_files\/054\/67a\/62d\/05467a62dae217ade00fef032ca435cf.png\" width=\"415\" height=\"386\" data-src=\"https:\/\/habrastorage.org\/getpro\/habr\/upload_files\/054\/67a\/62d\/05467a62dae217ade00fef032ca435cf.png\"\/><\/figure>\n<p>\u2026<\/p>\n<p>Here, we can see that Pss was reduced just because of the start of another process.<\/p>\n<h2>How does .NET store the data?<\/h2>\n<p>Previously, we talked about general Linux concepts. Now, let\u2019s talk about .NET. I would say that we have the following memory areas:<\/p>\n<ul>\n<li>\n<p>Heap<\/p>\n<\/li>\n<li>\n<p>Stack<\/p>\n<\/li>\n<\/ul>\n<p>Let\u2019s understand where each of these data types is located.<\/p>\n<h3>Heap<\/h3>\n<p>Let\u2019s start with a small experiment. I\u2019ve executed cat \/proc\/{proc id}\/status against a tiny .net console program and got the following output:<\/p>\n<pre><code>VmSize: 275116856 kB VmLck:     0 kB VmPin:     0 kB VmHWM: 67668 kB VmRSS: 65932 kB RssAnon:        8924 kB RssFile:       47500 kB RssShmem:       9508 kB VmData:   188776 kB VmStk:   132 kB VmExe:    92 kB VmLib: 41536 kB VmPTE:   596 kB VmSwap:    0 kB<\/code><\/pre>\n<p>Looks like my tiny program consumes 256GB of memory! In fact, this figure shows only the virtual memory. As you remember, not accessed virtual memory is not allocated, but let\u2019s dive deeper.<\/p>\n<p>We need to find segments with such high virtual memory consumption. It turns out that there is only one such segment:<\/p>\n<pre><code>7fa114001000-7fe088000000 ---p 00000000 00:00 0 Size:       266141692 kB KernelPageSize:    4 kB MMUPageSize:       4 kB Rss:               0 kB Pss:               0 kB Shared_Clean:      0 kB Shared_Dirty:      0 kB Private_Clean:     0 kB Private_Dirty:     0 kB Referenced:        0 kB Anonymous:         0 kB LazyFree:          0 kB AnonHugePages:     0 kB ShmemPmdMapped:    0 kB FilePmdMapped:     0 kB Shared_Hugetlb:    0 kB Private_Hugetlb:   0 kB Swap:              0 kB SwapPss:           0 kB Locked:            0 kB THPeligible:1 VmFlags: mr mw me dd<\/code><\/pre>\n<p>As you can see, Rss is zero. Moreover, the program cannot write to this memory because of the permissions.<\/p>\n<p>Why .NET needs this? That\u2019s because .NET garbage collector needs continuous address space for the heap, so it allocates the <a href=\"https:\/\/github.com\/dotnet\/runtime\/blob\/26158bfdf528fe164634f79777313f9cd7e310a9\/src\/coreclr\/gc\/unix\/gcenv.unix.cpp#L1091\" rel=\"noopener noreferrer nofollow\"><u>maximum available amount<\/u><\/a> of address space.<\/p>\n<p>When we create a new huge object on the heap, .NET allocates a part of this address space with read-write permissions and lets the program use it.<\/p>\n<p>Let\u2019s see how it works. I have the following controller in my .NET application.<\/p>\n<pre><code class=\"cs\">private static byte[] _hugeArray; private static byte[] _tinyArray;  [HttpPost] public void AllocateALot() {    var gb = 1 * 1024 * 1024 * 1024;    _hugeArray = new byte[gb];       _logger.LogInformation(\"AllocateALot done\"); }   [HttpPost] public void AllocateALittle() {    _tinyArray = new byte[1024];       _logger.LogInformation(\"AllocateALittle done\"); }   [HttpPost] public void WriteHuge(int index) {    _hugeArray[index] = 1;       _logger.LogInformation(\"WriteHuge done\"); }   [HttpPost] public void WriteTiny(int index) {    _tinyArray[index] = 1;       _logger.LogInformation(\"WriteTiny done\"); }<\/code><\/pre>\n<p>I started my application and executed the following commands<\/p>\n<pre><code>$ cat \/proc\/$(pgrep Memory)\/smaps > mem_before.txt  $ curl -X 'POST' \u00a0 'http:\/\/localhost:5000\/Experiments\/AllocateALot'  $ cat \/proc\/$(pgrep Memory)\/smaps > mem_after.txt<\/code><\/pre>\n<p>I used a Python script for smap analysis, and here are the results (size in Mb).<\/p>\n<figure class=\"full-width\"><img loading=\"lazy\" decoding=\"async\" src=\"https:\/\/habrastorage.org\/r\/w1560\/getpro\/habr\/upload_files\/9b2\/440\/643\/9b2440643a79fc42fa45480479dd11a9.png\" width=\"1557\" height=\"523\" data-src=\"https:\/\/habrastorage.org\/getpro\/habr\/upload_files\/9b2\/440\/643\/9b2440643a79fc42fa45480479dd11a9.png\"\/><\/figure>\n<p>.NET reduced that enormous segment and created two new segments. The size of the new array is 1Gb but only 2Mb was actually allocated.<\/p>\n<p>But let\u2019s try to write something to this array. I\u2019m going to write exactly in the middle of the array<\/p>\n<p>$ curl -X &#8216;POST&#8217; \u00a0 &#8216;<a href=\"http:\/\/localhost:5000\/Experiments\/WriteHuge?index=536870912\" rel=\"noopener noreferrer nofollow\"><u>http:\/\/localhost:5000\/Experiments\/WriteHuge?index=536870912<\/u><\/a>&#8216;<\/p>\n<figure class=\"full-width\"><img loading=\"lazy\" decoding=\"async\" src=\"https:\/\/habrastorage.org\/r\/w1560\/getpro\/habr\/upload_files\/b30\/ea2\/7e4\/b30ea27e41380408e88331f969c7b1ab.png\" width=\"721\" height=\"131\" data-src=\"https:\/\/habrastorage.org\/getpro\/habr\/upload_files\/b30\/ea2\/7e4\/b30ea27e41380408e88331f969c7b1ab.png\"\/><\/figure>\n<p>As we can see, RSS has grown from 2Kb to 4Kb, as one more page was allocated in the middle of the segment.<\/p>\n<p>Let\u2019s allocate a tiny array as well. I\u2019m not expecting .NET to create a new segment for it; that\u2019s why I use a dotnet-dump to get an address of this variable.<\/p>\n<pre><code>$ .\/dotnet-dump collect -p $(pgrep Memory) --type Full  Writing full to \/home\/rumata\/core_20231105_131345 Complete $ .\/dotnet-dump analyze \/home\/rumata\/core_20231105_131345 > dumpheap -type Memory.Controllers.ExperimentsController      Address           MT       Size 7ec94f814438 7f093cbd19b0         24 7ec950408480 7f093cbd19b0         24 7ec951c31628 7f093cbd19b0         24 7ec951c31b88 7f093d5dd1e8         24  Statistics:       MT Count TotalSize Class Name 7f093d5dd1e8 1    24 Microsoft.Extensions.Logging.Logger&lt;Memory.Controllers.ExperimentsController> 7f093cbd19b0 3    72 Memory.Controllers.ExperimentsController Total 4 objects, 96 bytes  > dumpobj 7ec94f814438 Name:    Memory.Controllers.ExperimentsController MethodTable: 00007f093cbd19b0 EEClass: 00007f093cbb7c20 Tracked Type: false Size:    24(0x18) bytes File:    \/home\/rumata\/MemoryTest\/Memory.dll Fields:           MTField   Offset             Type VT Attr        Value Name 00007f093a5e9238  4000008    8 ...s.Logging.ILogger  0 instance 00007ec951c31b88 _logger 00007f09396f41c0  4000007    8      System.Void  0   static 0000000000000000 _file 00007f093a5ba158  4000009   10    System.Byte[]  0   static 00007ec978000040 _hugeArray 00007f093a5ba158  400000a   18    System.Byte[]  0   static 00007ec94f814530 _tinyArray <\/code><\/pre>\n<p>I look at the MT(method table) to get the instance of the correct type, and I can choose any instance because I\u2019m interested in a static field.<\/p>\n<p>Now I can find the segment where this field sits:<\/p>\n<figure class=\"full-width\"><img loading=\"lazy\" decoding=\"async\" src=\"https:\/\/habrastorage.org\/r\/w1560\/getpro\/habr\/upload_files\/41f\/20d\/f9b\/41f20df9b5fe37bde4f0415d51d8df28.png\" width=\"712\" height=\"50\" data-src=\"https:\/\/habrastorage.org\/getpro\/habr\/upload_files\/41f\/20d\/f9b\/41f20df9b5fe37bde4f0415d51d8df28.png\"\/><\/figure>\n<p>This particular allocation hasn\u2019t changed any memory consumption figures. The memory was allocated in the existing segment, and physical RAM was allocated for this area earlier. It means that we can\u2019t see this particular allocation in any Linux memory statistics.<\/p>\n<h4>Stack<\/h4>\n<p>The easiest way to get the addresses of all the threads is to create a dump via dotnet-dump and then execute sos EEStack command.<\/p>\n<pre><code>$ .\/dotnet-dump collect -p $(pgrep Memory) --type Full $ .\/dotnet-dump analyze \/home\/rumata\/core_20231130_190143 > sos EEStack<\/code><\/pre>\n<p>The output would include stack traces for all the threads, including stack addresses of each frame. Let\u2019s get the bottom frame for each thread and then map it to the segments list.<\/p>\n<figure class=\"full-width\"><img loading=\"lazy\" decoding=\"async\" src=\"https:\/\/habrastorage.org\/r\/w1560\/getpro\/habr\/upload_files\/70c\/101\/5d6\/70c1015d6324619308c32b60a8643a21.png\" width=\"1072\" height=\"473\" data-src=\"https:\/\/habrastorage.org\/getpro\/habr\/upload_files\/70c\/101\/5d6\/70c1015d6324619308c32b60a8643a21.png\"\/><\/figure>\n<p>The last column here shows the thread ID of a thread which uses this segment as a stack.<\/p>\n<p>Let\u2019s discuss the sizes of the stack segments. In Linux, the stack segment for the initial thread has a dynamic size. It\u2019s located at the bottom of the address space and has a lot of space to grow. Keep in mind the stack grows from the lower addresses to higher addresses.<\/p>\n<figure class=\"full-width\"><img loading=\"lazy\" decoding=\"async\" src=\"https:\/\/habrastorage.org\/r\/w1560\/getpro\/habr\/upload_files\/dac\/be3\/9b0\/dacbe39b0c9837ca06d3f904da015a11.png\" width=\"1077\" height=\"323\" data-src=\"https:\/\/habrastorage.org\/getpro\/habr\/upload_files\/dac\/be3\/9b0\/dacbe39b0c9837ca06d3f904da015a11.png\"\/><\/figure>\n<p>So, once the program tries to access the memory out of the initially allocated space, the system will intercept the segmentation fault and enlarge the segment. But there is still a limit. You can get it by executing the following:<\/p>\n<pre><code>$ ulimit -s 8192<\/code><\/pre>\n<p>This value is used as a default thread size for all other threads as well, but can be overridden using the following .NET overload:<\/p>\n<p><code>public Thread (System.Threading.ThreadStart start, int maxStackSize);<\/code><\/p>\n<p>An interesting fact for readers who began with the .NET Framework on Windows: early versions of the .NET Framework committed all the stack memory, which is 1Mb by default on Windows. Commit allocates some space in the swap file or physical RAM to ensure that there will be no out-of-memory under any circumstances. In Linux, there is no such idea of memory commit because it treats physical memory plus swap as a nearly unlimited resource and allows overallocation. The only way to ensure that virtual space is backed by physical RAM is to access it. But on Linux .NET don\u2019t try to access all the stacks to be sure that all the memory is physically allocated and stack overflow can be handled gracefully. The positive side of this approach is small memory consumption by the stacks. As you can see in our example, all the stacks consume less than 50k RSS<\/p>\n<p>Conclusion<\/p>\n<p>Thus, to understand Linux memory statistics, we dived into the idea of virtual address space. This knowledge allowed us to understand all the most important metrics like RSS, anonymous and file-backed, private and shared memory. Considering all these, we revealed how .NET stores data in the heap and the stack.<\/p>\n<p>Understanding these mechanisms is important for optimising .NET applications and ensuring their efficient operation in a Linux environment.<\/p>\n<\/p>\n<\/div>\n<\/div>\n<\/div>\n<p><!----><!----><\/div>\n<p><!----><!----><br \/> \u0441\u0441\u044b\u043b\u043a\u0430 \u043d\u0430 \u043e\u0440\u0438\u0433\u0438\u043d\u0430\u043b \u0441\u0442\u0430\u0442\u044c\u0438 <a href=\"https:\/\/habr.com\/ru\/articles\/779564\/\"> https:\/\/habr.com\/ru\/articles\/779564\/<\/a><\/p>\n","protected":false},"excerpt":{"rendered":"<div><!--[--><!--]--><\/div>\n<div id=\"post-content-body\">\n<div>\n<div class=\"article-formatted-body article-formatted-body article-formatted-body_version-2\">\n<div xmlns=\"http:\/\/www.w3.org\/1999\/xhtml\">\n<h2>Introduction<\/h2>\n<p>In this article, I will cover the memory consumption of .NET applications on Linux. Firstly, we will try to understand the idea of virtual memory. Then, we will examine the memory statistics that Linux provides, such as RSS (Resident Set Size), VmData, RES (Resident Memory Size), and SWAP. Following that, we will delve into specifics related to the .NET.<\/p>\n<h2>Basic memory stats in Linux<\/h2>\n<p>Let\u2019s create a basic ASP.NET application from the default template, publish it in framework-dependent mode and run it on Linux.<\/p>\n<p>I\u2019ve started the application on Linux and executed the top command.<\/p>\n<figure class=\"full-width\"><\/figure>\n<p>We have three memory-related statistics here: 262Gb, 66Mb and 57Mb, but how do we find the \u201ctrue\u201d memory consumption? Unfortunately, it\u2019s impossible unless we understand the idea of virtual memory and the principles of memory allocation.<\/p>\n<h2>Virtual memory<\/h2>\n<p>In older operation systems and on devices like Arduino, you have direct access to physical memory. It means that operations like mov 0, 42 would actually load \u201c42\u201d to the very beginning of the CPU\u2019s physical memory. But modern OSs like Linux and Windows, with help from CPU don\u2019t give you access to physical memory. A mechanism called virtualisation is used instead.<\/p>\n<p>Let\u2019s say we have two processes which load some files from the disk and allocate some memory to store data.<\/p>\n<p>Each of the processes has so-called virtual memory. Only virtual memory can be accessed inside the process. When you try to put 42 to a memory address like 7f849c021000 it doesn\u2019t mean that this very address of the physical memory will be accessed. Moreover, it doesn\u2019t mean any physical storage is allocated before you access it.<\/p>\n<figure class=\"full-width\"><\/figure>\n<p>Let\u2019s look at the diagram showing two processes and go through all the memory segments these processes have in the virtual memory space.<\/p>\n<h4>First process<\/h4>\n<ul>\n<li>\n<p>Segment 1.1 &#8212; it\u2019s a so-called anonymous memory. Anonymous memory is a memory that doesn\u2019t correspond to any file on a disk. It\u2019s a type of memory you get when calling malloc in c or new in C#.<\/p>\n<\/li>\n<li>\n<p>Segment 1.2 It\u2019s a <a href=\"https:\/\/en.wikipedia.org\/wiki\/Memory-mapped_file\" rel=\"noopener noreferrer nofollow\"><u>memory-mapped file<\/u><\/a>. When you read this memory at the beginning of the segment, you\u2019ll get bytes from the beginning of the file. Memory-mapped files are a completely different approach than .NET File.Open or fopen in C. It is mainly used to load dynamic libraries (DLL in .NET and Windows and so in Linux). In this example, no physical memory is allocated. There can be two reasons for that. The first reason is this segment was never accessed, and the Linux kernel was too lazy to load it in advance. The second reason is that this file is rarely accessed, and the kernel removed it from physical memory. Nevertheless, the kernel will load the file or, most likely, only a small part of it into physical memory once the application tries to access memory from this segment.\u00a0<\/p>\n<\/li>\n<li>\n<p>Segment 1.3 is a memory-mapped file as well. It\u2019s actively used, so the kernel loads all the content into the memory. This file and the physical memory range are shared with the second process.<\/p>\n<\/li>\n<li>\n<p>Segment 1.4 is an anonymous memory-mapped file. This functionality allows for data exchange between processes. In .NET, the MemoryMappedFile class enables access to this feature.<\/p>\n<\/li>\n<li>\n<p>This segment is shared between processes but not mapped to any disk file.<\/p>\n<\/li>\n<\/ul>\n<h4>Second process<\/h4>\n<ul>\n<li>\n<p>Segment 2.1 is the same as segment 1.3 of the first file<\/p>\n<\/li>\n<li>\n<p>Segment 2.2 is the same as segment 1.4 of the first file<\/p>\n<\/li>\n<li>\n<p>Segment 2.3 is an ordinary private anonymous memory like segment 1.1 from the first process.<\/p>\n<\/li>\n<li>\n<p>Segment 2.4 is a memory-mapped file. The part of the file is frequently accessed, so the kernel allocated a physical memory for it, but some parts of this file are accessed very seldom or not accessed at all. That\u2019s why the kernel hasn\u2019t yet allocated physical memory for this file part.<\/p>\n<\/li>\n<li>\n<p>Segment 2.5 is an anonymous segment like 2.2, and the process writes to all the parts of the segments, so it used to be fully allocated. Then, this memory was not accessed for a long time, and there was a shortage of physical memory, so the kernel decided to swap this segment to the disk swap segment.<\/p>\n<\/li>\n<\/ul>\n<p>Let\u2019s try to generalise all these cases:<\/p>\n<ul>\n<li>\n<p>Anonymous vs. file-backed<\/p>\n<ul>\n<li>\n<p>We use an anonymous memory when we allocate some virtual memory via malloc or new.<\/p>\n<\/li>\n<li>\n<p>When we want to map some disk file to a memory, we use a memory-mapped file.<\/p>\n<\/li>\n<\/ul>\n<\/li>\n<li>\n<p>There is a difference between virtual memory allocation by malloc and physical memory allocation. <strong>Unaccessed memory, in most cases, is not physically allocated<\/strong>.<\/p>\n<\/li>\n<li>\n<p>The kernel tries not to hold unused data in physical memory. There can be two different cases:<\/p>\n<ul>\n<li>\n<p>For anonymous blocks, memory would be swapped, meaning that the kernel would write it to a swap segment<\/p>\n<\/li>\n<li>\n<p>For a not modified file-backed memory, the kernel can throw away data from physical memory and read it once again on demand if a process doesn\u2019t change it.<\/p>\n<\/li>\n<\/ul>\n<\/li>\n<\/ul>\n<h2>Linux memory stat<\/h2>\n<p>In Linux, there are several sources of memory-related information:<\/p>\n<ul>\n<li>\n<p>\/proc\/{proc id}\/smaps most accurate and detailed information. Divided by individual segments.<\/p>\n<\/li>\n<li>\n<p>\/proc\/{proc id}\/smaps_rollup the same as previous but summed up.<\/p>\n<\/li>\n<li>\n<p>\/proc\/{proc id}\/status much faster but less accurate<\/p>\n<\/li>\n<li>\n<p>top and ps same figures as above but less detailed<\/p>\n<\/li>\n<\/ul>\n<p>To get all the data at the same moment, I execute the following:<\/p>\n<pre><code>$ cat \/proc\/$(pgrep Memory)\/smaps > mem.txt;cat \/proc\/$(pgrep Memory)\/smaps_rollup;cat \/proc\/$(pgrep Memory)\/status;pmap $(pgrep Memory) -x > smap.txt;top  559468e18000-7ffc8b1ef000 ---p 00000000 00:00 0                      [rollup] Rss:           87812 kB Pss:           78760 kB Pss_Anon:      23416 kB Pss_File:      43144 kB Pss_Shmem:     12200 kB Shared_Clean:  16184 kB Shared_Dirty:      8 kB Private_Clean: 36052 kB Private_Dirty: 35568 kB Referenced:    84432 kB Anonymous:     23416 kB LazyFree:          0 kB AnonHugePages: 12288 kB ShmemPmdMapped:    0 kB FilePmdMapped:     0 kB Shared_Hugetlb:    0 kB Private_Hugetlb:   0 kB Swap:              0 kB SwapPss:           0 kB Locked:            0 kB    Name:   Memory Umask:  0022 State:  S (sleeping) Tgid:   10000 Ngid:   0 Pid:10000 PPid:   169 TracerPid:  0 Uid:1000100010001000 Gid:1000100010001000 FDSize: 256 Groups: 4 20 24 25 27 29 30 44 46 116 1000 1001 NStgid: 10000 NSpid:  10000 NSpgid: 10000 NSsid:  169 VmPeak: 275330800 kB VmSize: 275248656 kB VmLck:     0 kB VmPin:     0 kB VmHWM: 87360 kB VmRSS: 87360 kB RssAnon:       23096 kB RssFile:       52128 kB RssShmem:      12136 kB VmData:  1244932 kB VmStk:   132 kB VmExe:    92 kB VmLib: 42836 kB VmPTE:   700 kB VmSwap:    0 kB HugetlbPages:      0 kB CoreDumping:0 THP_enabled:1 Threads:    30 SigQ:   3\/61471 SigPnd: 0000000000000000 ShdPnd: 0000000000000000 SigBlk: 0000000000000000 SigIgn: 0000000000001000 SigCgt: 00000003080344fe CapInh: 0000000000000000 CapPrm: 0000000000000000 CapEff: 0000000000000000 CapBnd: 000001ffffffffff CapAmb: 0000000000000000 NoNewPrivs: 0 Seccomp:    0 Seccomp_filters:    0 Speculation_Store_Bypass:   thread vulnerable SpeculationIndirectBranch:  conditional enabled Cpus_allowed:   ffff Cpus_allowed_list:  0-15 Mems_allowed:   1 Mems_allowed_list:  0 voluntary_ctxt_switches:    70 nonvoluntary_ctxt_switches: 2 top - 20:15:05 up 15:44,  0 users,  load average: 0.00, 0.01, 0.01 Tasks:  25 total,   1 running,  24 sleeping,   0 stopped,   0 zombie %Cpu(s):  0.0 us,  0.0 sy,  0.0 ni,100.0 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st MiB Mem :  15374.3 total,459.8 free,   1654.7 used,  13259.8 buff\/cache MiB Swap:   4096.0 total,   4095.2 free,  0.8 used.  13380.6 avail Mem    PID USER  PR  NIVIRTRESSHR S  %CPU  %MEM TIME+ COMMAND   259 root  20   02340120  0 S   0.0   0.0   0:00.60 Relay(260)   260 rumata20   06208   5196   3408 S   0.0   0.0   0:00.64 bash   483 root  20   02340112  0 S   0.0   0.0   0:00.00 SessionLeader   484 root  20   02340120  0 S   0.0   0.0   0:00.19 Relay(485)   485 rumata20   06340   5312   3416 S   0.0   0.0   0:00.17 bash   882 root  20   02340112  0 S   0.0   0.0   0:00.00 SessionLeader   883 root  20   02340120  0 S   0.0   0.0   0:00.03 Relay(884)   884 rumata20   06340   5272   3416 S   0.0   0.0   0:00.10 bash   998 root  20   02340112  0 S   0.0   0.0   0:00.00 SessionLeader   999 root  20   02340120  0 S   0.0   0.0   0:00.02 Relay(1000)  1000 rumata20   06324   5404   3560 S   0.0   0.0   0:00.04 bash 10000 rumata20   0  262.5g  87360  64264 S   0.0   0.6   0:01.56 Memory 10289 rumata20   02772952864 S   0.0   0.0   0:00.00 memtest_sf<\/code><\/pre>\n<p>Let\u2019s take the most important values for each of these outputs and try to understand what do they mean and how to calculate them using smap.<\/p>\n<p>There are two primary data sources I use to understand the meaning of these outputs deeply:<\/p>\n<p><a href=\"https:\/\/www.kernel.org\/doc\/Documentation\/filesystems\/proc.txt\" rel=\"noopener noreferrer nofollow\"><u>https:\/\/www.kernel.org\/doc\/Documentation\/filesystems\/proc.txt<\/u><\/a><\/p>\n<p>https:\/\/man7.org\/linux\/man-pages\/man5\/proc.5.html<\/p>\n<h4>smaps_rollup<\/h4>\n<p><code>Rss: \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 87812 kB<\/code>\u00a0<\/p>\n<p>RSS is a \u201cthe amount of the mapping that is currently resident in RAM\u201d or, in other words, the physical memory consumption.<\/p>\n<pre><code>Pss: \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 78760 kB Pss_Anon:\u00a0 \u00a0 \u00a0 23416 kB Pss_File:\u00a0 \u00a0 \u00a0 43144 kB Pss_Shmem: \u00a0 \u00a0 12200 kB<\/code><\/pre>\n<p>\u201cproportional share of this mapping.\u201d I\u2019ll cover this in more detail a little bit later.<\/p>\n<pre><code>Shared_Clean:\u00a0 16184 kB Shared_Dirty:\u00a0 \u00a0 \u00a0 8 kB Private_Clean: 36052 kB Private_Dirty: 35568 kB<\/code><\/pre>\n<p>Each memory segment can be mapped by multiple processes. In this statistic, memory accessed by one process with sharable permissions will go to private. It will become shared only when the second process maps it.<\/p>\n<p>Clean memory is a memory we haven\u2019t modified. It can be either a memory-mapped file that we\u2019ve loaded from the disk and haven\u2019t modified or an anonymous memory we\u2019ve mapped but haven\u2019t written. Clean means that the kernel can swipe out this allocation from physical memory without writing it to the disk swap segment.<\/p>\n<p><code>Referenced:\u00a0 \u00a0 84432 kB<\/code><\/p>\n<p>\u201cthe amount of memory currently marked as referenced or accessed.\u201d<\/p>\n<p><code>Anonymous: \u00a0 \u00a0 23416 kB<\/code><\/p>\n<p>\u201cthe amount of memory that does not belong to any file\u201d. It\u2019s a memory allocated when we use malloc in c or new in C#.<\/p>\n<p>Anonymous memory can appear even in the file-backed segment when an application writes to this memory.<\/p>\n<h4>smap<\/h4>\n<p>smaps_rollup <\/p>\n<\/div>\n<\/div>\n<\/div>\n<\/div>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[],"tags":[],"class_list":["post-362495","post","type-post","status-publish","format-standard","hentry"],"_links":{"self":[{"href":"https:\/\/savepearlharbor.com\/index.php?rest_route=\/wp\/v2\/posts\/362495","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/savepearlharbor.com\/index.php?rest_route=\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/savepearlharbor.com\/index.php?rest_route=\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/savepearlharbor.com\/index.php?rest_route=\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/savepearlharbor.com\/index.php?rest_route=%2Fwp%2Fv2%2Fcomments&post=362495"}],"version-history":[{"count":0,"href":"https:\/\/savepearlharbor.com\/index.php?rest_route=\/wp\/v2\/posts\/362495\/revisions"}],"wp:attachment":[{"href":"https:\/\/savepearlharbor.com\/index.php?rest_route=%2Fwp%2Fv2%2Fmedia&parent=362495"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/savepearlharbor.com\/index.php?rest_route=%2Fwp%2Fv2%2Fcategories&post=362495"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/savepearlharbor.com\/index.php?rest_route=%2Fwp%2Fv2%2Ftags&post=362495"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}