{"id":979,"date":"2014-01-16T13:03:26","date_gmt":"2014-01-16T20:03:26","guid":{"rendered":"https:\/\/www.jumpcloud.com\/?p=979"},"modified":"2020-04-13T15:52:29","modified_gmt":"2020-04-13T21:52:29","slug":"my-linux-server-is-very-slow","status":"publish","type":"post","link":"https:\/\/jumpcloud.com\/blog\/my-linux-server-is-very-slow","title":{"rendered":"My Linux server is very slow…"},"content":{"rendered":"\n

Linux has a plethora of command line tools to use when troubleshooting performance or application issues. These utilities include top, df, ps, free, iostat, vmstat, lnstat, and others. Each tool is valuable and has its place in your troubleshooting and monitoring toolbox.<\/span><\/p>\n\n\n\n

Out of all of these, there\u2019s no tool combination I\u2019ve found more valuable for troubleshooting performance issues than vmstat and top. Many Linux users use top frequently, even if you don\u2019t understand everything it\u2019s telling you. I\u2019m going to be touching only lightly on top, but you may be surprised how much more you understand about top after reading this article as well.<\/span><\/p>\n\n\n\n

\"\"<\/figure><\/h3>\n\n\n\n

Virtual Memory: The History<\/h3>\n\n\n\n

Vmstat\u2019s man page describes it as: \u201cReport virtual memory statistics\u201d, which is great, if you know what that means. To understand this fully, and to understand what vmstat is telling you, we need to dive back into the 1960\u2019s and 70\u2019s when virtual memory concepts were really being developed.<\/span><\/p>\n\n\n\n

Back in those days, memory was at a premium. It was extremely costly, so machines rarely had anywhere near enough primary memory storage. By this, I mean RAM, something most of us today measure in gigabytes, but which back then was measured in kilobytes. Because you couldn\u2019t always do what you wanted in that small amount of memory, operating systems (such as UNIX) had the ability to swap out all or parts of programs to secondary storage (disks), when they didn\u2019t need to be in primary (RAM) memory.<\/span><\/p>\n\n\n\n

Linux today still provides this capability, even though memory is literally millions of times cheaper (in 1994, my company once bought 128MB of HP server memory at $1000\/MB, yes, mega). Today, memory costs about $10\/GB. Because memory\u2019s become so cheap, swapping data to disk is rarely necessary on most physical servers. However, given the move to smaller virtualized instances where memory is the limiting factor in the scalability of your virtual environment, memory is once again at a premium.<\/span><\/p>\n\n\n\n

Swapping and Secondary Memory<\/h3>\n\n\n\n

Today, we call it \u201cswap space\u201d or your \u201cswap partition\u201d or \u201cswap file\u201d, but no matter the term, it refers to either a raw disk partition or a file on your filesystem that the Linux kernel can use to shuffle rarely used data out to, if memory gets tight. Swap files are also called \u201cvirtual memory\u201d because the operating system (in concert with a CPU that supports it, which nearly all of them do, today) makes it look to your application as if there is more memory than there really is.<\/span><\/p>\n\n\n\n

All that\u2019s well and good: we seemingly get more memory for free. But, as with most things, there\u2019s a catch: obviously disk is not nearly as fast as your main memory. Solid state disks (SSDs) have helped with this quite a bit, but even the fastest SSD is still many times slower than main memory. And, because processors are so fast these days, the difference between running at full speed out of main memory and having to deal with swapping data to disk is more obvious than ever. So, you get even cheaper \u201cvirtual\u201d memory, but at a huge cost in performance.<\/span><\/p>\n\n\n\n

Swapping versus Paging: What\u2019s the difference?<\/h3>\n\n\n\n

You may have seen the terms, swapping, and paging, and not known what the difference is. Swapping has historically referred to swapping entire processes to a swap file and taking them out of primary memory to make space available for other processes. Paging means taking only the least recently used blocks of memory held by a process, and writing those parts out to secondary storage.<\/span><\/p>\n\n\n\n

Linux and most, if not all, modern OSes, actually do paging, and never swap out an entire process, or at least don\u2019t look at swapping in that way. They may swap out all the data that makes up a process, but they won\u2019t look at a process and say, let\u2019s take that whole process out. It\u2019s inefficient, and generally not a great idea: it\u2019s very expensive for the disk I\/O required, and is overkill for most situations.<\/span><\/p>\n\n\n\n

Swapping has become a term that is generally synonymous with paging because the distinction has become moot in nearly all cases today. From now on, I\u2019ll use the term \u201cswapping\u201d, because that\u2019s what most everyone uses. Paging is actually more accurate, but less used.<\/span><\/p>\n\n\n\n

The Performance Impacts of Swapping<\/h3>\n\n\n\n

Generally, you want your applications to never use swap space, except when data is rarely needed, so you don\u2019t see an impact on your performance. But, you may have experienced your Linux server\/instance slowing down to a crawl, and had trouble figuring out why. Maybe you just reboot it at that point, but sometimes that results in rebooting several times a day, which isn\u2019t good for uptime. The answer almost always lies in this one question: is my Linux server swapping (paging) on a consistent basis?<\/span><\/p>\n\n\n\n

Remember, it generally doesn\u2019t hurt to swap if the data being swapped out is rarely being used. So, it\u2019s safe to use swap space to get your application up and running and your server stabilized. If your server doesn\u2019t stabilize, however (memory keeps growing, or too much is allocated and released on a frequent basis), your application will slow to a horrendous crawl.<\/span><\/p>\n\n\n\n

How can you determine whether swapping is going on? That\u2019s where vmstat comes in: it shows you what\u2019s happening with virtual memory on a system-wide basis. Where vmstat gives you the system-wide view, top helps you zero-in on a particular process or processes.<\/span><\/p>\n\n\n\n

Vmstat\u2019s output is pretty cryptic:<\/span><\/p>\n\n\n\n

[devbox opt]$ vmstat 1\nprocs ———–memory———- —swap– —–io—- –system– —–cpu—–\nr  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy id wa st\n0  0   4036  94372 159648 339188    0    0     2     6   43   33  1  0 99  0  0\n0  0   4036  94364 159648 339188    0    0     0     0   32   39  0  0 100  0  0\n0  0   4036  94364 159648 339188    0    0     0     0   41   69  0  1 99  0  0<\/pre>\n\n\n\n
 <\/b><\/b>(Note: the \u201c1\u201d after the vmstat tells is to wait one second before showing another sample, and to keep repeating, which is handy to write out to a log file, so you can see how things change over time.)<\/p>\n\n\n\n
But, let\u2019s break it down with this simple table:
\n<\/b><\/b><\/p>\n\n\n\n
\n