why unix | RBL service | netrs | please | ripcalc | linescroll
hosted services

hosted services

Recently I've had to do a lot of in depth analysis of memory usage on a particular host on the network to try and diagnose various problems.

meminfo

Some of the most interesting things come as a result of graphing the various details in /proc/meminfo.

MemTotal

Total usable RAM in kilobytes (i.e. physical memory minus a few reserved bytes and the kernel binary code)

MemFree

The amount of physical RAM left unused by the system.

Buffers

The amount of physical RAM used for file buffers.

Cached

The amount of physical RAM used as cache memory. This is memory in the pagecache (diskcache) minus SwapCache.

SwapCache

This is the amount of Swap used as cache memory. So, memory that once was swapped out, is swapped back in, but is still in the swapfile.

Active

The total amount of buffer or page cache memory, that is active. This part of the memory is used recently and usually not reclaimed unless absolutely necessary.

Inactive

The total amount of buffer or page cache memory that are free and available. This is memory that has not been recently used and can be reclaimed for other purposes by the paging algorithm.

HighTotal

is the total amount of memory in the high region. The HighTotal value can vary based on the type of kernel used. Kernel uses indirect tricks to access the high memory region. Data cache can go in this memory region.

LowTotal

The total amount of non-highmem memory.

LowFree

The amount of free memory of the low memory region. This is the memory the kernel can address directly. All kernel datastructures need to go into low memory

SwapTotal

Total amount of physical swap memory.

SwapFree

Total amount of swap memory free.

Dirty

The total amount of memory waiting to be written back to the disk.

Writeback

The total amount of memory actively being written back to the disk.

Mapped

The total amount of memory that has been mmap()'d, such as libraries.

Bounce buffer

This is used to copy pages to and from when a device cannot access the full memory that is available, such as using >32bits of memory address (4GiG+ on a 32bit machine, such as Intel PAE enabled).

Committed_AS

An estimate of how much RAM you would need to make a 99.99% guarantee that there never is OOM (out of memory) for this workload. Normally the kernel will overcommit memory. This parameter represents the worst case scenario value, and also includes swap memory.

So, if you want to view what your optimal system memory is, just grep this value:

awk -F: '{ if( $1 == "Committed_AS" ) { split( $2, a, " " ); printf "%dMB\n", a[1]/1024 } }' /proc/meminfo

invalidating the cache

One of the problems I've had recently is that the disk cache on the box that I've been testing has as much cache as it does disk, so it's not very easy to ensure that you're reading or writing data from the disk platters.

One way to ensure that you're working on the platters (or at least increase the probability that you are) is to tinker with /proc/sys/vm/drop_caches:

  • To free pagecache:

    echo 1 > /proc/sys/vm/drop_caches

  • To free dentries and inodes:

    echo 2 > /proc/sys/vm/drop_caches

  • To free pagecache, dentries and inodes:

    echo 3 > /proc/sys/vm/drop_caches

So sending 3 to the drop_caches file is the most reliable, but once you've finished your test and want to use memory caches again, send a 0 to the file. You should run sync in order make sure items are flushed.

process level memory

Each process stores it's memory counters in /proc/$$/status (where $$ is the PID of the process you're interested in).

Within the status file you'll find lines beginning with VmRSS and VmSize, while the total memory required by the process is stored the VmSize value the memory in use right now is stored in the VmRSS value.

If you're interested in processing these values at all you can do something along these lines.

# ( echo -n '(' ; grep '^VmRSS:' /proc/*/status \
| awk '{ print $2 }' \
| tr '\n' '+' ; echo -e '0)*1024\n' ) | bc -l

This can produce some interesting figures, especially if you're thinking about monitoring the memory of something like Apache that's running on your system. If this is what you're after then you can switch /proc/*/status to 

for i in $( pgrep apache ) ; do grep '^VmRSS:' /proc/$i/status ; done |

So,

# ( echo -n '(' ; for i in $( pgrep apache ) ; do \
grep '^VmRSS:' /proc/$i/status ; done \
| awk '{ print $2 }' \
| tr '\n' '+' ; echo -e '0)*1024\n' ) | bc -l