How to reduce system resources

Often one might need to reduce system resources while running benchmarks; i.e. run a benchmark on a system with X amount of RAM and then run it again on 2X the amount of RAM. This post shows you how.
OS: Centos

Kernel:2.6.18-8

• Reducing system memory by creating a ramdisk on tmpfs:

Drop cache. Note that the "echo" instruction is a one-time deal. It immediately clears the cache, after which the kernel starts caching stuff again.: https://github.com/torvalds/linux/blob/master/fs/drop_caches.c

# sync; echo 3 > /proc/sys/vm/drop_caches

How much memory do we have?
# cat /proc/meminfo | head -3

MemTotal:      3895864 kB
MemFree:       3818628 kB
Buffers:          1800 kB

Create mount point for a 2G ramdisk:
# mkdir -p /mnt/ramdisk
# mount -t tmpfs -o size=2g none /mnt/ramdisk/

Still have almost same amount of free memory:

MemTotal:      3895864 kB
MemFree:       3818024 kB
Buffers:          1944 kB

# df -h

Filesystem            Size  Used Avail Use% Mounted on
/dev/mapper/VolGroup00-LogVol00
                      287G   55G  218G  21% /
/dev/sda1              99M   11M   83M  12% /boot
tmpfs                 1.9G     0  1.9G   0% /dev/shm
none                  2.0G     0  2.0G   0% /mnt/ramdisk

It is rumored that older versions of the Linux kernel will let you write to RAMdisk forever. If true, this could be a very bad thing, so please use this command carefully.
# dd if=/dev/zero of=/mnt/ramdisk/dummy bs=1024 count=2095100
2095100+0 records in
2095100+0 records out
2145382400 bytes (2.1 GB) copied, 10.9631 seconds, 196 MB/s

Lets check memory usage again:
# df -h
Filesystem            Size  Used Avail Use% Mounted on
/dev/mapper/VolGroup00-LogVol00
                      287G   55G  218G  21% /
/dev/sda1              99M   11M   83M  12% /boot
tmpfs                 1.9G     0  1.9G   0% /dev/shm
none                  2.0G  2.0G     0 100% /mnt/ramdisk
# cat /proc/meminfo | head -3
MemTotal:      3895864 kB
MemFree:       1719076 kB
Buffers:          2008 kB

Do the math to verify:
# bc -l
bc 1.06
Copyright 1991-1994, 1997, 1998, 2000 Free Software Foundation, Inc.
This is free software with ABSOLUTELY NO WARRANTY.
For details type `warranty'.

3818024/1024/1024
3.64115142822265625000
1719076/1024/1024
1.63943862915039062500

Remove dummy file:

# rm /mnt/ramdisk/dummy
rm: remove regular file `/mnt/ramdisk/dummy'? y

Did we free up memory?
# df -h
Filesystem            Size  Used Avail Use% Mounted on
/dev/mapper/VolGroup00-LogVol00
                      287G   55G  218G  21% /
/dev/sda1              99M   11M   83M  12% /boot
tmpfs                 1.9G     0  1.9G   0% /dev/shm
none                  2.0G     0  2.0G   0% /mnt/ramdisk
# cat /proc/meminfo | head -3
MemTotal:      3895864 kB
MemFree:       3817468 kB
Buffers:          2056 kB

• Reducing the # of cores available to your system:

How many cores do you have?

# cat /proc/cpuinfo |grep processor

processor       : 0

processor       : 1

Stick them into a bash variable:

# CORES=`cat /proc/cpuinfo |grep processor|awk '{print $NF}'`

You can't disable core 0. Otherwise this would be a very risky script to run :-)

# for i in $CORES; do echo "CORE $i"; ls -l /sys/devices/system/cpu/cpu${i}; echo "0" > /sys/devices/system/cpu/cpu${i}/online; done

CORE 0

total 0

-r-------- 1 root root 4096 Mar 23 14:13 crash_notes

drwxr-xr-x 2 root root    0 Mar 23 13:46 topology/

-bash: /sys/devices/system/cpu/cpu0/online: Permission denied

CORE 1

total 0

-r-------- 1 root root 4096 Mar 23 14:14 crash_notes

-rw------- 1 root root    0 Mar 23 14:17 online

drwxr-xr-x 2 root root    0 Mar 23 14:17 topology/

Zapped Core 1

# cat /proc/cpuinfo |grep processor

processor       : 0

Lets bring it back online:

# for i in $CORES; do echo "CORE $i"; ls -l /sys/devices/system/cpu/cpu${i}; echo "1" > /sys/devices/system/cpu/cpu${i}/online; done

CORE 0

total 0

-r-------- 1 root root 4096 Mar 23 14:13 crash_notes

drwxr-xr-x 2 root root    0 Mar 23 13:46 topology/

-bash: /sys/devices/system/cpu/cpu0/online: Permission denied

CORE 1

total 0

-r-------- 1 root root 4096 Mar 23 14:14 crash_notes

-rw------- 1 root root    0 Mar 23 14:18 online

Did it work?

# cat /proc/cpuinfo |grep processor

processor       : 0

processor       : 1

Getting sysbench to work with Postgres on Flash SSD

System information:

OS: RHEL 5.8.
Sysbench: 0.4.12
Postgres: 9.2.3
RAM: 64GB
CPU: 16 cores E5-2650 0 @ 2.00GHz
SSD: STEC s1120 2TB PCIe
STEC s840 2TB SAS
Running EXT3 on the SSDs

Installing Sysbench.

1. Download sysbench 0.4.12 from http://sourceforge.net/projects/sysbench/
tar zxvf

 ./configure --without-mysql --with-pgsql
2. Edit ./sysbench/drivers/pgsql/drv_pgsql.c

as per

http://jkshah.blogspot.com/2010/10/postgres-9-and-sysbench-0412.html

otherwise sysbench prepare will take a very long time to populate the table.

3. make
Make barfs at libtool with some weird error that looks like its from X11.
../libtool: line 838: X--tag=CC: command not found
../libtool: line 871: libtool: ignoring unknown tag : command not found
../libtool: line 838: X--mode=link: command not found
../libtool: line 1004: *** Warning: inferring the mode of operation is deprecated.: command not found
../libtool: line 1005: *** Future versions of Libtool will require --mode=MODE be specified.: command not found

This isn't an X11 error. Its because libtool sets ECHO=echo and then uses $echo. Edit libtool so all occurrences of $echo are changed to $ECHO, and make will work.

4. make install 

Installs sysbench into /usr/local/bin by default. Make sure you have that in your path.

Installing/initializing Postgres

1. Follow instructions for install PG RPMs here:

http://yum.postgresql.org/

and here

http://wiki.postgresql.org/wiki/YUM_Installation

Make sure the right drive is mounted properly on /var/lib/pgsql before running any initialization commands.

2. Start up PG:

/usr/pgsql-9.2/bin/pg_ctl -D /var/lib/pgsql/9.2/data start

Configuration/tuning

1. Mount ext3 FS on the SSDs with the following options.
data=writeback,noatime,nodiratime
Ext4 has better options for SSDs, but is recommended only for RHEL 6.x distributions with the newer kernel.
2. Postgres docs recommend raising kernel.shmmax and kernel.shmall, however with 64G of memory these values appear to be pretty large anyway.
3. Changes to postgresql.conf:

shared_buffers = 32GB        # Half of available memory
work_mem = 12GB              # Increase the amount of memory used for in-memory sorts/user.
maintenance_work_mem = 1GB    # Used in vacuum    
effective_io_concurrency = 16      # Increase # of I/O threads. Could probably bump this higher for SSD.
checkpoint_segments = 256   # Checkpoint every 4GB instead of every 48M which is the default with value =3      
effective_cache_size = 32GB      # Half of available memory.
default_statistics_target = 1000   # Used by query planner
update_process_title = off           # Don't update ps -e with SQL statements.
autovacuum_max_workers = 16      # Increase to # of cores available to speed up optimization.

I didn't test this, but apparently changing random_page_cost to 1 can help as well.
http://www.databasesoup.com/2012/05/random-page-cost-revisited.html

Running sysbench

1. First create the PG database:

su - postgres

createdb dbtest

# Confirm its been created

psql -c "\l"

2. Run sysbench prepare which populates the table in PG (sbtest by default)

sysbench --test=oltp --pgsql-user=postgres_user --pgsql-password=postgres_user_password --pgsql-host="" --pgsql-db=dbtest --db-driver=pgsql --oltp-table-size=100000000 --num-threads=16 prepare

We're creating a 100M row table here which consumes about 30G of space. This can take about 10 minutes or so.

3. Run the actual test:

THREADS="8 16 24 32 48 60 72 84 100"

for t in $THREADS

do

  # First clear cache

  vmstat -SM

  sync; echo 3 > /proc/sys/vm/drop_caches

  vmstat -SM

  sysbench --test=oltp   --pgsql-user=postgres_user --pgsql-password=postgres_user_password --pgsql-host="" --pgsql-db=dbtest --db-driver=pgsql --oltp-dist-type=gaussian --oltp-table-size=100000000  --oltp-test-mode=complex --num-threads=$t  --max-time=600 --max-requests=25000000 run

done

If you run sysbench with oltp-test-mode=complex, it seems to behave differently with MySQL/InnoDB vs. Postgres as described here:

http://jkshah.blogspot.com/2010/11/sysbench-postgresql-90-and-oltp-complex.html

If your table size is small (i.e. 1M rows) or if you use more threads, you will run into deadlocks.

I worked around this by increasing the number of rows in sbtest to 100M , and restricting max-requests to 25M. Need to work with the sysbench folks to fix this.

How fast is your file transfer proceeding?

D=`ls -l $dest_file|awk '{print $5}'`
S=`ls -l $src_file | awk '{print $5}'`
echo "$D/$S * 100" | bc -l

Prints out a %age value of the copied file.