Some cool articles on Threading vs. Multiprocessing:

https://dzone.com/articles/threads-done-right-with-tcl

https://timber.io/blog/multiprocessing-vs-multithreading-in-python-what-you-need-to-know/#the-lock

Converting VMWare VM into KVM on Centos 6.7

1. On ESXi, power off guest.

2. Copy files from datastore to CentOS system :
root@kvm1#  scp -v root@esxi:/vmfs/volumes/< datastore name >/ < VM Name >/*  .

Note that there are additional steps if your VMWare VM has snapshots, multiple drives etc. I didn't, so YMMV. Please check the links at the bottom of this post for more info.

3. Convert vmdk to qcow2:

root@kvm1# qemu-img convert ./myvm.vmdk -O qcow2 mykvm.img

4. Download vmware2libvirt script from:

http://bazaar.launchpad.net/~ubuntu-virt/virt-goodies/trunk/view/head:/vmware2libvirt

5. In my case, the vmx file had a value for memSize, while the script looked for memsize. Change accordingly and run it against the vmx file:

root@kvm1# vmware2libvirt -f myvm.vmx > mykvm.xml

6. Edit mykvm.xml if you want to change the domain name. I also made the following changes to mykvm.xml:

a. Edit the value for < emulator > from /usr/bin/kvm to /usr/libexec/qemu-kvm, otherwise the following step ("Import XML") will fail.

b. Edit the value for "< source file=" to the correct location of the img file you created earlier.
c. Add  < driver name='qemu' type='qcow2' cache='none' / > before the source file tag.
d. If you want VNC, edit the line with "vnc" with the appropriate IP address on your KVM hypervisor: eg: < graphics type='vnc' port='-1' listen='10.X.X.X' keymap='en-us'/ > where 10.X.X.X is the IP that your hypervisor listens to VNC on.
e. ESXi has networks, KVM has bridges. Please replace:
    < interface type='network' >
      < mac address='00:XX:YY:ZZ:AA:BB'/ >
      < source network='default'/ >
    < /interface >
With
    < interface type='bridge' >
      < mac address='00:XX:YY:ZZ:AA:BB'/ >
      < source bridge='br0'/ >
    < /interface >

That's if you want proper, pass-through, bridged networking. Its useful if you run services on your VM.

If you're OK with NAT where outside systems can't get to your KVM, but your KVM can get out, then you should be good with the default setup.

Blogger.com doesn't like text immediately after a "<" or ">" so when you update your XML file please adjust accordingly.

7. Import XML. This will create an XML file in /etc/libvirt/qemu/mykvm.xml
root@kvm1# virsh -c qemu:///system define mykvm.xml

8. Verify that the domain has been defined.
root@kvm1# virsh list --all
 Id    Name                           State
----------------------------------------------------
  -     mykvm                           shut off

9. Start it up.
root@kvm1# virsh start mykvm
root@kvm1# virsh list --all
 Id    Name                           State
----------------------------------------------------
  -     mykvm                           running

10. Networking can be tricky. This is a great resource.
https://jamielinux.com/docs/libvirt-networking-handbook/

Other great links:
http://www.held.org.il/blog/2012/04/how-i-converted-vmware-vm-to-kvm/
http://linsec.ca/2013/01/27/converting-a-vmware-fusion-virtual-machine-to-kvm/
https://raymii.org/s/tutorials/KVM_add_disk_image_or_swap_image_to_virtual_machine_with_virsh.html

KVM Install

Host: Centos
Guest Centos
Admin laptop: Windows.

Follow steps in:
http://www.cyberciti.biz/faq/kvm-virtualization-in-redhat-centos-scientific-linux-6/

Start the KVM Installation as described. You'll notice you can't connect to the VNC server.,

VNC connect to the installer using the following steps:
1. Edit /etc/libvirt/qemu.conf

Uncomment and edit the appropriate line to:
vnc_listen = "10.X.X.X"

Where 10.X.X.X is your private IP.
If it is a public IP please be sure to use TLS etc.

2. Add the following rule to /etc/sysconfig/iptables:
-A INPUT -m state --state NEW -m tcp -p tcp --dport 5900 -j ACCEPT

and restart iptables.

3. Start up vncviewer on your Windows laptop and point to 10.X.X.X::5900

You should see the Centos installation process running on the Centos guest.

Find recipes

• Traverse through a directory, skip certain folders:

find .  ! -path "./dir1/*" ! -path "./.dir2/*" ! -path "./dir3/dir4/*" -type f

• Find matches for certain files:

find . -iname "*.css" -o -iname "*.js" -type f
OR

find . -type f -a \( -iname "*.css" -o -iname "*.js" \)

expr1 expr2 is the same as expr1 -a expr2

Best practices for tuning MySQL to run on Flash.

Written for PCIe SSD, however the suggestions should work for all SSDs.

http://cariapa.net/mysql_wp/mysql_best_practices.pdf

http://cariapa.net/mysql_wp/my.cnf

Stuff to remember in my.cnf:
max_connections = 30000
I was using the my.cnf file for a benchmark so needed to up max_connections to a massive number. Chances are you won't need do.

innodb_buffer_pool_size = 5G
Deliberately made this a tiny number because we wanted to avoid memory and go to SSD. You might want to up it a little.

open-files-limit = 8192
My benchmark opened and closed a lot of tables, which were created as separate files (innodb_file_per_table) Which is why I needed to up this number and tweak ulimits in limit.conf.

innodb_log_block_size=4096
4K was the block size of the SSD I was using. Match the log block size to this and you'll get great performance. Unfortunately it seems to be unique only to Percona.

Building a distcache RPM

Required for building the httpd RPM if you need the mod_socache_dc RPM.
Download distcache-1.5.1.tar.bz2 from http://sourceforge.net/projects/distcache/files
Save in /root/rpmbuild/SOURCES, where /root/rpmbuild is the value returned by:
# rpm --eval '%{_topdir}'

Save the text after the dotted line in /root/rpmbuild/SPECS/distcache.spec

# cd /root/rpmbuild/SPECS

# rpmbuild -ba distcache.spec

This will create /root/rpmbuild/RPMS/x86_64/distcache-devel-1.5.1-1.x86_64.rpm
# rpm -ivh /root/rpmbuild/RPMS/x86_64/distcache-devel-1.5.1-1.x86_64.rpm

================== distcache.spec =============
Name:           distcache-devel
Version:        1.5.1
Release:        1
Summary:        Distributed session caching

Group:          Development/Libraries
License:        GNU Library or Lesser General Public License version 2.0 (LGPLv2)
URL:            http://sourceforge.net/projects/distcache/
Source0:        distcache-%{version}.tar.bz2
BuildRoot:      %{_builddir}/distcache-%{version}-root

%description
Distributed session caching tools and APIs, primarily for SSL/TLS servers though perhaps useful for other (non-SSL/TLS) circumstances. Also includes a self-contained network abstraction library (libnal), and the sslswamp SSL/TLS benchmark/test utility.

%prep
%setup -q -n distcache-%{version}
# Update proto_fd.c to avoid the LONG_MIN undeclared error. Just one line, so we don't need a patch file.
# rpmbuild untars source and cd's in the directory, so we use a relative path for proto_fd.c
sed -i.bak '29 i\#include <limits.h>' libnal/proto_fd.c
%build
./configure CFLAGS="-fPIC"
make
%install
rm -rf $RPM_BUILD_ROOT
make DESTDIR=$RPM_BUILD_ROOT install
%clean
rm -rf $RPM_BUILD_ROOT
%files
%defattr(755,root,root,-)
/usr/local/bin/dc_client
/usr/local/bin/dc_server
/usr/local/bin/dc_snoop
/usr/local/bin/dc_test
/usr/local/bin/nal_echo
/usr/local/bin/nal_hose
/usr/local/bin/nal_ping
/usr/local/bin/nal_pong
/usr/local/bin/nal_proxy
/usr/local/bin/nal_test
/usr/local/bin/piper
/usr/local/bin/sslswamp
/usr/local/lib/libdistcache.la
/usr/local/lib/libdistcacheserver.la
/usr/local/lib/libnal.la
/usr/local/lib/libnalssl.la
%defattr(644,root,root,-)
/usr/local/include/distcache/dc_client.h
/usr/local/include/distcache/dc_plug.h
/usr/local/include/distcache/dc_server.h
/usr/local/include/libnal/nal_devel.h
/usr/local/include/libnal/nal.h
/usr/local/include/libnal/nal_ssl.h
/usr/local/lib/libdistcache.a
/usr/local/lib/libdistcacheserver.a
/usr/local/lib/libnal.a
/usr/local/lib/libnalssl.a
/usr/local/man/man1/dc_client.1
/usr/local/man/man1/dc_server.1
/usr/local/man/man1/dc_snoop.1
/usr/local/man/man1/dc_test.1
/usr/local/man/man1/sslswamp.1
/usr/local/man/man2/DC_CTX_new.2
/usr/local/man/man2/DC_PLUG_new.2
/usr/local/man/man2/DC_PLUG_read.2
/usr/local/man/man2/DC_SERVER_new.2
/usr/local/man/man2/NAL_ADDRESS_new.2
/usr/local/man/man2/NAL_BUFFER_new.2
/usr/local/man/man2/NAL_CONNECTION_new.2
/usr/local/man/man2/NAL_decode_uint32.2
/usr/local/man/man2/NAL_LISTENER_new.2
/usr/local/man/man2/NAL_SELECTOR_new.2
/usr/local/man/man8/distcache.8
/usr/local/share/swamp/A-client.pem
/usr/local/share/swamp/CA.pem
%doc

%changelog

Building apache httpd RPMs from source on Centos 6

Probably works on RHEL as well. I used a fresh install of Centos 6.6.

# mkdir -p ~/rpmbuild/{SOURCES,SPECS,BUILD,RPMS,SRPMS}
# cd ~rpmbuild/SOURCES

Wget the latest source for httpd and apr, apr-util from your closest mirror:
# wget http://apache.mirrors.pair.com/httpd/httpd-2.4.16.tar.bz2

# wget http://apache.arvixe.com/apr/apr-1.5.2.tar.bz2

# wget http://apache.arvixe.com/apr/apr-util-1.5.4.tar.bz2

Build and install apr:

# rpmbuild -tb apr-1.5.2.tar.bz2

# rpm -ivh ~/rpmbuild/RPMS/x86_64/apr-1.5.2-1.x86_64.rpm \
~/rpmbuild/RPMS/x86_64/apr-devel-1.5.2-1.x86_64.rpm

Build apr-util:

 # rpmbuild -tb apr-util-1.5.4.tar.bz2

error: Failed build dependencies:

        expat-devel is needed by apr-util-1.5.4-1.x86_64

        libuuid-devel is needed by apr-util-1.5.4-1.x86_64

        freetds-devel is needed by apr-util-1.5.4-1.x86_64

        unixODBC-devel is needed by apr-util-1.5.4-1.x86_64

Alrighty then:

# yum -y install expat-devel libuuid-devel freetds-devel unixODBC-devel

freetds is from the epel repo. Be sure to enable that, since it isn't installed by default.

# rpmbuild -tb apr-util-1.5.4.tar.bz2 => OK!

# rpm -ivh ~/rpmbuild/RPMS/x86_64/apr-util-1.5.4-1.x86_64.rpm \
~/rpmbuild/RPMS/x86_64/apr-util-devel-1.5.4-1.x86_64.rpm

Httpd requires distcache for mod_socache_dc. Distcache doesn't have an RPM. If there is no need for  mod_socache_dc, you can try commenting out the following lines in httpd.spec:
%package -n mod_socache_dc
Group: System Environment/Daemons
Summary: Distcache shared object cache module for the Apache HTTP server
BuildRequires: distcache-devel
Requires: httpd = %{version}-%{release}, httpd-mmn = %{mmn}

Otherwise:
1. Use this distcache.spec file and follow instructions. If you go this this route you don't need to use "--nodeps" in the rpmbuild process in the final step.
2. Download distcache-1.5.1.tar.bz2 and build the old way from http://sourceforge.net/projects/distcache/files.

# tar xjf distcache-1.5.1.tar.bz2

# cd distcache-1.5.1

-fPIC fixed a weird httpd build issue for me. May not be needed otherwise.

# ./configure CFLAGS="-fPIC"

# make

Make fails with 'error "LONG_MIN" undeclared' . Fix it by adding:

# include <limits.h>

on line 29 of /distcache-1.5.1/libnal/proto_fd.c

# make

# make install

Now that distcache has been installed (or skipped), we can build httpd:

# rpmbuild -tb httpd-2.4.16.tar.bz2

error: Failed build dependencies:
     pcre-devel is needed by httpd-2.4.16-1.x86_64
     lua-devel is needed by httpd-2.4.16-1.x86_64

     distcache-devel is needed by httpd-2.4.16-1.x86_64
Alrighty then:
# yum -y install pcre-devel lua-devel

If we installed distcache the old school way (compiling without RPMs!) use --nodeps to ignore distcache-devel. If you created an RPM for distcache-devel you wouldn't need --nodeps.

# rpmbuild -tb --nodeps httpd-2.4.16.tar.bz2

# rpm -ivh ~/rpmbuild/RPMS/x86_64/httpd-2.4.16-1.x86_64.rpm \
~/rpmbuild/RPMS/x86_64/httpd-tools-2.4.16-1.x86_64.rpm \
~/rpmbuild/RPMS/x86_64/httpd-devel-2.4.16-1.x86_64.rpm \
~/rpmbuild/RPMS/x86_64/mod_ssl-2.4.16-1.x86_64.rpm

That's it!

Thanks to:
http://www.erikwebb.net/blog/compile-and-install-apache-24-red-hat-enterprise-linux-rhel-6-or-centos-6/

For having me look in the right direction.