Sunday, December 18, 2011

install PPTP VPN on AWS CentOS 6 machine

Configuring the server

yum install ppp -y
rpm -Uhv http://poptop.sourceforge.net/yum/stable/packages/pptpd-1.3.4-2.el6.i686.rpm
echo "localip 192.168.55.1" >> /etc/pptpd.conf
echo "remoteip 192.168.55.2-100" >> /etc/pptpd.conf 
echo "ms-dns 8.8.8.8" >> /etc/ppp/options.pptpd
echo "ms-dns 4.2.2.1" >> /etc/ppp/options.pptpd
echo "$USERNAME pptpd $PASSWORD *" >> /etc/ppp/chap-secrets
chkconfig iptables on
service iptables start
echo "net.ipv4.ip_forward = 1" >> /etc/sysctl.conf
sysctl -p
echo "iptables -t nat -A POSTROUTING -o eth0 -j MASQUERADE" >> /etc/rc.local
iptables -t nat -A POSTROUTING -o eth0 -j MASQUERADE
service iptables restart
service iptables save
chkconfig pptpd on

On the AWS console, go to "Security Groups", select either your security group or "default", select the "Inbound" tab. Select "Custom TCP rule" with "Port Range" 1723, Click "Add Rule", then click "Apply rule changes". At this point the PPTP server should be ready to accept connections. You can test if the server is properly started by running

telnet $SERVERIP 1723

where $SERVERIP is the public IP of the PPTP server machine.

Configuring a KDE client

Open "System Settings", click "Network Settings" and go to the VPN tab. Click the "Add" drop-button and select "PPTP".

Optionally check the "Connect automatically" and the "System connection" checkboxes.
Enter a connection name.
Enter $SERVERIP for "Gateway".
Enter $USERNAME for Login.
Enter $PASSWORD for Password.
Click the "Advanced" button in the bottom of the dialogue.
In the middle check the "Use MPPE Encryption" checkbox, and select "128 bit" in the Crypto drop down.
Click OK twice to close the settings dialogue.

Open a shell and test the connection:

ping 192.168.55.1

Configuring Network Manager on linux manually

Create a new connection file under /etc/NetworkManager/system-connections/ with the following content:

[connection]
id=CONNECTION NAME
type=vpn

[vpn]
service-type=org.freedesktop.NetworkManager.pptp
gateway=$SERVERIP
user=$USERNAME
require-mppe-128=yes

[vpn-secrets]
password=$PASSWORD

[ipv6]
method=ignore

[ipv4]
method=auto

Configuring a Windows XP client

Click Start > Sttings > Control Panel > Network Connections
Click File > New connection
Click Next
Select “Connect to the Network at my Workplace”, click Next
Select "Virtual Private Network connection", click Next
Enter a name for the connection, click Next
Enter the $SERVERIP, click Next
Click Finish

A new dialogue box will pop prompting for the username and password for the new connection.
Enter both, and check the "Save this username ..." checkbox.
Click "Connect".
After a successful connection you can test the connection by running the following in a cmd window:

ping 192.168.55.1

Guide with nice screen shots: http://doc.m0n0.ch/handbook/pptp-windows.html

Configuring an iPhone client

Go to Settings and open the "General" settings
Select "Network"
Select "VPN"
Choose "Add VPN Configuration"
Select the "PPTP" tab.
Enter a name for the VPN connection
Enter the address of the server, the login and the password.
Ensure that "Send All Traffic" is "ON&".
Click "Save".
Switch the "VPN" to "ON".

Guide with nice screen shots: http://www.dikant.de/2011/10/03/configuring-a-pptp-vpn-on-ios-and-android/

Configuring Apache for sites on the private network

In /etc/httpd/conf/httpd.conf, add

NameVirtualHost 192.168.55.1:80

Then add a new VirtualHost section for each site:

<VirtualHost 192.168.55.1:80>
 ...
</VirtualHost>

Reload apache configuration:

service httpd reload

Saturday, October 29, 2011

Moving from one amazon ec2 instance to another

I wanted to upgrade my web server in Amazon's ec2 from CentOS 5 to CentOS 6.
Instead of upgrading I decided to start a fresh new server.

I used the CentOS 6 AMI from http://support.rightscale.com/21-Community/RightScale_OSS
For some reason I did not notice that they also provide an EBS based AMI, so I went the hard way - started with an instance-store AMI and converted it to EBS stored one.
I largely followed this guide: http://www.capsunlock.net/2009/12/create-ebs-boot-ami.html
I used this script (which is a slightly modified version from https://gist.github.com/249915) to create the snapshot from the instance-store:

#!/bin/bash
# Run this script on the instance to be bundled

EBS_DEVICE=${1:-'/dev/xvda1'}
IMAGE_DIR=${2:-'/mnt/tmp'}
EBS_MOUNT_POINT=${3:-'/mnt/ebs'}

mkdir -p $EBS_MOUNT_POINT
mkfs.ext3 ${EBS_DEVICE}
mount ${EBS_DEVICE} $EBS_MOUNT_POINT

# make a local working copy
mkdir -p $IMAGE_DIR
rsync --stats -av --exclude='/root/.bash_history' --exclude='/home/*/.bash_history' --exclude='/etc/ssh/ssh_host_*' --exclude='/etc/ssh/moduli' --exclude='/etc/udev/rules.d/*persistent-net.rules' --exclude='/var/lib/ec2/*' --exclude='/mnt/*' --exclude='/proc/*' --exclude='/tmp/*' / $IMAGE_DIR

#clear out log files
cd $IMAGE_DIR/var/log
for i in `ls ./**/*`; do
    echo $i && echo -n> $i
done

cd $IMAGE_DIR
tar -cSf - -C ./ . | tar xvf - -C $EBS_MOUNT_POINT
#NOTE, You could rsync / directly to EBS_MOUNT_POINT, but this tar trickery saves some space in the snapshot

umount $EBS_MOUNT_POINT

Get the kernel id info about the running instance

ec2-describe-images $INSTANCE_STORE_AMI

ec2-register --snapshot $SNAPID --kernel $KERNEL --description "NEW OS" --name "new-os-i386-yyyymmdd" --architecture i386 --block-device-mapping /dev/sda2=ephemeral0 --root-device-name /dev/sda1

Once the snapshot is registered and the new AMI is created, launch it. Make sure to launch it in the same availability zone as the main EBS volumes that we will attach later
Get the connection settings from AWS console, login into the new instance and run

yum update
yum install yum-priorities httpd php mysql mysql-server php-cli php-gd php-mbstring php-pdo php-pecl-fileinfo php-mysql php-imap php-ldap php-pear php-xml php-xmlrpc mod_ssl ImageMagick elinks emacs fetchmail mod_python netpbm netpbm-progs nmap ntp strace webalizer rdiff-backup trac gcc gcc-c++ bzr 
rpm -Uvh http://s3.amazonaws.com/ec2-downloads/ec2-ami-tools.noarch.rpm

wget http://s3.amazonaws.com/ec2-downloads/ec2-api-tools.zip
cd /usr/local/ec2
unzip /root/ec2-api-tools.zip
ln -s ec2-api-tools-* apitools

pear install HTML_QuickForm
pear install Mail
pear install Net_SMTP

chkconfig rightscale off
chkconfig iptables off
chkconfig netfs off
chkconfig mcstrans off

chkconfig httpd on
chkconfig mysqld on
chkconfig ntpd on

# edit /etc/logrotate.d/httpd and add /var/www/logs/*log

export MYUSER=oldusername
export GID=oldgroupid
export UID=olduserid
groupadd -g $GID  $MYUSER
useradd -g apache -G $MYUSER -u $UID $MYUSER
passwd $MYUSER
passwd

#copy crontab for $MYUSER from old system

mkdir /vol

edit /etc/fstab and add

/dev/vg1/lv1          /vol              ext3    defaults,noatime        0 0
/vol/home             /home             none    bind                    0 0
/vol/etc/httpd        /etc/httpd        none    bind                    0 0
/vol/etc/pki          /etc/pki          none    bind                    0 0
/vol/var/www          /var/www          none    bind                    0 0
/vol/var/trac         /var/trac         none    bind                    0 0
/vol/var/lib/mysql    /var/lib/mysql    none    bind                    0 0

On the old instance stop the services and unmount the filesystems. Then disable LVM

/etc/init.d/httpd stop
/etc/init.d/mysqld stop

umount /home/jail/var/lib/mysql
umount /vol/var/lib/mysql
umount /vol/var/www
umount /vol/var/trac
umount /vol/etc/pki
umount /vol/etc/httpd
umount /vol/home
umount /vol

vgchange -a n

From AWS Console detach the EBS volumes from the old instance and attach them to the instance.
Then on the new instance run:

pvscan
pvdisplay
vgscan
vgdisplay
vgchange -a y
lvscan

mount /vol
mount /home
mount /etc/httpd
mount /etc/pki
mount /var/www
mount /var/trac
mkdir /var/trac
mount /var/trac
mount /var/lib/mysql

/etc/init.d/ntpd start
/etc/init.d/mysqld start
/etc/init.d/httpd start

Check where httpd.pid is located. I had to edit /etc/sysconfig/httpd and set

PIDFILE=/var/run/httpd.pid

Edit /etc/php.ini and set the timezone to avoid php 5.3 warnings:

date.timezone = America/New_York

Edit /etc/sysconfig/network and set the HOSTNAME (e.g. example.com). Also set on the shell, for example:

hostname example

From AWS Console, associate the reserved IP with the new instance

Sunday, September 25, 2011

Rebuilding a debian/ubuntu deb package

mkdir rebuild
mv $PACKAGE.deb rebuild
cd rebuild
dpkg -x $PACKAGE.deb common
dpkg --control $PACKAGE.deb
nano -w DEBIAN/control
remove the "Dependency: libc..."
cp -a DEBIAN/ common/
dpkg -b common $PACKAGE.deb
sudo dpkg --force-all -i $PACKAGE.deb
rm -rf common DEBIAN

Saturday, May 21, 2011

The not so obvious list of movies I really like from the past few years

The Incredibles
The Visitor
There Will Be Blood
In Bruges
Into the Wild
City of God
Ethernal sunshine of the spotless mind
Sideways

Other pretty ok movies

Gone baby gone
Lost in Translation
Before the devil knows you're dead
No Country For Old Men
District 9
Pan's Labyrinth

Sunday, January 31, 2010

Preparing a CentOS 5.4 AMI for LAMP server

I used the RightScale AMI with id ami-f8b35e91, which has a pretty minimalistic installation of CentOS 5.4. Here are the steps I did to make it ready to run my sites

install rpmforge
```
yum install yum-priorities
```
Edit the .repo files in /etc/yum.repos.d/ and set up priorities by adding the line:
priority=1 in the [base], [addons], [updates], [extras] sections, and the line
priority=2 in the [centosplus],[contrib]
```
rpm --import http://dag.wieers.com/rpm/packages/RPM-GPG-KEY.dag.txt
wget http://packages.sw.be/rpmforge-release/rpmforge-release-0.5.1-1.el5.rf.i386.rpm
rpm -K rpmforge-release-0.5.1-1.el5.rf.i386.rpm
rpm -ivh rpmforge-release-0.5.1-1.el5.rf.i386.rpm
```
Edit /etc/yum.repos.d/rpmforge.repo and add the line
```
priority=10
```
Edit /etc/yum.repos.d/rpmforge-testing.repo and add the line
```
priority=15
```
update the system
```
yum update
```

install apache, mysql, php and some other packages

yum install httpd php mysql mysql-server php-cli php-gd php-mbstring php-pdo php-pecl-fileinfo php-mysql php-imap php-ldap php-pear php-xml php-xmlrpc
 mod_ssl ImageMagick elinks emacs fetchmail mod_python netpbm netpbm-progs nmap ntp strace webalizer rdiff-backup trac

install amazon's AMI tools

rpm -Uvh http://s3.amazonaws.com/ec2-downloads/ec2-ami-tools.noarch.rpm

turn off some services

chkconfig rightscale off
chkconfig jexec off
chkconfig --level 06 jexec off
chkconfig ip6tables off
chkconfig iptables off
chkconfig netfs off
chkconfig nfslock off
chkconfig portmap off
chkconfig xfs off
chkconfig mcstrans off

install EC2 API tools

wget http://s3.amazonaws.com/ec2-downloads/ec2-api-tools.zip
cd /usr/local/ec2
unzip /root/ec2-api-tools.zip
ln -s ec2-api-tools-* apitools

get rid of some unneeded packages

yum remove cups gcc-gnat libgnat cups-libs libobjc gcc-objc++ kernel-2.6.18-164.11.1.el5 kernel-headers-2.6.18-164.11.1.el5

add back c, c++ compilers
```
yum install gcc gcc-c++
```

turn on services we need

chkconfig httpd on
chkconfig mysqld on
chkconfig ntpd on

change /etc/sysconfig/network-scripts/ifcfg-eth0 to contain

DEVICE=eth0
BOOTPROTO=dhcp
ONBOOT=yes
TYPE=Ethernet
USERCTL=yes
PEERDNS=yes
IPV6INIT=no

change /etc/sysconfig/network to contain

HOSTNAME=yourhost.yourdomain
NETWORKING=yes
NETWORKING_IPV6=no

My site is an old legacy php site, which needs register_globals and register_long_arrays. So edit /etc/php.ini to set
```
register_globals = On
register_long_arrays = On
memory_limit = 128M
```

edit /etc/logrotate.d/httpd and add /var/www/logs/*log

add user USER

groupadd -g GID USER
useradd -g GID -u UID USER

change passwords for root and USER
```
passwd USER
passwd
```

add crontab for USER

create the directory where EBS LVM will be mounted
```
mkdir /vol
```

edit /etc/fstab to add

/dev/vg1/lv1          /vol              ext3    defaults,noatime        0 0
/vol/home             /home             none    bind                    0 0
/vol/etc/httpd        /etc/httpd        none    bind                    0 0
/vol/etc/pki          /etc/pki          none    bind                    0 0
/vol/var/www          /var/www          none    bind                    0 0
/vol/var/trac         /var/trac         none    bind                    0 0
/vol/var/lib/mysql    /var/lib/mysql    none    bind                    0 0

edit /etc/ssh/sshd_config and set
```
PasswordAuthentication yes
```

install the QuickForm pear package
```
pear install HTML_QuickForm
```
get rid of the RightScale motd
```
echo -n > /etc/motd
```

use the re-bundle script from my previous post to create the AMI

Friday, October 16, 2009

associative arrays in bash

Bash 4 will have built-in associative arrays. But until then here is a nice trick I found (the least invasive that I found) to emulate a similar behavior.

function idx {
cnt=${#cases[*]};
eval 'case $1 in '${cases[*]}' *) [ "$1" ] && { cases['$cnt']='\''"'\''$1'\''") echo '$cnt';;'\''; echo '$cnt';}; esac';
}

Then you can do

arr[`idx john`]=doe
echo ${arr[`idx john`]}

Credit for this trick goes to Tafuni Vito

The idea is this: every time the function idx is called, it appends a strings to the cases array - composed of the passed variable, made ready for bash's case statements (in example above it would be "john"), then the word "echo" (without quotes), and finally the string "N;;" where N is the current size of the cases array. So if the call "idx john" was the first call, the the first element of the cases array would be

"john") echo 0;;

if we then call "idx jack", the cases array would look like this:

"john") echo 0;;
"jack") echo 1;;

This means that the expression "case $1 in ${cases[*]}" will look like this:

case $1 "john") echo 0;; "jack") echo 1;;

Which is exactly the appropriate beginning for a bash switch statement. The rest of the function handles the "default" case of the switch, by appending to the cases array, and returning its current size.

Thursday, July 23, 2009

From VMware to Amazon EC2

I have a VMware image running CentOS 5.3 (32 bit). Here are the steps it took to convert it to an amazon EC2 AMI. Note that since I did this from within the VM (not outside, on the vmk files), the same method would apply to non VM installations of CentOS.

Make sure ruby and java 1.6 are installed.
Download and install AMI tools from http://developer.amazonwebservices.com/connect/entry.jspa?externalID=368 (requires ruby).
```
wget http://s3.amazonaws.com/ec2-downloads/ec2-ami-tools.noarch.rpm
rpm -Uvh ec2-ami-tools.noarch.rpm
```

Donwload and install API tools from http://developer.amazonwebservices.com/connect/entry.jspa?externalID=351 (requires java).

wget http://s3.amazonaws.com/ec2-downloads/ec2-api-tools.zip
cd /usr/local/ec2 
unzip /root/ec2-api-tools.zip 
ln -s ec2-api-tools-* apitools

cleanup the VM disks. I had to cleanup some logs and backups to free up some space for the AMI image. Note that the AMI images are saved in "sparse" format, so they occupy less space than they appear (and show with ls -l)
Register for EC2 at http://aws.amazon.com/ec2/
Once registered it gives you the AWS Access Id and the AWS Access Secret key.
Create the X.509 key pair and download them into the VM image under /root/.ec2. Also save this locally on your desktop.
install the ElastickFox firefox extension from http://s3.amazonaws.com/ec2-downloads/elasticfox.xpi

Add the following into /root/.bash_profile

export JAVA_HOME=/usr/lib/jvm/java 
export EC2_HOME=/usr/local/ec2/apitools
export EC2_PRIVATE_KEY=$HOME/.ec2/pk-YOUR_X509_KEY.pem
export EC2_CERT=$HOME/.ec2/cert-YOUR_X509_CERT.pem 
export PATH=$PATH:$EC2_HOME/bin

change /etc/sysconfig/network-scripts/ifcfg-eth0 to contain

DEVICE=eth0
BOOTPROTO=dhcp
ONBOOT=yes
TYPE=Ethernet
USERCTL=yes
PEERDNS=yes
IPV6INIT=no

download and install some xen modules (Important: the module below is for 32 bit version, if you are converting a 64 bit OS, use ?)

wget http://alestic-downloads.s3.amazonaws.com/ec2-kernel-modules-2.6.16-xenU.tgz
cd /lib/modules
tar -xvzf ec2-kernel-modules-2.6.16-xenU.tgz

EC2 boots into runlevel 4. Disable unneeded services:
```
for i in cpuspeed pcscd messagebus restorecond mcstrans \
lvm2-monitor iptables ip6tables isdn \
netfs apmd acpid cups sendmail gpm anacron \
atd yum-updatesd avahi-daemon smartd httpd mysqld; do
chkconfig --level 4 $i off; 
done
```
Note that I disabled iptables as to avoid any possible problems when logging in for the first time. Eventually when the image is running properly I will enable iptables back. Similarly httpd and mysqld are disabled because I don't have the mounts for /var/www /var/lib/mysql properly setup yet. Again once this is done on the running instance, I will enable httpd and mysqld services.

Make sure to enable some important services:

for i in network syslog sshd; do 
chkconfig --level 4 $i on; 
done

Make sure to disable selinux. Either use the system-config-securitylevel-tui tool or simply edit the file in /etc/selinux/config and add the line
```
SELINUX=disabled
```
create the AMI
```
mkdir /image
ec2-bundle-vol -c $EC2_CERT -k $EC2_PRIVATE_KEY --user AWS_ACCOUNT_NUMBER -d /image -e /image,/home,/var/www,/var/lib/mysql -r i386 -s 4096 --no-inherit --generate-fstab
```
In my case I excluded /home, /var/www, /var/lib/mysql directories because I'm planning to mount these on a EBS. Note that the tool is smart enough to exclude some system directories on its own (e.g. /dev, /proc), so you don't need to add these manually in the exclusion list.

This command will take a pretty long time and will create an AMI of size 4GB. Your /image directory will have a file called image, many files called image.partXX and the manifest file called image.manifest.xml. You could examine what has been put into your AMI filesystem by mounting the image file: mount -o loop /image/image /mnt
upload your AMI to Amazon S3
```
ec2-upload-bundle -b YOUR_BUCKET_NAME -m /image/image.manifest.xml -a YOUR_AWS_ACCESS_ID -s YOUR_AWS_ACCESS_SECRET_KEY
```
YOUR_BUCKET_NAME can be any name (e.g. my-cool-ami). A bucket in S3 is similar to a directory on a filesystem. If it does not exist, it will be created. Otherwise, if it's yours, your image files will be uploaded into that bucket (and possibly overwrite existing files with the same names - so be careful). You can manage your S3 buckets and files using the S3Fox firefox extension from http://www.s3fox.net/release/latest/s3fox.xpi.
Register the AMI
```
ec2-register YOUR_BUCKET_NAME/image.manifest.xml
```
The output of this command will show your AMI ID. You can also see it by running
```
ec2-describe-images
```
Create a key pair for ssh
```
??????? $HOME/.ec2/YOUR_KEY_PAIR.pem
```
We're ready to start up our instance:
```
ec2-run-instances AMI_ID -k $HOME/.ec2/YOUR_KEY_PAIR.pem
```
To see the status, the instance id and the host name that was assigned to your instance run:
```
ec2-describe-instances
```

If everything went ok, you can try to ssh to the new instance:

ssh -i $HOME/.ec2/YOUR_KEY_PAIR.pem root@YOUR_INSTANCE_HOST_NAME

EBS Volumes

I decided to use LVM in spanning mode with EBS disks, so that I have an easy way to expand my storage size when needed. Initially I will create two EBS disks, each 20GiB, so total of 40GiB will be available in my logical volume.

First get the zone for your running instance. It's important to create the volumes on the same availability zone as the instance you will be attaching them to (otherwise the attach process will fail).
```
ec2-describe-instances
```
The zone for your instance will appear as the previous to last column in the output. Also make note of the 2nd column in the output - that's the instance id, you will need it when attaching the volumes to it.
In my case the zone is us-east-1c.

Now create the 2 volumes of 20GiB

ec2-create-volume -s 20 -z us-east-1c
ec2-create-volume -s 20 -z us-east-1c

Check the status and the volume ids

ec2-describe-volumes

If the status is "available", we can attach them:

ec2-attach-volume VOLUME_ID1 -i INSTANCE_ID -d /dev/sdf
ec2-attach-volume VOLUME_ID2 -i INSTANCE_ID -d /dev/sdg

At this point, you can either just create file systems on these volumes, mount them and start using them, or you can opt for LVM, which I will describe below.

LVM with EBS

First create the physical volumes
```
pvcreate /dev/sdf
pvcreate /dev/sdg
```
Create the volume group called vg1
```
vgcreate vg1 -s 256m /dev/sdf /dev/sdg
```
Note that we use a physical extent size of 256MB (instead of the default 4MB). This will allow us to have logical volumes up to ~64k*256MB = 16TB, which seams enough for now. This also puts a constraint that the logical volume can only be of size multiple of 256MB, which is ok for us.

Show the volumn group information

vgdisplay vg1

My output looks like this:

--- Volume group ---
VG Name               vg1
System ID
Format                lvm2
Metadata Areas        2
Metadata Sequence No  1
VG Access             read/write
VG Status             resizable
MAX LV                0
Cur LV                0
Open LV               0
Max PV                0
Cur PV                2
Act PV                2
VG Size               39.50 GB
PE Size               256.00 MB
Total PE              158
Alloc PE / Size       0 / 0
Free  PE / Size       158 / 39.50 GB
VG UUID               0SsrGe-X6u2-VB84-fh3z-GiuZ-ypC2-uJA8v9

The line marked in red tells us that we have 158 physical extents.

Create the logical volume called lv1, with all the physical extents available
```
lvcreate -l158 -n lv1 vg1
```

Show the logical volume information

lvdisplay /dev/vg1/lv1