Salvaging my IT skills....

OpenVZ changes and randomizes cron times

2012-02-17T02:27:00.000-08:00

We use the OpenVZ virtulization platform for our servers, we have servers that run up to 32 virtual machines based on the configuration of the hardware node.

Our setup requires that each of these virtual machines running on top of the hardware node be the same in every respect, down to the timing of the last cron. We have cooked up a custom template of a virtual machine, loaded with all that we want and use puppet to quickly deploy and destroy virtual machines. During operations we realized that our crons were not running at the same time, rather we saw the results of the cron scripts appear at odd times, when we did not want them to run. On checking the various cron files on the VM's we discovered that the cron times for all the cron jobs had changed.

container 111
# start atop daily at midnight
32 1 * * * root /etc/atop/atop.daily
container 112
# start atop daily at midnight
6 3 * * * root /etc/atop/atop.daily
container 113
# start atop daily at midnight
40 1 * * * root /etc/atop/atop.daily
container 114
# start atop daily at midnight
18 1 * * * root /etc/atop/atop.daily
container 115
# start atop daily at midnight
9 2 * * * root /etc/atop/atop.daily
container 116
# start atop daily at midnight
25 1 * * * root /etc/atop/atop.daily
container 117
# start atop daily at midnight
43 2 * * * root /etc/atop/atop.daily
container 118
# start atop daily at midnight
29 1 * * * root /etc/atop/atop.daily

We were surprised because we were creating the VM's from the same custom template, hence they should have had the same cron job times, poking around further we realized that the vzctl runs a script /etc/vz/dists/scripts/postcreate.sh that is run inside each VM when it is created. Apart from the other thing that it does, it also randomized the cron job times in the following files /etc/crontab and all files under /etc/cron.d/

Though this might be a wanted feature on a shared hosting platform, it was creating problems for us, as we are not using OpenVZ for shared hosting, We raised a bug report for this, and the OpenVZ dev team has been kind enough to look into this, starting vzctl-3.0.31, you can just put the following token VZCTL_POSTCREATE_SKIP as a comment somewhere in the cron file so that postcreate.sh does not randomize the cron jobs in that file.

If you want to stop randomizing your crons before the new vzctl release or are sure that you will not be ever using the crontab randomizer, like we are, it bodes well to just comment out the call to the function randcrontab in the postcreate.sh script, and never worry about it ever again.

Squid 3.1 forwarded-for not working to hide client IP

2012-01-23T00:05:00.000-08:00

Before allowing an action some websites test the IP address of the connection as a policy, some go as far to check the X-Forwarded-For header, this can create problems for users, as most of the times the IP's that admin's will use behind a proxy server will be private IP's and as such the website may disallow access to a client on getting this private IP through the X-forwarded-For header, qatarliving.com is one such website that checks the X-Forwarded-For header , I had blogged about this specific case here

To stop squid from sending the IP address of the client, the option that needs to be used in the squid.conf file is forwarded_for, previously I was using squid-2.6 stable, and the parameter forwarded_for off sufficed and removed the X-forwarded-for header, but in squid-3.1 I learn from the release notes that the arguments to the parameter forwarded-for have been changed

If set to "transparent",
Squid will not alter the X-Forwarded-For header in any way.

If set to "delete",
Squid will delete the entire X-Forwarded-For header.

If set to "truncate",
Squid will remove all existing X-Forwarded-For entries, and place the client IP as the sole entry.

Thus if, you want to hide the client IP in squid-3.1 you will have to set the forwarded-for parameter as:

forwarded-for delete

This should now delete the complete X-forwarded-For header from the proxy request sent by squid

Creating a simple yum repository

2012-01-18T02:14:00.000-08:00

This is a simple howto about creating your own YUM Repo. We will use apache to expose this repository to yum clients

Steps to be performed on the server
# Install the createrepo rpm

yum install createrepo

# create the directory that apache will use to serve the rpm's and other contents, the x86_64 directory will hold 64 bit rpm's and i386 will hold 32 bit rpm's

mkdir -p /var/www/html/repo/x86_64/base
mkdir -p /var/www/html/repo/i386/base

# copy a rpm to these directories

cp trafficserver-3.0.0-2.x86_64.rpm /var/www/html/repo/x86_64/base
cp trafficserver-3.0.0-2.i386.rpm /var/www/html/repo/i386/base

# now we will run createrepo so that it will create the metadata files required by the yum client

cd /var/www/html/repo/x86_64/base
createrepo -v .

cd /var/www/html/repo/i386/base
createrepo -v .

# expose the repository using apache, a simple vhost configuration would be something like below

vi /etc/httpd/conf.d/repo.conf
<VirtualHost repo.example.com>
    ServerAdmin webmaster@repo.example.com
    DocumentRoot /var/www/html/repo
    ServerName repo.example.com
    ErrorLog logs/repo-error_log
    CustomLog logs/repo-access_log common
</VirtualHost>

Now, we move on to the clinet, assume that we are working on a 64 bit machine, and require to install the trafficserver-3.0.0-2.x86_64.rpm rpm. We need to create a repo file under /etc/yum.repos.d

# create a repo file under /etc/yum.repos.d

vi /etc/yum.repos.d/myrepo.repo
[myrepo]
name = My Very Own Repo
baseurl = http://repo.example.com/x86_64/base
enabled = 1
gpgcheck = 0

# next, install the trafficserver rpm

yum install trafficserver

Puppetizing OpenVZ container creation

2012-01-09T07:13:00.000-08:00

I am currently using puppet-virt module by Carla Souza to puppetize vm creation and management, the module is quite extensive and easy to understand and implement.

Here, I am going to handle the creation of a simple centos VM . For people out there who do not know how to use a third party puppet module, just like me before using this one, read on. The module is hosted here https://github.com/carlasouza/puppet-virt/

Move into your modules directory and then do a git pull

git pull http://github.com/carlasouza/puppet-virt.git

Move into the /etc/puppet/modules/puppet-virt/manifests directory and backup the init.pp, we are not going to use this for this tutorial, we will write our own.

Now let us create a file init.pp and define a simple VM inside it with resource parameters.

class puppet-virt {
    virt { 'testserver.com':
                ipaddr => ['192.168.3.80'],
                ensure     => 'installed',
                id       => 615,
                os_template => 'centos-5',
                virt_type => 'openvz',
                autoboot   => 'false',
                ve_root => '/vz/root/$VEID',
                ve_private => '/vz/private/$VEID',
                configfile => 'vzsplit',
                resources_parameters => ["VMGUARPAGES=3145728","OOMGUARPAGES=3145728","PRIVVMPAGES=3145728"],
                nameserver => ['4.2.2.2','8.8.8.8'],
                diskspace => '26214400:26214400',
        }

}

Add, include puppet-virt in your sites.pp under the appropriate node and run the puppet agent, if you are using CentOS you will get the following error

"Sorry, this provider is not supported for your Operation System, yet"

To resolve this, just add CentOS to /etc/puppet/modules/puppet-virt/lib/puppet/provider/virt/openvz.rb file on the line that looks like this(currently it is line no 10)

if [ "Ubuntu", "Debian"].any? { |os| Facter.value(:operatingsystem) == os }

 make the above line look like

if [ "Ubuntu", "Debian", "CentOS"].any? { |os| Facter.value(:operatingsystem) == os }

Now, run the puppet agent again and your virtual machine should be up and running.

OpenVZ vzctl Disk Quota exceeded error

2012-01-09T06:45:00.000-08:00

I was creating some VM's on a new hardware node using the vzctl --create command, half way through the command I started getting Disk Quota exceeded error

Creating container private area (centos-5-x86_64)
tar: ./usr/lib64/mono/1.0/mcs.exe: Cannot write: Disk quota exceeded
tar: Skipping to next header
tar: Archive contains obsolescent base-64 headers
tar: ./usr/lib64/mono/1.0/setreg.exe.mdb: Cannot write: Disk quota exceeded
tar: Skipping to next header
tar: ./usr/lib64/mono/1.0/mozroots.exe.mdb: Cannot write: Disk quota exceeded

There were also error messages in /var/log/messages

Jan 7 09:48:51 server kernel: VZ QUOTA: disk softlimit exceeded for id=111
Jan 7 09:48:54 server kernel: VZ QUOTA: disk hardlimit reached for id=111

Inspite of lot of googling, there was not much that I could use to troubleshoot, I checked to see if there are disk quota's enabled on the disk, but found none.

After a little looking around and poking I found that I had used vzsplit to create a template configuration file. I passed -n 32 to vzsplit on a disk worth 150G. As a result vzsplit split 150G into 32 parts and set the DISKSPACE parameter very low. Thus during VM creation as the image was being untared, the low disk limit was being reached and vzctl started to complain about disk limit reached.
I was using the 150G disk for the OS, and the plan was to use other secondary disks for the VM root and private area.

This unveiled another thing about vzsplit, that while splitting resources it only considers the disk that has the OS installed on it, and ignores all other secondary disks attached to the server.

To continue creating the VM's I manually edited the DISKSPACE parameter set in the file generated by vzsplit to some high value and the --create command completed successfully.

Squid proxy 403 forbidden error

2012-01-05T04:59:00.000-08:00

Recently we had set up squid proxy for our office migrating away from microsoft ISA, everything went ahead fine until someone complained that they had started getting forbidden errors on some websites.

The actual error message that they encountered on accessing those sites was

Forbidden

You do not have permission to access / on this server.

Additionally, A 500 Internal Server error was encountered while trying to use an ErrorDocument to handle the request.

On further testing we found that if we bypassed squid then the site would open up normally, but if we went through squid the site would just return the above error.

Some friends on IRC suggested that I run tcpdump on the two connections to check the difference between the http headers of web requests sent through squid and the ones sent bypassing squid. Tcpdump revealed requests that were sent throughsquid(2.6) in my case was sending out GET HTTP/1.0 requests, whereas the ones that were being sent bypassing squid, were using GET HTTP/1.1

To further validate that this itself was the issue, I installed squid-2.7 that has experimental HTTP/1.1 support and turned the following directive on in squid.conf, this causes squid to send HTTP/1.1 requests.

server_http11 on

Turning the directive on and sending the requests through squid resulted in normal loading of the the webpage, then I turned the same directive off so that squid would send out HTTP/1.0 headers and the site again stopped working, Amos on the squid mailing list suggests using squid-3.1.

During the investigation process, I also tried out Apache Traffic Server, and was amazed with it, my perception is that ATS is faster and cleaner, so in case you want to give squid a miss, you can try ATS too, at least I am sticking with Apache Traffic Server as of now and playing with it.

Squid proxy and qatarliving.com issues

2012-01-03T09:48:00.000-08:00

Note: Refer here if you are using squid-3.1 or above

We recently moved to squid proxy for our office today from ISA(yes linux will finally take over Qatar too inshaAllah), considering that our office is based out of Qatar, most people use the website qatarliving.com.

qatarliving.com implements a policy of allowing only people from Qatar to post classifieds, in a bid to accompalish this, the code at qatarliving checks the source IP, after putting in squid, users in my office started getting errors from qatarliving where they are unable to post classifieds as qatarliving did not recognize them as being from within Qatar.

I tried whatismyip.com and got a proper Qatar IP, but still qatarliving was not recognizing my connections that emanated from behind squid as being from Qatar, frantic googleling revealed that squid also adds a forwarded-for header with the IP of the users machine(private in my case), thus because of this header qatarliving was considering my internal ips as the source ip and ignoring the actual ip that made the connection.

Fixing, this is fairly simple, just turn the forwarded-for option in squid.conf to off.

forwarded-for off

So now squid will not send this header any more, and qatarliving will now consider the connecting source IP.

WebService::Solr and solr group query support

2011-12-16T10:02:00.000-08:00

WebService::Solr is an amazing perl module for interacting with Apache Solr, by default after you fire a query using WebService::Solr, it returns an array with the response from the solr server.

Unfortunately, it seems that the using the response->docs method, fails on grouped queries with the following error

Attribute (docs) does not pass the type constraint because: Validation failed for 'ArrayRef' failed with value undef at perl-group.pl line 30

But before the above error, WebService::Solr does receive the response from the Solr server and it is just a matter of accessing step by step and making it available, below is the code snippet that can be used to get the complete group result

Step-1: Query the solr server

# search & get hits
my $response = $solr->search( $sent_query, {'rows' => '1000000', 'group' => 'true', 'group.field' => 'id', 'group.limit' => '50', 'fl' => 'text'});

Step-2: Save the hashref of deserialized JSON data from the response by WebService::Solr in to a variable

my $hits = $response->content;
my %hits = %$hits;

Step-3: Get the groups that contain the results

my $response_header = $hits{grouped}{id}{groups};
my @response_header = @$response_header;

Step-4: Each group contains arrays that contain element with response from the solr server

foreach my $group (@response_header) {
        my $textarray = $group->{doclist}{docs};
        my @textarray = @$textarray;

        foreach(@textarray) {
                print $_->{text}[0],"\n";
        }
}

Here is sample code to use in case the above explanation is confusing.

Fetch numFound using Webservice::Solr

2011-12-11T21:24:00.000-08:00

WebService::Solr is a perl module that you can use to interface with Apache Solr, inserting, retrieving/seraching documents and other indexing functions.

When you make a query using WebService::Solr it returns(the document states) among other stuff

# an array with the actual search results
# an hash ref to that contains the entire response from solr

Using WebService::Solr you can set the number of rows that you want to receive, but sometimes you may just want to know the number of records found with a particular search, you can do this by counting the number of elements in the array that WebService::Solr returns as part of the response packet, but if you receive like a million records, it defeats the purpose of speed.

Thus another way is to set rows => 0, so that solr does not spend time in sending the actual search results but only sends you the response header with the numFound field, it sends this header with search results also.

But, here is where the easy part ends, as fetching the numFound field resulted in dumping hashref after hashref to finally reach the numFound record, you can save time by using the code snippet below.

Basically you have to shun receiving the array after firing the query and will have to store the hashref content

# we are not interested in the array from the response
my $response = $solr->search( $sent_query, {'rows' => '0'});
my @hits = $response->docs;

# we are not interested in the array from the response
my $response = $solr->search( $sent_query, {'rows' => '0'});
my $hashref = $response->content;

Once we have the hashref, then its only a matter of dereferencing it and reaching the numFound key, or we can dereference in the same step as below

$numfound = $response->content->{response}->{numFound};

Complete code snippet for this operation can be found here

Getting started with Apache Solr

2011-12-09T04:41:00.000-08:00

Apache solr as the site describes "Solr is the popular, blazing fast open source enterprise search platform from the Apache Lucene project. Its major features include powerful full-text search, hit highlighting, faceted search, dynamic clustering, database integration, rich document (e.g., Word, PDF) handling, and geospatial search. Solr is highly scalable, providing distributed search and index replication, and it powers the search and navigation features of many of the world's largest internet sites."

cool, but beyond this most of the tutorials on the web expect some pre knowldge about search, indexing, handling xml based configuration files to the extent that after a few paragraphs I started to loose the author.

The way to download and install solr are well documented in the solr tutorial, and getting it up and running is pretty simple, the challenge begins after firing up solr.

assume the following scenario, I want to index some pdf's and make solr return the filename of the pdf once the user performs a search. The catch here is that first you will have to define a field in the example/solr/conf/schema.xml file to hold the filename, in the schema file look for fields tag and define a filename field

<field name="filename" type="string" indexed="true" stored="true"/>

where, name = name of the field
type=datatype
indexed=should data in this field be indexed
stored=if this field data should be returned in search

now, time to index a pdf file, we will use curl to do this, you have to also send in a field literal.filename to hold the name of the file that you are indexing literal.id is the unique id that you want this document to have

curl "http://localhost:8983/solr/update/extract?literal.id=doc1&literal.filename=test.pdf" -F "myfile=@test.pdf"

That's it, your pdf is indexed and now you can go to the solr search interface and fire up search queries that will return the name of the pdf.

But sooner or later you will realize that if you search with a search query like 'test.pdf' ie the name of the pdf file, inspite of you asking solr to index it in the schema.xml file, your search turns up blank.

This is because the default search field in the schema.xml file is set to "text", the default field which is used for search if you do not specify the field to search in. Thus if you want to search the filename field you will have to make you query look like

filename:test.pdf

If you do now want to specify the field name in the search, then another method is to ask solr to copy the contents of the filename filed into the text filed using the below syntax in the schema.xml file

<copyField source="filename" dest="text"/>

Now, you should have solr return your filename in the search, with or without mentioning the fieldname

Recusring through FTP Directory Structure and getting File Listing

2011-12-05T09:54:00.000-08:00

We have a remote FTP site that hosts a lot of files for a project that we are currently conducting, yesterday one user requested if there was a way for them to enter some string and search if a filename existed with that string, so that they can then go ahead and download it.

The idea here is not to download the files hosted on the FTP site, just get the file names, there were lot of helpful perl modules and sample scripts around, finally after reading a lot of excellent stuff on the net I was able to cook up this perl script with a recursion function, that crawls the FTP site and collects the file names.

The script uses Net::FTP and Net::FTP::File to accompalish this task and a recurse function crawl_tree(), here is the script on github.

Another way to accomplish the same thing is is via wget but with a little twist, using the --spider, --no-delete-listing options

wget --ftp-user='user' --ftp-password='pass' ftp://ftpsite.com --spider -r --no-remove-listing -l 0

This creates the entire FTP dir structure, but because of the --spider option no files are downloaded and because of the --no-remove-listing, a listing. file is created in each directory that you can then check to get the names of the files that are supposed to be present in that directory.

Beyond this, things were quiet simple, wrote a simple perl script that traversed through the directory structure created by wget on the local machine, reading listing. files in each directory getting a list of files that are available at the FTP site, created an index and gave my users a search interface.

Migrating OpenVZ Vm's

2011-11-28T08:31:00.000-08:00

Recently one of our OpenVZ servers was getting blocked on disk, thus we decided to add disk and move out some of the VM's to this new disk.

Below is the process that we followed to avoid check pointing etc.

create the new directory structure

# mkdir /disk/vz/private
# mkdir /disk/vz/root
# chmod 700 /disk/vz/private
# chmod 700 /disk/vz/root

stop the VM

# vzctl stop 111

move the VM data to the new disk

# mv /vz/private/111 /disk/vz/private/111

delete the VM directory in the old root section

# rmdir /vz/root/111

delete the old quota file for this vm

# rm -f /var/openvz/quota.111

The above step is important because, the old quota file contains quota information for the old paths that do not exist anymore, thus vzquota on fails with the following error

[root@server ~]# vzctl start 111
Starting container ...
vzquota : (error) Quota on syscall for id 165: No such file or directory
vzquota on failed [3]

start the VM

[root@server~]# vzctl start 111
Starting container ...
Initializing quota ...
Container is mounted
Adding IP address(es): x.x.x.x
Setting CPU units: xxxx
Set hostname: xx.xx.xx
File resolv.conf was modified
Container start in progress...

Authenticating web application with google and yahoo

2011-11-09T10:23:00.000-08:00

Ok, just some hours ago I was grappling with how to authenticate a user with google, specifically with a gmail id and with a yahoo email id. Most of the documentation that I found on the web assumed too much background knowledge, this is my attempt to present the topic for the absolute beginner like... ME.

Basically, this is the idea, rather than you authenticating the user, you ask google and yahoo to authenticate users for you and then take it from there, the idea is not to pass username and password to yahoo or google, rather it is to let yahoo and google handle everything, right from accepting username and password to authenticating the user. All you do is tell google or yahoo that, here is this user who wants to sign in my website, do you know him ? and if you do, can you please return some of his information, like his username, emailaddress, contact details, if the user wants to share it with the world.

We will use PHP to write all the code, first off download lightopenid, in this package you will find example-google.php and the actual library openid.php.

here is a hacked version of the example-google.php script that requests for the emailaddress along with user authentication request and then sets a encrypted cookie on the user browser.

The code is self explanatory, the important parts are the following elements

$openid->identity = 'https://www.google.com/accounts/o8/id';

This url is the actual url where you will redirect the client to authenticate with google, the technical term for this url is 'openid discovery url'

So now it is fairly simple, if you want to authenticate with yahoo, you need to google for 'yahoo openid discovery url' which will return http://me.yahoo.com , this is all the change that you will be required to do in your script in case you want to authenticate with yahoo.

So now once the user is authenticated google or yahoo will return to you a sea of information, you can find it in the address bar of the browser, most of which you may not be interested in, so I looked at the entire string(cryptic) and tried to look for the email address of the user, and.... it was not there, some more looking around and I found that you will have to explicitly ask the provider(google,yahoo) to send you the email address that was used for authentication, with the following line in your code.

$openid->required = array('contact/email');

To access this information, we can use the following lines of code

$array = $openid->getAttributes();

echo array['contact/email']);

OpenVZ and setting time and timezone

2011-11-09T06:43:00.000-08:00

Changing the time on a linux system can be broken into two parts, one is to change the time and the other is to change the timezone.

Recently we needed to change the timezone of all our OpenVZ containers to IST, this exersice involved changing the time and the timezone on the hardware node, and changing the timezone on the virtual containers, this is because VM containers take the time from the hardware node itself, so on the virtual container you need to change just the timezone and not worry about changing the time.

Changing time and timezone on the hardware node

Remove the current timezone file

#rm -f /etc/localtime

Link the new timezone file

#ln -s /usr/share/zoneinfo/Asia/Calcutta /etc/localtime

Sync the date using ntpdate

#ntpdate 0.us.pool.ntp.org

Now, on the Virtual machines all we need to do is change the timezone

#rm -f /etc/localtime

#ln -s /usr/share/zoneinfo/Asia/Calcutta /etc/localtime

In case the changes do not reflect in the container you may have to reboot it.

# vzctl restart 111

Done. 

Perl script to get client IP

2011-11-08T04:26:00.000-08:00

Sometimes you need to know the IP address of the client machine that is connecting to your website. There is a perl module CGI which can be used to accomplish this, there are two subroutines that can be used with slight different results

CGI->remote_addr - Returns the IP address of the client connecting.

CGI->remote_addr - Returns the A record of the client connecting, in case there is no A record the ip
address of the client will be returned.

Perl code to implement can be found here

You will need to put this script and enable your web server to run perl scripts. In case you use nginx here is how to do it.

github and perl to manage openvz containers

2011-11-06T10:09:00.000-08:00

This post here just inspired me to get a github account, it took me hardly minutes to set this up, indeed with every passing day, I am realizing the advantages of git and the fact that github can help me leverage them for professional advancement as well as helping the larger community makes it all the more sweet.

I had just written some perl code for managing openvz containers and was prepared to share it with the larger opensource community, The code is too primitive in my view to bundle it as a perl module, I just wanted to share it as is, but was looking for a easy way to do this, github fits the bill, so here is the link to my brand new github account

Another thing is that I can always have versions of my code accessible from anywhere, no carrying around my code in USB sticks and flash drives when changing jobs, laptops, desktops...

The pearl code a package can be found under the openvz directory, it basically consists of perl subs that do small tasks like starting, stopping, rebooting, listing openvz containers running on a hardware node.

Executing perl php and cgi scripts with nginx

2011-11-05T09:13:00.000-07:00

I had to deal today with using nginx as a frontend for nagios, this involved execution of cgi scripts and php scripts.

The point is that nginx does not have any inbuilt capability to execute external scripts, thus if you want to execute any external script you will have to run a wrapper for that script and then tell nginx to proxy the execute requests to that wrapper.

The another thing that I noticed is that it seems you cannot use the same wrapper for php and perl, cgi. So I had to install two wrapper, one for php and another one for perl and cgi.

To start this wrappers at boot time we will use spawn-cgi, note spawn-cgi is no wrapper it is just there to start, stop, restart the wrapper script.

Lets begin with installing the spawn-cgi rpm

yum install spawn-cgi

Make sure that you have php-cgi command installed and working on your machine, which is actually a wrapper for php scripts.
Now we need to install another wrapper for perl and cgi scripts , follow this excellent howto to install the fcgiwrap, skip the spawn-cgi part there, read that here below. fastwrap is the wrapper for perl and cgi scripts

After we have installed these two wrappers, one for php and another one for perl and cgi scripts, we need init scripts to start these wrapper and manage them.

we will create two init scripts spawn-fcgi and spawn-fcgi-php to manage the two wappers

1 - fastwrap

Edit the /etc/sysconfig/spawn-fcgi file

FCGI_SOCKET=/var/run/fcgiwrap.socket
FCGI_PROGRAM=/usr/local/sbin/fcgiwrap
FCGI_USER=nginx
FCGI_GROUP=nginx
FCGI_EXTRA_OPTIONS="-M 0700"
OPTIONS="-u $FCGI_USER -g $FCGI_GROUP -s $FCGI_SOCKET -S $FCGI_EXTRA_OPTIONS -F 1 -P /var/run/spawn-fcgi.pid -- $FCGI_PROGRAM"

I am using a unix socket instead of a inet address, you can use the latter instead with the -p option, now start the wrapper

[11:55]:root@mis:vhosts:#:>/etc/init.d/spawn-fcgi start
Starting spawn-fcgi:                                       [ OK ]
[11:56]:root@xxx:vhosts:#:>ps aux | grep cgi
nginx    29732 0.0 0.0   8208   472 ?        Ss   11:56   0:00 /usr/local/sbin/fcgiwrap
root     29745 0.0 0.0 103152   816 pts/2    S+   11:56   0:00 grep cgi
[11:56]:root@xxx:vhosts:#:>

2 - php-cgi

Copy over the file /etc/sysconfig/spawn-cgi file and create another one named /etc/sysconfig/spawn-cgi-php

The contents of this spawn-cgi-php file should look like

SOCKET=/var/run/php-fcgi.sock
OPTIONS="-u nginx -g nginx -s $SOCKET -S -M 0600 -C 32 -F 1 -P /var/run/spawn-fcgi-php.pid -- /usr/bin/php-cgi"

next, copy over /etc/init.d/spawn-fcgi file to /etc/init.d/spawn-fcgi-php, in this file look for the variable prog and config and change it to

prog="spawn-fcgi-php"
config="/etc/sysconfig/spawn-fcgi-php"

chmod the file 755 and add spawn-fcgi-php to chkconfig, so that the wrapper comes up on startup

[12:01]:root@xxx:vhosts:#:>chmod 755 /etc/init.d/spawn-fcgi-php
[12:01]:root@xxx:vhosts:#:>chkconfig --add spawn-fcgi-php
[12:01]:root@xxx:vhosts:#:>chkconfig spawn-fcgi-php on
[12:01]:root@xxx:vhosts:#:>chkconfig spawn-fcgi on

Start the wrapper

[12:03]:root@mis:vhosts:#:>/etc/init.d/spawn-fcgi-php start
Starting spawn-fcgi-php:                                   [ OK ]
[12:03]:root@mis:vhosts:#:>ps aux | grep cgi
nginx    29732 0.0 0.0   8208   472 ?        Ss   11:55   0:00 /usr/local/sbin/fcgiwrap
nginx    31110 0.2 0.2 209192 5888 ?        Ss   12:03   0:00 /usr/bin/php-cgi
nginx    31111 0.0 0.1 209192 2540 ?        S    12:03   0:00 /usr/bin/php-cgi
nginx    31112 0.0 0.1 209192 2540 ?        S    12:03   0:00 /usr/bin/php-cgi
nginx    31113 0.0 0.1 209192 2540 ?        S    12:03   0:00 /usr/bin/php-cgi
nginx    31114 0.0 0.1 209192 2540 ?        S    12:03   0:00 /usr/bin/php-cgi
nginx    31115 0.0 0.1 209192 2540 ?        S    12:03   0:00 /usr/bin/php-cgi
nginx    31116 0.0 0.1 209192 2540 ?        S    12:03   0:00 /usr/bin/php-cgi

Now onto testing stuff, here is a simple nginx vhost file

server {
listen 80;
server_name yourserver.com;

   access_log /var/log/nginx/yourserver.access.log;
   error_log   /var/log/nginx/yourserver.error.log;

   root    /var/www/html/manage;

   location ~ \.php$ {
      include /etc/nginx/fastcgi.conf;
      fastcgi_pass unix:/var/run/php-fcgi.sock;
}

   location ~ \.pl$ {
      include /etc/nginx/fastcgi.conf;
      fastcgi_pass unix:/var/run/fcgiwrap.socket;
   }

   location ~ \.cgi$ {
      include /etc/nginx/fastcgi.conf;
     fastcgi_pass unix:/var/run/fcgiwrap.socket;
   }

}

Now, you can call your scripts like http://yourserver.com/test.pl or http://yourserver.com/test.php

postfix-2.8 RPM creation from SRPM errors

2011-11-04T13:52:00.000-07:00

I downloaded postfix-2.8.6-1.fc15.src.rpm SRPM from the fedora repo onto my centos 5.6 box, I encountered a number of errors in trying to recompile the SRPM as documented below.

First off, I was unable to install the SRPM due to the following error

[root@relay32 SRPMS]# rpm -ivh postfix-2.8.6-1.fc15.src.rpm
warning: postfix-2.8.6-1.fc15.src.rpm: Header V3 RSA/SHA256 signature: NOKEY, key ID 069c8460
1:postfix warning: user mockbuild does not exist - using root
warning: group mockbuild does not exist - using root
########################################### [100%]
error: unpacking of archive failed on file /usr/src/redhat/SOURCES/README-Postfix-SASL-RedHat.txt;4eb44cae: cpio: MD5 sum mismatch

This is because I think FC15 uses XZ lossless data compression, and may also be signed using alternative (and stronger) hash algorithms such as SHA-256.

Nonetheless, the workaround here is to use --nomd5 when installing the srpm.

[root@relay32 SRPMS]# rpm -ivh postfix-2.8.6-1.fc15.src.rpm --nomd5
warning: postfix-2.8.6-1.fc15.src.rpm: Header V3 RSA/SHA256 signature: NOKEY, key ID 069c8460

1:postfix ########################################### [100%]

On trying to recreate the rpm I got the following error

[root@relay32 SPECS]# rpmbuild -ba postfix.spec
error: Failed build dependencies
libdb-devel is needed by postfix-2.8.6-1.x86_64

I tried to google around for this rpm, but could not find anything, some more digging around the web, I found that the rpm that was missing was db4-devel on my setup, and the postfix.spec file was referring to that package as libdb-devel, so I edited the BuildRequires line in the spec file

BuildRequires: db4-devel, pkgconfig, zlib-devel

Post, these changes the rpmbuild went ahead fine and I had my rpm tucked in the RPMS/x86_64 directory.

yum [Errno -3] Error performing checksum

2011-11-02T11:53:00.000-07:00

EDIT: One of my friends naresh has worked recently on this problem, refer to his blog here

I created a postfix rpm today and then created a yum repository to serve it, but whenever I would try to install this rpm from one of my client servers, I would get the error below

[root@server yum]# yum list postfix
Loaded plugins: fastestmirror
Loading mirror speeds from cached hostfile
* base: centos.kiewel-online.ch
* epel: ftp.uni-koeln.de
* extras: centos.kiewel-online.ch
* rpmforge: fr2.rpmfind.net
* updates: centos.kiewel-online.ch
my                                                                                                                                             | 2.6 kB     00:00
my/primary_db                                                                                                                                  | 4.6 kB     00:00
http://server/repo/RPMS/x86_64/base/repodata/primary.sqlite.bz2: [Errno -3] Error performing checksum
Trying other mirror.
viva/primary_db                                                                                                                                  | 4.6 kB     00:00
http://server/repo/RPMS/x86_64/base/repodata/primary.sqlite.bz2: [Errno -3] Error performing checksum
Trying other mirror.
Error: failure: repodata/primary.sqlite.bz2 from my: [Errno 256] No more mirrors to try.

Google did not turn up too many helpful links, digging deeper in some Redhat docs, I found this

Because RPM packages for Red Hat Enterprise Linux 6 are compressed using the XZ lossless data compression format, and may also be signed using alternative (and stronger) hash algorithms such as SHA-256, it is not possible to run createrepo on Red Hat Enterprise Linux 5 to create the package metadata for Red Hat Enterprise Linux 6 packages. The createrepo command relies on rpm to open and inspect the packages, and rpm on Red Hat Enterprise Linux 5 is not able to open the improved Red Hat Enterprise Linux 6 RPM package format.

I then realized that I had created the rpm on Centos-5 and was creating the repo on a server running Centos-6, to see if that was the problem, I quickly created a test repo on a Centos-5 server and things worked as expected.

Another change in Centos-6 from Centos-5 is the location where one builds the rpm, the good old /usr/src/redhat is now extinct, now with Centos-6 the current location is /root/rpmbuild.

On not checking root mails and lost backups

2011-11-01T23:12:00.000-07:00

Not checking root mails is a bad habit, it's like not listening to your body(server).

Because of an incorrect script name in the crontab, my database backups were not running for the past 20 days or so, and, because I was not checking root mails, I could not be informed in time.

Somehow, Thank God, today I 'happped' to type the mail command and see all the crontab mails lying around, and fixed the problem.

Cron job in Cron.d fails to run

2011-11-01T00:23:00.000-07:00

I came across a weird situation today where I Installed a cron file in the /etc/cron.d directory to run every 5 minutes, the cron was supposed to be run as a non-root user. The problem was that the cron was just not running, neither was there any mail to root, nor any indication the /var/log/cron file that the cron could not be run due to some errors.

A quick google and I got the feeling that crond is unaware of the existence of the cron, hastily to fix the problem I did a crond restart and my cron started to run again, but I should have first tested with a reload.

I tried to reproduce this by putting another cron in /etc/cron.d file, but it started to execute without any problems.

GIT repo for Puppet howto

2011-10-29T06:11:00.000-07:00

This is a easy howto to set up a git repo(local) for puppet, complete with some interesting hooks

# Initialize the repo

mkdir /var/git/puppet
cd /var/git/puppet
git --bare init

# next set up the working directory for puppet

mkdir /etc/puppet
cd /etc/puppet
git init
git remote add origin file:///var/git/puppet/

# make some changes in the ~/.gitconfig file

[remote "origin"]
url = file:///var/git/puppet
fetch = +refs/heads/*:refs/remotes/origin/*

[branch "master"]
remote = origin
merge = refs/heads/master

# Initialize the repo by adding a file

touch a
git add a
git commit -m "Bismillah, initialize"
git push origin master

# Now set email notification on when users push changes to the repo

vi /var/git/puppet/config
[core]
        repositoryformatversion = 0
        filemode = true
        bare = true

[hooks]
        mailinglist = sysad@madinix.com
        showrev = "git show -C %s; echo"
        emailprefix = "[viva puppet] "

# Initialize the email script by changing the name of the hook from post-receive.sample to post-receive

cd /var/git/puppet/hooks
mv post-receive.sample post-receive

# Uncomment the line that sends mail in the file post-receive

. /usr/share/git-core/contrib/hooks/post-receive-email

Now, whenever someone does a push, an email with the details of the push will be sent to the email address sysad@madinix.com. Multiple email addresses can be supplied, seperated with ','

# There is another thing that I want to do when users push to the repo, the working directory should get updated on each push, thus we use this script and add it to the post-receive hook file

vi /usr/share/git-core/contrib/hooks/post-receive-pull

#!/bin/sh

cd /etc/puppet || exit
env -i git pull origin master

# Add this script path to /var/git/puppet/hooks/post-receive, so now whenever someone pushes the changes to the repo, a mail will be sent to the email addresses defined in the config file as well as a pull will be initiated from the working directory /etc/puppet

# Next you need to give write perms for the users who will be issuing 'git push' to the working directory as well as the repo, my favorite way is setfacl

setfact -Rm user:sameer:rwx /etc/puppet
setfact -Rm user:sameer:rwx /var/git/puppet

# Cloning the repo for the contributor sameer

su - sameer
git clone file:///var/git/puppet

Grep wont work

2011-10-18T22:57:00.000-07:00

This morning I had to write some scripts, I was just running some sample commands to test the proof of concept

[06:45]:root@CentOS-56-64-minimal:~:#:>postcat -q 4BFDD18C1A9
*** ENVELOPE RECORDS maildrop/0BF2118C031 ***
message_arrival_time: Wed Oct 19 06:44:53 2011
named_attribute: rewrite_context=local
sender_fullname: root
sender: root
recipient: samix_119@yahoo.com
*** MESSAGE CONTENTS maildrop/0BF2118C031 ***
To: samix_119@yahoo.com
Subject: test

Then I tried to grep the subject line from the output, but grep wont return anything.

[07:52]:root@CentOS-56-64-minimal:~:#:>postcat -q 4BFDD18C1A9 | grep subject
[07:52]:root@CentOS-56-64-minimal:~:#:>

I tried to then grep the output of ls, no avail, I then tried to see if grep is really grep, and bam I found the problem

[07:52]:root@CentOS-56-64-minimal:~:#:> file /bin/grep
/bin/grep: Empty File

It seems that somone inadvertatnly over-wrote the /bin/grep file and caused this unlikely bug, I just uninstalled and reinstalled the grep rpm to fix this problem.

Yahoo Dictact - Don't switch back

2011-06-28T23:22:00.000-07:00

Yahoo seems to be acting as a medivial, despotic war lord, Inspite of the new mail UI being buggy, yahoo has disabled switching back to the good old classic format, nice way to kill your buisness and drive people to gmail.

Powerful wget Options

2011-06-16T03:04:00.000-07:00

Often while downloading stuff via wget, firewalls and other abuse detection software block the IP becuase wget has caused a lot of load on the server.

Using wget and being a good netizen go hand in hand, when using wget you can use the --wait option to specify the time between two hits thereby reducing the load caused on the server

wget http://link_to_the_site.com --wait=10 --recursive

To save on bandwidth you can also specify the type of file that you want to do download, or the type of file you do not want to download

-R sets the file extenstions that you do not want to download and -A sets the file extentions that you want to download, so

wget http://link_to_the_site.com --wait=10 --recursive -A html,php,xml

-A html,php,xml - will only download files that end with html,php,xml

wget http://link_to_the_site.com --wait=10 --recursive -R png,mp3,wav

-R png,mp3,wav - will not download any files that end with these extentions

This can help you save on bandwidth and also avoid downloading useless files.

Useless crontab mails for tar

2011-06-13T03:03:00.000-07:00

If you have ever used tar in a script or directly as a crontab command you may have got crontab mails each time tar ran with the following text

tar: Removing leading `/' from member names

Though the excersise, ie removing the leading '/' is indeed good and wanted, the mails that cron sends in every morning are indeed irritating, what to do ?

A quick google seach and helpful comments came up with the -C switch, so the command

tar -cvf archive.tar /application/repo/data_dir

becomes,

tar -cvf archive.tar -C /application/repo data_dir

-C causes tar to first switch root and go to directory /application/repo data_dir, leaving no need to remove any leading slashes (/)

Precedence of the -C switch before the names of the file or directory to be tar'ed matters, the following invocation is incorrect and tar will error out

tar -cvf archive.tar data_dir -C /application/repo

tar: data_dir: Cannot stat: No such file or directory
tar: Error exit delayed from previous errors

The correct invocation would be

tar -cvf archive.tar -C /application/repo data_dir

There you go, no more useless mails from cron.

How to use lock files

2011-06-01T05:05:00.000-07:00

Many a time you may want to launch an instance of a script if no other instance of the same script is currently running, I have seen many sysadmin friends use a lock file for this purpose because a lock file is kind of cool.

I have observed many faulty approches to using lock files. Generally when using lock files, simple callous scripts work like this

if [ ! -e $lockfile ]
then
   touch $lockfile
   do_something
   rm $lockfile
else
   #already running
   exit
fi

One of the places that this simple mechanism will fail is if the script after the function do_something is unable to delete the lock file for some reason, due to this every time the next instance of the script initiates it will just exit expecting that one instance is already running.

This can be overcome by storing the PID of the script in the lockfile and then checking if the same PID exists in the process list, the above script can become something like

if [ ! -e $lockfile ]
then
   echo $$ > $lockfile
   do_something
   rm $lockfile
elif [ ps aux | grep `cat $lockfile` ]
   #one instance already active
   exit
else
   #lockfile is stale
   rm $lockfile
   echo $$ > $lockfile
   do_something
   rm $lockfile
fi

There are alternatives to figure out if the script is already running without using the lock file, this can be done by grepping the name of the script in the output of the ps command without any need for any additional file touching and set up.

if [ ! ps aux | grep -q [n]ame_of_script ]
then
do_something
else
#exit, already running
exit
fi

Short and simple though maybe less reliable

What is this SPF anyways ?

2011-05-30T06:44:00.000-07:00

Simple, let us assume that you are the owner of the domain myprivatedomain.com and have recently noticed that some spammer is using your domain name to send spam emails. What are your options ?

Of cource you cannot stop the spammer from using your domain name more than you can stop him from using his own domain name, right ? this is where SPF steps in, if there was a way for you to publish to the world that you only send out emails from IPs 1.1.1.1 and 2.2.2.2 then you will be able to expose the spammer for what they are. An Imposter.

This can be done by publishing a DNS TXT record or a DNS SPF record(if supported by the nameserver) where you specify the IP's through which you send emails for your domain, so basically you will associate the IP's of your outbound mail servers with your domain name and publish this association for all to see and decide.

---------------- 1.1.1.1
myprivatedomain.com ----------|
---------------- 2.2.2.2

In the above simple diagram I have associated two IP's 1.1.1.1 and 2.2.2.2 with the domain name myprivatedomain.com, this information will be available to any mail server that implements a SPF checking policy and depending upon the check the server can instantly figure weather the mail has been generated by your server or through a malicious spammer, That's SPF for you.

I have deliberately left out the technical parts and explanations as they abound on the internet.

SAMBA File Sharing - I

2011-05-15T01:02:00.000-07:00

Recently I was required to setup file sharing for our users, naturally the users are all on windows and we wanted a Linux back end, so what better than SAMBA ?

But we wanted a lot more features like :

Exposing the home directory of each user via SAMBA
Exposing a public_html directory for each user over the web
Exposing the SAMBA shares via the web but with authentication.
An integrated web interface for users to manage their passwords, reset etc
Audit user actions when they access the share via SAMBA
Scripts to add user, edit and delete them
And of course puppetizing the entire setup

Well, let us begin with setting up the SAMBA shares first in this post, the reqiured SAMBA rpms are

samba-client
samba-common
samba

Next, the conf file

[global]
        workgroup = COMPANY
        server string = COMPANY
        invalid users = root bin daemon nobody named sys tty disk mem kmem users
        encrypt passwords = yes
        syslog only = yes
        syslog = 2
        log level = 2
        printing = bsd

[DEPT1]
        comment = DEPT1 common share Directory
        valid users = @DEPT1
        read only = yes
        write list = @DEPT1
        path = /var/www/html/shares/DEPT1
        create mode = 0660
        directory mode = 0770

        vfs object = full_audit
        full_audit:prefix = %u|%I|%m|%S
        full_audit:success = mkdir rename unlink rmdir pwrite

[DEPT2]
        comment = DEPT2 common share Directory
        valid users = @DEPT2
        read only = yes
        write list = @DEPT2
        path = /var/www/html/shares/DEPT2
        create mode = 0660
        directory mode = 0770

        vfs object = full_audit
        full_audit:prefix = %u|%I|%m|%S
        full_audit:success = mkdir rename unlink rmdir pwrite

[PUBLIC]
        comment = common share Directory
        valid users = @DEPT1, @DEPT2
        read only = yes
        write list = @DEPT1, @DEPT2
        path = /var/www/html/shares/PUBLIC
        create mode = 0660

        vfs object = full_audit
        full_audit:prefix = %u|%I|%m|%S
        full_audit:success = mkdir rename unlink rmdir pwrite

[homes]
        browsable = no
        writable = yes

Most of the parameters are self explainatory, but lets a look at some of them.

invalid users:
It is always good to set root as an invalid user as it is a good security practice

valid users:
Indicates the users that are allowed to access this share, remember if you want a user to have write acess they have to be defined in the valid users section. here you can either specify a unix user or you an specify a unix group with the @symbol

vfs object:
As we want to audit all that our users are doing on the shares we have to use the stackable vfs objects, the vfs object that we are using in this case is audit

full_audit:prefix:
Prefix each log line with this information %u|%I|%m|%S where

%u - Username
%I - IP of the machine through which the action is being performed
%m - Hostname of the machine
%S - Share that is being affected

full_audit:success:
Log the following file operations mkdir rename unlink rmdir where

mkdir - make directory
unlink - delete file
rmdir - delete directory

syslog only:
Send all log lines to syslog, even audit log lines will go there

You should get log lines similar to this in your messages file

May 15 06:40:21 server smbd_audit: user.name|172.16.10.118|host252|DEPT1|linux_setlease|fail (Permission denied)|resource.pdf

Crontab Nicknames

2011-05-04T22:09:00.000-07:00

Now this is great, it was right there under my nose in the crontab manual and I never got to it except today.

My weekly cron was acting funny and running daily, so back I was pouring over the man page to refresh the meaning of each of the 5 control fields and I saw that for some standard schedules nick names can be used instead of the 5 control fields, to quote the man page

These special time specification "nicknames" are supported, which replace the 5 initial time and date fields, and are prefixed by the ’@’ character:

@reboot    :    Run once, at startup.
@yearly    :    Run once a year, ie. "0 0 1 1 *".
@annually :    Run once a year, ie. "0 0 1 1 *".
@monthly   :    Run once a month, ie. "0 0 1 * *".
@weekly    :    Run once a week, ie. "0 0 * * 0".
@daily     :    Run once a day, ie.   "0 0 * * *".
@hourly    :    Run once an hour, ie. "0 * * * *".

So, I edited my posgresql data backup cronjob to look like:

# This is for the postgresql DB backup - daily
@daily root /opt/scripts/wikidb_backup.sh daily

# This is for the postgresql DB backup - daily
@weekly root /opt/scripts/wikidb_backup.sh weekly

Adding Images MediaWiki

2011-05-04T03:07:00.000-07:00

I wanted to put up a HOWTO on our internal wiki, which is mediawiki, and wanted to upload some images but coudnt find out how, even the official media wiki document says:

This page explains the image syntax when editing the wiki. You or another user must usually upload an image before you can use it on a page.

But it does not explain how, another page states:

If file uploads are enabled, you can upload certain types of files to the wiki. This is particularly useful for uploading images, which you want to place on an article, but you can also upload other types of files.

But where to enable it ? finally with a little grep I found the switch in the Localsettings.php file and set it to true

$wgEnableUploads = true;

Reloaded my wiki page and got an 'upload file' option in the sidebar under toolbox

PostgreSQL Backup

2011-05-03T23:22:00.000-07:00

We have mediawiki installed with a postgresql back end and I have procrastinated for too long about setting up backup so today I finally decided to do it using shell + pg_dump + cron

I use the same script to do weekly as well as daily backup by passing the argument weekly or daily.

Next, after zipping and taring the backup I delete all backups that are more than 5 days old to save on disk space.

Here is the shell script:

#!/bin/bash

PATH=$PATH:/usr/kerberos/sbin:/usr/kerberos/bin:/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin:/root/bin

if [ $# -lt 1 ]
then
        echo "usage: $0 daily|weekly"
        exit 1
fi

if [ "$1" == "daily" ]
then
        BACKUPDIR=/database_backup/daily
        pg_dump -Uwikiuser wikidb > $BACKUPDIR/wikidb-daily-`date +%d%m%Y`
        if tar czf $BACKUPDIR/wikidb-daily-`date +%d%m%Y`.tar.gz $BACKUPDIR/wikidb-daily-`date +%d%m%Y`
        then
                rm -f $BACKUPDIR/wikidb-daily-`date +%d%m%Y`
        fi
        find $BACKUPDIR -name wikidb-daily\*tar.gz -mtime +5 -exec rm {} \;
        exit 0
fi

if [ "$1" == "weekly" ]
then
        BACKUPDIR=/database_backup/weekly
        pg_dump -Uwikiuser wikidb > $BACKUPDIR/wikidb-weekly-`date +%d%m%Y`
        if tar czf $BACKUPDIR/wikidb-weekly-`date +%d%m%Y`.tar.gz $BACKUPDIR/wikidb-weekly-`date +%d%m%Y`
        then
                rm -f $BACKUPDIR/wikidb-weekly-`date +%d%m%Y`
        fi
        find $BACKUPDIR -name wikidb-weekly\*tar.gz -mtime +5 -exec rm {} \;
        exit 0
fi

echo "usage: $0 daily|weekly"
exit 1

I normally set the path like above in all my shell scripts that I run from the cron so that I do not have to write full paths for command name, its just not natural..

Next shove it into cron so that it runs every day every and weekly

# This is for the postgresql DB backup - daily
01 00 * * * root /opt/scripts/wikidb_backup.sh daily

# This is for the postgresql DB backup - daily
01 00 * * 4 root /opt/scripts/wikidb_backup.sh weekly

I also plan to scp the backup to some remote server so that if this server crashes completely I still can restore from there.

Abusing PostgreSQL Vacuum

2011-04-30T23:43:00.000-07:00

I run a wiki site in my company here, and we use media wiki for this purpose, the back end of the site is a PostgreSQL database, It just occurred to me last night that I am not vacuuming the database, so I thought that I might as well set it up.

The postgresql manual explains Vacuuming as,

VACUUM reclaims storage occupied by dead tuples. In normal PostgreSQL operation, tuples that are deleted or obsoleted by an update are not physically removed from their table; they remain present until a VACUUM is done. Therefore it's necessary to do VACUUM periodically, especially on frequently-updated tables.

I am running postgresql 8.4.4

template1=# show server_version;
server_version
----------------
8.4.4
(1 row)

Reading postgresql wiki, I figured that running a full vacuum on my database is almost useless, rather detrimental as most of the operations on our wiki site are insert operations, and we rarely delete anything once it is put in.

Many people, either based on misguided advice on the 'net or on the assumption that it must be "better", periodically run VACUUM FULL on their tables. This is generally a really bad idea and can make your database slower not faster.

So, I thought of putting a vacuumdb cronjob without the -f 'full' option, but a little more reading and I found that postgreSQL database engines 8.4 and above come with an autovaccum daemon which takes care of the vacuum requirements, I did a quick ps and found the launcher process

[root@myhost ~]# ps aux | grep -i auto
postgres 2901 0.0 0.0 50348 776 ? Ss Apr27 0:00 postgres: autovacuum launcher process

In case you do not like the autovacuum daemon, you can disable autovacuum in the postgresql.conf file

autovacuum = off #on by default

Perl logging to syslog

2011-04-28T01:55:00.000-07:00

Have you ever felt the need to log stuff from your perl script to syslog because you want to send the debug information to the relevant place or just because it looks cool ?

look no further, here is how you can do this in a couple of steps using the Sys::Syslog module, here is a simple script to demonstrate this

#!/usr/bin/perl
use strict;
use warnings;

use Sys::Syslog qw(:DEFAULT setlogsock);
setlogsock('unix');
openlog('label-for -your-script','','info');

syslog('info', "look for me in messages");

So, now when you execute this script you will get a line logged in /var/log/messages, similar to this

Apr 28 08:35:09 myhost label-for -your-script: look for me in messages

One of the places where I used this was in a script that does a couple of tasks on a user request and does copius logging to syslog, the problem that I faced was that there was no way for me except time to club together the various log lines.

Inspired by queue id I just edited the script to generate a unique 6 character key that I tagged along with each log line so that I can then grep for that unique character from the messages file

#!/usr/bin/perl
use strict;
use warnings;

use Sys::Syslog qw(:DEFAULT setlogsock);
setlogsock('unix');
openlog('label-for -your-script','','info');

my $random_id = random_string(6);

syslog('info', "$random_id: jack and jill");
syslog('info', "$random_id: went up the hill");
syslog('info', "$random_id: to fetch a pail of water");

# generate a 6 digit random charecter
sub random_string {
my $length_of_randomstring=shift;
my @chars=('a'..'z','A'..'Z','0'..'9','_');
my $random_string;

foreach (1..$length_of_randomstring){
$random_string.=$chars[rand @chars];
}

return $random_string;
}

Now, if I run the script two three times, using the $random_id I can figure which log line was printed during which run. so with a little grep magic I can arrange the logs according to each run of the script

[root@myhost scripts]# grep LCEd9X: /var/log/messages
Apr 28 08:48:05 myhost label-for -your-script: LCEd9X: jack and jill
Apr 28 08:48:05 myhost label-for -your-script: LCEd9X: went up the hill
Apr 28 08:48:05 myhost label-for -your-script: LCEd9X: to fetch a pail of water

[root@myhost scripts]# grep uYLs6s: /var/log/messages
Apr 28 08:48:05 myhost label-for -your-script: uYLs6s: jack and jill
Apr 28 08:48:05 myhost label-for -your-script: uYLs6s: went up the hill
Apr 28 08:48:05 myhost label-for -your-script: uYLs6s: to fetch a pail of water

[root@myhost scripts]# grep MWCz8p: /var/log/messages

Apr 28 08:48:03 myhost label-for -your-script: MWCz8p: jack and jill
Apr 28 08:48:03 myhost label-for -your-script: MWCz8p: went up the hill
Apr 28 08:48:03 myhost label-for -your-script: MWCz8p: to fetch a pail of water

Lesser known keyboard combinations

2011-04-27T21:17:00.000-07:00

Here is one interesting keyboard that may come in handy while troubleshooting

shift+scrollLock

This key combination dumps a lot of information with respect to the system memory, a dump similar to this, by the way this works only on the tty consoles.

cpu 0 hot: high 0, batch 1 used:0
cpu 0 cold: high 0, batch 1 used:0
cpu 1 hot: high 0, batch 1 used:0
cpu 1 cold: high 0, batch 1 used:0
DMA32 per-cpu: empty
Normal per-cpu:
cpu 0 hot: high 186, batch 31 used:27
cpu 0 cold: high 62, batch 15 used:7
cpu 1 hot: high 186, batch 31 used:165
cpu 1 cold: high 62, batch 15 used:2
HighMem per-cpu:
cpu 0 hot: high 42, batch 7 used:6
cpu 0 cold: high 14, batch 3 used:12
cpu 1 hot: high 42, batch 7 used:11
cpu 1 cold: high 14, batch 3 used:0
Free pages: 66288kB (496kB HighMem)
Active:185046 inactive:35966 dirty:36 writeback:0 unstable:0 free:16572 slab:10810 mapped-file:25269 mapped-anon:149285 pagetables:1542
DMA free:4316kB min:68kB low:84kB high:100kB active:7708kB inactive:0kB present:16384kB pages_scanned:0 all_unreclaimable? no
lowmem_reserve[]: 0 0 880 998
DMA32 free:0kB min:0kB low:0kB high:0kB active:0kB inactive:0kB present:0kB pages_scanned:0 all_unreclaimable? no
lowmem_reserve[]: 0 0 880 998
Normal free:61476kB min:3756kB low:4692kB high:5632kB active:624292kB inactive:135404kB present:901120kB pages_scanned:0 all_unreclaimable? no
lowmem_reserve[]: 0 0 0 948
HighMem free:496kB min:128kB low:252kB high:380kB active:108184kB inactive:8460kB present:121368kB pages_scanned:0 all_unreclaimable? no
lowmem_reserve[]: 0 0 0 0
DMA: 11*4kB 2*8kB 4*16kB 5*32kB 1*64kB 1*128kB 1*256kB 1*512kB 1*1024kB 1*2048kB 0*4096kB = 4316kB
DMA32: empty
Normal: 843*4kB 4217*8kB 939*16kB 78*32kB 21*64kB 3*128kB 0*256kB 0*512kB 1*1024kB 0*2048kB 1*4096kB = 61476kB
HighMem: 52*4kB 6*8kB 1*16kB 1*32kB 1*64kB 1*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 496kB
71729 pagecache pages
Swap cache: add 188, delete 188, find 0/0, race 0+0
Free swap = 2007332kB
Total swap = 2008084kB

Enabling arabic on centos

2011-04-27T04:05:00.000-07:00

considering that fact that I am working in the middle east, one of the jobs I have to do too often is to enable arabic on linux systems that we run so here is how to get this done

Install the arabic font rpm

yum install fonts-arabic 

now to enable switching between languages (english and arabic) run the following command

setxkbmap -symbols "pc(pc102)+jp(latin)+ara:2+group(alt_shift_toggle) " -option "grp_led:caps"

puppet manifest for this rpm left as an exercise

Yum for new users...

2011-04-26T23:28:00.000-07:00

normally under centos or Redhat like operating systems package management is in the form of RPMS

initially installing RPMs and their dependencies was a very complicated and long drawn manual process, until the arival of yum, so now with the appropriate repositories installed, yum does all the dirty work for you when the magical command is run

yum install packagename # and so it will be done , after you press a 'y' ofcourse...

now some terms that confuse new users

Repositories:
Repositories are mainly ftp ot http sites that expose rpms and metadata files, these repositories can be anywhere in the universe as long as the internet connection can span.

Accessing the repositories is what matters a mortal user and yum does it for you using .repo files which are stored under /etc/yum.repos.d, a typical .repo file will look like this

name=Software Repository
baseurl=http://fqdn.for.repo/repo/
enabled=1
gpgcheck=1
gpgkey=http://fqdn.for.repo/RPM-GPG-KEY

You can keep adding repositories like rpmforge etc by installing rpms that they provide.

How Yum works
So now when you type "yum install packagename" yum goes and reads the .repo files under /etc/yum.repos.d and connects to each of these localtions and tries to retrive the list of packages that the repository is holding ,the number of repositories that yum connects to is equal to the number of .repo files that have "enabled=1"

next, it figures out if the package you requested in available and then installs it.

Yum options
Yum provides many other commands like list,update,groupinstall etc, read the manual for yum to figure out their operations.

Desktop Recording

2011-04-26T22:56:00.000-07:00

Have you ever felt the need to record your dektop session ? well now you can with - recordmydesktop, as the name suggests it is a desktop recording utility for GNU/Linux.

Some days ago we needed to record a vnc session, vncserver does not seem to provide any such facility thus we installed the x11vnc which exposes -display 0 to the vnc-client, on top of this we ran recordmydesktop to record the session.

Step 1: Install x11vnc
yum install x11vnc

Step 2: start x11vnc
x11vnc -display:0 -forever

The -forever option allows x11vnc to run like a daemon, sans "-forever" the clinet will be able to take only one vnc session and the x11vnc deamon will terminate

This sets up the vnc serverside stuff for us, now onto recordmydesktop

Step 1: Install recordmydesktop
yum install recordmydesktop

Step 2: start recordmydesktop
recordmydesktop --display :0 -o recordingfilename.ogv #this can be run from the command line

Now lets go a step further and try to write puppet manifests for these two applications

Puppet

manifest for recordmydesktop -

class recordmydesktop {
   package { "recordmydesktop":
       ensure => installed,
   }
}

manifest for x11vnc -

class x11vnc {
   package { "x11vnc":
      ensure => "installed", # here you can specufy a version or latest
      notify => Service["x11vnc"],
      provider => "yum",
}

   ## if you have a init script to handle
   file { "/etc/init.d/x11vnc":
      ensure => "present",
      source => "puppet:///modules/x11vnc/x11vnc.$hostname",
      owner => "root",
      group => "root",
   }

   service { "x11vnc":
      ensure => "running",
   }
}

To be or not to be...

2011-04-26T03:15:00.000-07:00

After working for three years in Directi in an intense IT setup I switched to Qatar to work in the Qatar Civil Aviation authority.

Things are great here and we have a limited setup leaving a lot of time for me to delve in the vast ocean of open source technologies.

This blog will be an attempt InshaAllah to salvage my current skills and foster them, I will be delighted If I am able to help people along the path..