Info
2005-12-30 multipart html emails
To deliver a plain text email consiting of a subject line and a body that contains 'Hello' we would create a session to a mail server with the following converstaion (S: denotes server, C: denotes client):
Connected to 192.168.0.1. Escape character is '^]'. S: 220 ednevitible.co.uk ESMTP C: helo S: 250 ednevitible.co.uk C: mail from: bob@mydomain.com S: 250 ok C: rcpt to: ed@mydomain.comS: 250 ok C: data S: 354 go ahead C: Subject: Hello C: C: Hello C: . S: 250 ok 1135982020 qp 7638 C: quit S: 221 ednevitible.co.uk Connection closed by foreign host.
What if we want to create a message that contains HTML text? Although, being a UNIX beard I have to say "Halt! HTML email is evil, read html email is evil before posting in HTML", please be aware, only a few people are able to read HTML email correctly, there are a LOT of users who have reading disabilities or simply decide not to read that kind of mail for reasons described in the last link.
helo mail from: ed@mydomain.com rcpt to: ed@mydomain.com data MIME-Version: 1.0 Content-Type: multipart/alternative; boundary="----=_Part_7816_22876637.1135968477396" Subject: mulitpart test From: ed@mydomain.com To: ed@mydomain.com ------=_Part_7816_22876637.1135968477396 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable Content-Disposition: inline Hello this is the text part, test failed ------=_Part_7816_22876637.1135968477396 Content-Type: text/html; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable Content-Disposition: inline <html> <body> <ol> <li>List</li> <li>List</li> <li>List</li> <li>List</li> </ol> <p style="left:50px">paragraph with indent. test passed</p> </body> </html> ------=_Part_7816_22876637.1135968477396-- .
Now, although it might look complex initially, this message body is divided into three parts:
- message headers
- text part
- html part
The actual parts of the message payload are divided using the Content-Type: multipart/alternative; boundary="----=_Part_7816_22876637.1135968477396" header. Each of the parts must leave a blank line between part headers and part body.
If the mailer can handle HTML parts then it should automatically display the HTML section. Do not put information in the HTML part which is not in the text part, the reader cannot be assumed to have ability to read HTML, it should only be used as an enhancement and remember, if it can't be said in plain text, then it's not worth saying.
Oh... this one is so old it's almost funny again:
[ed@shell]: who | grep -i blonde | talk ; cd ~ ; wine ; talk ; touch ; unzip ; touch ; strip ; gasp ; finger ; gasp ; mount ; fsck ; more ; yes ; gasp ; umount ; make clean ; sleep
2005-12-27 c recursion directory scan
I wrote something similar to this when aged 14, took ages to figure out why some segfaults were occouring, to find that I was freeing more memory than allocated (by one char).
int *countfiles( char* directory, int *counter )
{
struct dirent *dir;
DIR *handle;
handle = opendir( directory );
if( handle == NULL )
{
switch( errno )
{
case EACCES:
{
fprintf( stderr, "Access denied\n" );
break;
}
case EMFILE:
{
fprintf( stderr, "Too many files are currently open in the system\n" );
break;
}
case ENOENT:
{
fprintf( stderr, "Directory does not exist, or name is an empty string\n" );
break;
}
case ENOMEM:
{
fprintf( stderr, "Insufficient memory to complete the operation\n" );
break;
}
case ENOTDIR:
{
fprintf( stderr, "%s is not a directory\n", directory );
break;
}
}
}
while( handle != NULL && ( dir = readdir( handle ) ) != NULL )
{
if( dir->d_type == 8 )
{
*counter++;
fprintf( stderr, "< %s\n", dir->d_name );
}
if( dir->d_type == 4 && strcmp( dir->d_name, "." ) != 0 && strcmp( dir->d_name, ".." ) != 0 )
{
char *buffer=(char *)malloc( strlen( directory ) + strlen( dir->d_name ) + 2 );
strcpy( buffer, directory );
strcat( buffer, dir->d_name );
strcat( buffer, "/" );
fprintf( stderr, "> %s\n", buffer );
counter = countfiles( buffer, counter );
free( buffer );
}
}
if( handle != NULL )
{
closedir( handle );
}
return( counter );
}
int queuefiles( void )
{
int *b=0;
int *count;
b=countfiles( "/var/qmail/queue/mess/", b );
printf( "%d\n", b);
return 0;
}
qmail greylisting
One of the most infuriating things about having an email address is spam. Greylisting forces the sender to retry a few hours later, this is often after the spammer has finished their spamming run. which is great, because legitimate servers queue mail for about 7 days (or in ntlworld's case just 4).
The qgreylist perl program can be downloaded from http://www.jonatkins.com/qgreylist/. As it sits in front of qmail-smtpd, I believe it would agrovate users who send mail via smtpauth. Whitelisting does not sidestep the problem as they could be roadwarriors (users who roam and connect from any ISP).
No the only way to solve this is to patch qmail itself. This I am working on.
2005-12-25 double vision
Santa left a nice 19" tft under the tree pour moi, I'm now seeing double. From experience of my Dell Inspiron 5100 (radeon) getting xinerama to work was a nightmare, the radeon chip is not very good for this. However, to much of my surprise Nvidia have really gone an extra mile in making TwinView work really well. Rather than putting config into the ServerLayout, Monitor, Screen and Driver subsections, all the config just goes in Driver. So the only settings required, for my, regular settings are as follows:
69 Section "Device" 70 Identifier "geforce1" 71 Driver "nvidia" 72 Option "RenderAccel" "true" 73 Option "backingstore" "true" 74 Option "NoLogo" "true" 75 Option "TwinView" "true" 76 Option "ConnectedMonitor" "TFT, TFT" 77 Option "TwinViewOrientation" "DFP-0 LeftOf CRT-0 78 Option "AllowGLXWithComposite" "true" 79 VideoRam 2661120 80 Option "MetaModes" "1280x1024,1280x1024" 81 EndSection
2005-12-23 forking hell
This has nothing to do with a knife and fork topic, this is, basic multiprocessing. Unlike threads, forks are separate processes. Communication between forked processes must be done with other IPC (Inter Process Communication) techniques (such as pipes/sockets).
#include <sys/types.h>
#include <unistd.h>
#include <stdio.h>
int main( int argc, char *argv[] )
{
int pid=fork();
for( ;; )
{
printf( "In %s...\n", ( pid==0 ? "child" : "parent" ) );
sleep( 1 );
if( pid != 0 )
{
break;
}
}
}
What fork() does is literally copy the current userspace process into a new memory space and begin excution from the point where the fork() was called, but the return value of fork is 0 in the child process, in the parent it is the value of the child's pid. This is a very costly operation for the system.
Lets suppose though that you wish to create five child processes, one might think to just loop a fork() call five time, but alas, you would create 5 + 4 + 3 + 2 + 1 processes, as each child would also loop. The solution is to check the pid return value against !0:
#include <sys/types.h>
#include <unistd.h>
#include <stdio.h>
int main( int argc, char *argv[] )
{
int x=0;
int pid;
while( ( pid=fork() ) != 0 && x++<5 )
{
}
for( ;; )
{
printf( "In %s...\n", ( pid==0 ? "child" : "parent" ) );
sleep( 1 );
if( pid != 0 )
{
break;
}
}
}
2005-12-22 disk stats
For one reason or another, I have to write something to report system stats using a TCP socket, so I'm doing the task in c. The first module is to report disk usage, here's that module, earlier than the rest as it proved to be a pain to find the right syscall etc.
#include <stdio.h>
#include <string.h>
#include <sys/types.h>
#include <sys/vfs.h>
#include <sys/stat.h>
#include <unistd.h>
#define MAXPATH 4096
int strpos( char *ptr, char chr )
{
int pos = 0;
char *p=ptr;
while( p[pos] )
{
if( p[pos] != chr )
{
pos++;
}
else
{
break;
}
}
return( pos );
}
char *mountpoint( char *s, char *buffer )
{
const int index = 1;
char *p = s;
strncpy( buffer, s, MAXPATH );
int count = 0;
while( buffer[0] != ' ' | count < index )
{
buffer++;
if( buffer[0] == ' ' )
{
count++;
}
}
buffer++;
count = strpos( buffer, ' ' );
buffer[count]='\0';
return( buffer );
}
int main( int argc, char* argv[] )
{
int x;
struct statfs buf;
char ptr[MAXPATH], strmountpoint[MAXPATH]; // FIXME: use maxpath in future
FILE *fp;
if( ( fp=fopen( "/proc/mounts", "r" ) ) == NULL )
{
printf( "Could not open /proc/mounts\n" );
return;
}
while( ( fgets( ptr, MAXPATH, fp ) ) != NULL )
{
strncpy( strmountpoint, mountpoint( ptr, strmountpoint ), MAXPATH ), '\0';
if( strstr( ptr, "rootfs" ) || strstr( strmountpoint, "/dev/root" ) || strstr( strmountpoint, "/proc" ) || strstr( strmountpoint, "/dev" ) || strstr( strmountpoint, "/sys" ) )
{
continue;
}
// printf( "%s\n", strmountpoint );
x=statfs( strmountpoint, &buf );
if( x == 0 )
{
long long size = ( (long long)buf.f_blocks * (long long)buf.f_bsize ) / 1024 / 1024;
long long avail = ( (long long)buf.f_bavail * (long long)buf.f_bsize ) / 1024 / 1024 ;
float percent = ( ( (float)100 / (float)size ) * (float)avail );
// printf( "%ld %ld %f\n", size, avail, (float)( (float)100 / size ) );
printf( "%s %LdMB, free %LdMB %.0f%% free\n", strmountpoint, size, avail, percent );
}
}
fclose( fp );
return( 0 );
}
2005-12-20 just some links
Brilliant BSD wallpapers: http://www.bsdnexus.com/wallpapers.htm, and some okish Linux wallpapers: http://hackersplayground.org/humor/linuxchicks/
2005-12-18 atmosphere
The layers of the atmosphere are (starting at the Earth):
- troposphere
- stratosphere
- mesosphere
- thermosphere
- exosphere
books I have read
- Perl for the web
ISBN: 0-7357-1114-3
http://books.perl.org/book/152
This covers more information about making applications which are likely to be called repeatidly from a webserver than application development. - Perl for C programmers
ISBN: 0-7357-1228-X
http://www.amazon.co.uk/exec/obidos/ASIN/073571228X
Subjects covered include basic OOP and structs, file system functions, modules, inline c and GUIs. Quite a good read. This book came in second condition and therefore is lacking a nice cover. - The Complete Reference - Python
ISBN: 0-07-212718-X
http://www.amazon.co.uk/exec/obidos/ASIN/007212718X
This is the only book in the 'complete reference' serise that I have, and I'm quite impressed. The book is close in format to the Black Book serise, covering functions and classes in depth. The book opens with a comparision of Python to over languages which is good if the reader is unfamiliar to the language. Threads, network programming and GUI is also covered. Do not be daunted by the book's physical size as the paper is quite thick. - Advanced UNIX Programming
ISBN: 0-13-141154-3
http://www.amazon.co.uk/exec/obidos/ASIN/0131411543 Positively the best book on UNIX programming that i have read. It's packed full of knowledge. This is more of an encyclopaedia of information than a regular text book. The reader should have some foundation of C before attempting to read this, but then who picks up a book on 'Advanced UNIX Programming' without expecting that? From the beginning this book taught me things that I was not aware of and continues throughout the book. Highly recommended. I seriously advise the buyer invests 10 minutes to cover this with a clear sticky back plastic as you will have this book open for weeks before closing it. - UNIX Internals A practical approach
ISBN: 0-201-87721-X
http://www.amazon.co.uk/exec/obidos/ASIN/020187721x This book is great. It contains all the structures and information required to get coding for a SCO kernel. Although the cover states that this book is for SCO, don't let that put you off. At almost every chance the author states where code is SCO specific.
I've read many books on unix and linux and this is by far one of the best, you should not be afraid to get your sleeves rolled up for this book, and that's what makes it such a great read. - Linux game programming
ISBN: 0-7615-3255-2
http://www.amazon.co.uk/exec/obidos/ASIN/0761532552
Well, it's not as bad as the other reviews make out only if you have no idea how to start programming a game in Linux. I suppose the content would be better put into a shorter how-to (if one does not already exist that is). I had no idea how to start writing a game in Linux, didn't know where to start, but there are some basic guidelines in this. The book coveres some very basic theory such as game loops and very basic AI principles. - The UNIX programming environment
ISBN: 0-13-937681-X
http://www.amazon.co.uk/exec/obidos/ASIN/013937681X
The issue which I have is a little out dated compared to the one now advertised on Amazon with same ISBN. Topics covered such as shell and C programming. Covers mainly what the programmer needs to keep in mind for user space programming. - BSD Hacks
ISBN: 0-596-00679-9
http://www.amazon.co.uk/exec/obidos/ASIN/0596006799
This book covers a lot of topics in short, many of the hacks are user contributed. Listing all the topics here would take too much space. This book would be ideal for a new system admin. Most of the system admin features that a linux user migrating to BSD is after are covered in this book. - Linux - I didn't know you could do that...
ISBN: 0-7821-2612-X
http://www.amazon.co.uk/exec/obidos/ASIN/078212612X
Worst book ever. Very little technical substance. - For dummies - Linux Programming
ISBN: 0-7645-0691-9
http://www.amazon.co.uk/exec/obidos/ASIN/0764506919
Second worst book ever. Very basic shell scripting guide. - C++: Black book
ISBN: 1-57610-777-9
http://www.amazon.co.uk/exec/obidos/ASIN/1576107779
Quite a good read. Seriously good information in this book. Most of the focus is on C++ objects. File handling, pointers, overloading, operators etc are all covered in depth. - Java 2: Black book
ISBN: 1-58880-097-0
http://www.amazon.co.uk/exec/obidos/ASIN/1588800970
This book was bought for me some time ago when I was studying Java at college (1999). The most notable things which have changed from Java 2 tto 5.0 are Date and typed ArrayLists (and some Swing). But most remains the same. This is without any doubt the largest book on my shelf. Remains excellent and I have read from cover to cover twice now! - C Programming - A modern approach
ISBN: 0-393-96945-2
http://www.amazon.co.uk/exec/obidos/ASIN/0393969452
I rate this book closly to Advanced UNIX programming. I advise that anyone who plans to read AUP purchase this book first as important parts of AUP require prior C knowledge. Most important features covered are linked lists, structs, file IO and pointers. - Fundametals of Electronics
ISBN: pre ISBN
Really technical book, degree level material. - Engineering problem solving with ANSI C
ISBN: 0-13-061607-9
http://www.amazon.co.uk/exec/obidos/ASIN/0130616079
This is somewhere between a book on engineering and c. The actual C is intended for someone who is familar with computers and maths. This book was originnaly purchased as I wanted some engineering knowledge, believe it or not, it caught my eye. - Turbo C++ programmers guide
ISBN: no ISBN
This covers most of the features of the TurboC++ compiler and streams. Not much to note though. - C++ for dummies
ISBN: 1-56884-163-9
http://www.amazon.co.uk/exec/obidos/ASIN/1568841639
This coveres some surprisingly intermidiate concepts of C++. I purchased this in 1994 after TurboC++ programmers guide, at the time this made a lot more sense. I can't count the number of times that I was going between the IDE and this book. Almost everything you need to get coding C++ is covered in this book, all beit beginner or intermediate level. - C for dummies
ISBN: 1-56884-915-X
http://www.amazon.co.uk/exec/obidos/ASIN/156884915X
This does what it says on the cover, it's C for dummies, but ... not eveyrthing that is c, it's what a dummy needs to know. - Visual C++ 6.0 programmers guide
ISBN: 1-57231-866-X
http://www.amazon.co.uk/exec/obidos/ASIN/157231866X
This really is just for those who need to know how to use the various parts of the IDE. There are some sections in the book which are not directly related the IDE, but these are few and far between. One thing to note is that the code style influenced me slightly, I used to write functionname(&ptr,x,g,j), however, the general style in this book I found helps code legibility, functionname( &ptr, x, g, j ), isn't that just more useful? - Absolute OpenBSD: UNIX for the practical paranoid
ISBN: 1-886411-99-9
http://www.amazon.co.uk/exec/obidos/ASIN/1886411999
Absolute OpenBSD coveres some very unique features of OpenBSD, such as secure levels and pf. There are some good examples of firewalls at the end of the book. One interesting chapter in this book covers compiling your own bsd kernel, which is something that is not covered in depth anywhere that I know of. - Practical UNIX & Internet Security (2nd edition)
ISBN: 1-56592-148-8
http://www.amazon.co.uk/exec/obidos/ASIN/0596003234
For a 900-page document this book did not keep me occupied for too long. It might be that most of the text is common place, some obvious and old flaws are discussed along with mediocre system security such as file permissions, pid owners etc, and some basic ipv4/ports. - Managing AFS - Andrew File System
ISBN: 0-13-802729-3
http://www.amazon.co.uk/exec/obidos/ASIN/0138027293
Unfortunately I have not had time to read this book cover to cover like the rest. Building a file store network is something which takes much time, especially when you have to meadle with the kernel modules time and time again so that the afs module is up-to-date with the rest of the kernel (dpkg can help, but it's still a chore). There are many points that must be covered with AFS, all explained there in. Had I ordered from Amazon and not one of the shops I would have had it sooner. - Building firewalls with OpenBSD and PF 2nd Edt
ISBN: 83-916651-1-9
http://www.amazon.co.uk/exec/obidos/ASIN/8391665119
Brilliant book that covers everything you need for a single node firewall. Certain things could be covered in further depth though such as CARP/pfsync and route-to/reply-to which are required for building larger corporate firewalls. That aside this book is tops, highly recommeneded for anyone who wants to be a firewall admin.
2005-12-17 night out and stuff at venom night club
Well, as the title suggests I plan on a night out at the club. I have not been out much this year (can count outings on a single hand), so I plan to go a bit 'mental' tonight.
In other news, I've written next to no code today. How disappointing, Saturday is normally my most productive day of the week, no distractions, just get my head down and usually churn out quite a bit.
2005-12-15 java is so 90's
I consider C/C++ to be very usable languages, there's nothing wrong with them. C can produce very efficient, stable and portable programs (programmer willing of course). C++ is capable of good OOP, again, nothing wrong with this either. Yet I consider java to be strong, stable, portable and recent.
An article at business week suggests that perl/python/php are the language of the future. Personally they should not be thrown into the same comparision. All these languages are different, have their own strengths and weaknesses and any form of comparison is ludicrous and pointless.
In other news, IBM is the favourite UNIX vendor, with Sun comming last in survey results.
2005-12-14 openbsd does nice things
I have never been one for iptables so I am unsure if it is possible with the linux kernel to balance outgoing connections. At least with openbsd/pf it is possible to route outbound packets through a pool of connections. This is, however limited to the number of interfaces you have in the box. My initial attempt was to use IP aliasing, that would have been just too perfect to expect it, but well, it's not too much to ask really, either run out of PCI slots of have a 4 port net card, which probably limits to 20 connections in the pool.
I have yet to do sufficient testing to evaluate the various performance loads though major connection pooling. I imagine it will be minor on a £250 box, however I have to do some testing for it is running in CARP/pfsync. As you have probably guessed, I am somewhat impressed, proud and pleased with this. Oh and I'm not about to give away it's configuration.
2005-12-12 clean tags
Recently I have needed to remove all text from HTML leaving just the tags. This was required for some form processing such that I could examine just certain parts of the document where the paragraph text would just get in the way. The following will remove everything other than elements.
import java.io.*;
/*
* 20051211 en removes text from html leaving just tags
**/
public class CleanTag
{
public CleanTag( String s )
{
}
private static String removeFormatting( String text )
{
return( text.replaceAll( "\n", " " ) );
}
public static String clean( String s )
{
StringBuffer data = new StringBuffer( 5000 );
int pos = 0,nextpos=0;
try
{
// find next tag
while( ( nextpos = s.indexOf( "<" ) ) != -1 )
{
pos = s.indexOf( ">" );
if( pos < nextpos )
{
while( pos < nextpos )
{
s = s.substring( nextpos );
pos = s.indexOf( ">" );
nextpos = s.indexOf( "<" );
}
}
data.append( removeFormatting( s.substring( nextpos, pos+1 ) ) + "\n" );
s = s.substring( pos+1 );
}
}
catch( Exception e )
{
DP.o( "Nextpos " + nextpos + " pos " + pos );
e.printStackTrace();
System.exit( 0 );
}
return( data.toString() );
}
}
It may not look like much, but it's here just for reference.
In other rambling thoughts, there are more jobs running in sh/bash which I have written, but comprise fewer lines than the fewer jobs I have in other languages. Is this a sign of higher level languages acrually doing less than functional languages?
2005-12-09 makes a change
This post has no technical merit. I've been quite busy checking and double checking things. On the other hand some interesting things have happened. Two co-workers and myself have been photographed for the local paper, the print went in twice in the jobs section, in colour.
In terms of training, I have decided that I am consuming more than I am working off during a regular week. Maybe it's just the festive spirit but the chances are that my cardiovascular work needs extending. One other fact to consider is that my running speed on the treadmill has been decreasing this past year, so one of two things has happened, my heath has deteriorated, or I'm getting lazy.
2005-12-03 ssh tunnel
SSH (Secure SHell) is perhaps more widely used than FTP these days and for good reason too, despite it being able to do 90% of you can do with regular FTP in addtion to it's networking applications.
You might want to create some keypairs to ease the login process first (2005-10) to avoid having to manually log in each time you create a tunnel.
Tunnels can be used on hosts with default SSH configuration, using the following command:
ssh -L [localbindport]:[remote network ip]:[remote bind port] [sshd server]
- localbindport
This is the port that you should consider the tunnel to run as on the local system - remote network ip
The remote network IP is the IP that that the tunnel binds to on the destination side, where the sshd server is running - remote bind ip
This is the IP address that the destination tunnel connects to - sshd server
This should be self explanatory, this is where you wish to tunnel through
For example if the gateway we want to connect to has the IP address of 14.14.14.14 on the internet with a private network of 192.168.100.0/24 with a host on that network of 192.168.100.5, running remote desktop, we would use the following to bind localhost:65500 to the 192.168.100.5:3389:
ssh -L 65500:192.168.100.5:3389 14.14.14.14
There are other applications for SSH, it has the well known scp (secure copy) and perhaps lesser known scponlyc shell, which restricts logins to use scp only. These applications have added extreme possibilities to the way that we use the internet. FTP has, for example given administrators great difficulty depending on the connecting network. But with great power comes great responsibility, there are many scanning programs in the wild which test user login credentials with desire to gain entry and escalate permissions through local exploits.