Back in 2009 I made a PHP script that tested network speed, wrote the results to text file and generated PNG graphs to be shown on web. It used speedtest.net partner servers and it got the data usage from ifconfig thus making it read the total bandwidth, not just what had been left for the php script itself. I set it up on my home router so I could prove my ISP that their internet is far form what they advertised.

I also uploaded it to Sourceforge and forgot about it’s existence for some time.

Years have passed and it has been downloaded almost 17 000 times, even though it doesn’t really work in every case and it wasn’t that easy to set up.

So I figured I should make a new version, written in Python (just to practice it), and so I did.

The new Tespeed is available at Github: https://github.com/Janhouse/tespeed

It is licensed under MIT license.

There are still some bugs and planned features left to sort out, but after testing it on multiple computers it is working fine.

Tespeed in automatic mode looking for best testing server and doing download and upload tests.

The new version finds closest servers and then picks the one with lowest latency.

Available server list top 25 (by distance)

You can also get a list of all available servers (and see the approximate distance to them) by using list-servers command.

Manually specified test server.

Please send me some feedback so I could make it better.

And go thank speedtest.net for this great service.

Since some of you wanted more information, I figured I should write about it. This will still only scratch the surface of it and you should do some additional reading if you want to get a better understanding of how these things work and what they can be used for.

Let’s begin!

The idea behind this tutorial is that you have 2 servers. One is used as a proxy and static content provider (yes, this is the box your other server is  hiding behind) and the other one is the server you keep your actual site on (database, scripts, etc.).

When user requests page on your site, proxy server passes this request to the backend server. Backend server then generates the page and sends it back to the proxy. Proxy then returns the page to the user requesting it.

To make it a bit faster, static content like images, CSS and javascript files are stored on your proxy server. These static files are synchronised from backend.

Also, to make it more secure and fast, multiple SSH tunnels are used between proxy and backend. All page requests from proxy to backend go through these SSH tunnels and web server on the backend listens for connections only locally so the site would not be accessible by making request to the backend directly.

Following diagram illustrates how it works:

Remember to tunnel everything, including your SMTP (e-mail) and possibly some other traffic. I already wrote about mini_sendmail that can help you tunneling SMTP.

Also, log files on the proxy server should be disabled and backend server IP address should not appear in any configuration file on the proxy server.

In my case I used Nginx as web server and proxy, rsync as the tool to synchronise static files between the servers, autossh to keep tunnels open, mini_sendmail to help tunnel SMTP.

To start the reverse tunnel using Autossh, use something like this on backend:

export AUTOSSH_PIDFILE=/tmp/tun1.pid && autossh -M 2001 -fNgR 5101:127.0.0.1:10005 user@example.com
export AUTOSSH_PIDFILE=/tmp/tun2.pid && autossh -M 2003 -fNgR 5102:127.0.0.1:10005 user@example.com
export AUTOSSH_PIDFILE=/tmp/tun3.pid && autossh -M 2005 -fNgR 5103:127.0.0.1:10005 user@example.com
export AUTOSSH_PIDFILE=/tmp/tun4.pid && autossh -M 2007 -fNgR 5104:127.0.0.1:10005 user@example.com

I would make some script to easily start or stop them all at once and add it to server startup. Also remember that you should use key authentication to open SSH connection automatically.

Make backend Nginx listen only locally:

listen 127.0.0.1:10005 default;

Set up proxy box to pass all requests to backend:

upstream backend {
    server 127.0.0.1:5101;
    server 127.0.0.1:5102;
    server 127.0.0.1:5103;
    server 127.0.0.1:5104;
}
server {
    listen 80;
    server_name example.com;
    location / {
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_pass http://backend;
    }
}

Also add virtual host or location block for static content on proxy box.

For static file synchronisation, create rsync.sh, make it executable and add to crontab.

#!/bin/bash
SOURCEPATH='/srv/static/'
DESTPATH='/srv/www/'
DESTHOST='example.com'
DESTUSER='user'
LOGFILE='rsynsync.log'
rsync -av -rsh=ssh $SOURCEPATH $DESTUSER@$DESTHOST:$DESTPATH 2>&1 >> $LOGFILE
echo "Completed at: `/bin/date`" >> $LOGFILE

These are just the basic steps but they should give you an idea how to use this.

For example, I use Nginx proxying to make server changes unnoticeable. Some Bittorrent trackers use this method to hide their servers and come back sooner after their proxy servers are taken down.

Depending on the site you are hosting, every setup can be different, so use your imagination to make it suit your needs.

What I meant by “blurry line between real world and virtual one”  is that what happens on internet not always stays on internet. If you read news, it is possible that you have noticed articles about police raids on some sort of network based service providers or their servers, governments asking communication service providers (e-mail, IM, others) to hand them private conversations to help in their investigations, or blocking network access and disturbing donation receiving.

While some “shady” network services were started by paranoid people (in a good sense), who think a lot about their security and anonymity, most others don’t realise how important it could be or just don’t care.

Who should care about this? People who call themselves media, political organisations, internet pirates and people who want to stay anonymous.

Here are some quick tips that could be of some use to you:

1. Domain name

Since most of the visitors reach your service through domain name, make sure you use domain name(s) that can’t be easily taken away by your country’s government or get you in problems for using them.

2. Location

2.1. Choosing country

Make sure your files are hosted on servers outside of your government’s reach. In case of  piracy, countries that don’t care about piracy or anti-piracy are best suited for this (Canda?).

Also make sure that your country doesn’t have official international investigation agreements with the hoster country.

2.2. Choosing data center

It would not hurt if the data center would be located deep under ground in some fortress that was previously used as a bomb shelter. Probably most countries have those.

2.3. Don’t bring your work to home

If there is something physical that can’t be encrypted and hidden, don’t keep it at home. Hell, don’t keep it at home even if it is encrypted!

3. Laws of your country

In certain cases, make sure that you can’t be extradited to other country. If necessary, move to new home. If you are dealing with some really powerful people, this could be really tricky (Wikileaks case).

4. Inability to reach users after takedown

Nothing scares away users more than leaving them wondering what is going one. Make sure you have some social messaging account that is hosted on safe grounds, that all your users know about. In case of problems inform them using those tools.

5. Encryption

5.1.Disk encryption

Keep your data encrypted and don’t give the key to anyone. Also recommend this to your clients.

Since computers are getting more powerful all the time, complaining about CPU power needed to use data encryption is silly.

Also make sure that your encryption key can’t be accessed using cold boot attack. If needed, glue the RAM to the motherboard! Seriously, this could help.

5.2. Connection encryption

Use connection encryption between your server and clients.

If you don’t trust your government and certificates given out by some companies, make your own and make sure your clients recognize it.

7. Data loss and downtime

Good old saying “Real men don’t make backups” is meant more like a joke and should not be taken seriously. Do make backups! Keep them far away from your main server, hide them, encrypt them, but make sure you have them (and try not to loose the key).

Having not only your data but also server configuration backed up could help reduce downtime in case of server change.

8. Hiding

8.1. Fake identities for fake servers

Hiding your super powerful server behind cheap, anonymous VPS could help you stay unidentified by less powerful people. There are some hosting companies that provide cheap VPS hosting and allow you to enter fake owner data for small fee. If you can then hide your payment account and fake your domain name owner data, you could stay anonymous as long as your proxy hoster doesn’t give out your real server address. In this case even if your proxy is taken down, reopening is just a matter of getting new proxy server in some other part of the world.

8.2. Anonymous administration

There is always some risk that your server could be taken and data searched for leads.

If possible, leave no log files about your clients and administrators or make up some fake ones.

Use proxy! Tor Project should help you hide yourself.

8.3. Don’t use Skype

Since Skype was bought by Micro$oft, it can’t provide anonymous communication anymore. As an extra, they probably added the famous “generic crash library” to it. OK, to be serious, most of the public IM and e-mail networks should be considered unsafe. Set up your own private encrypted IM network and don’t log stuff.

But if we keep talking about Micro$soft, try to stay away from it. You never know what is hiding in their lame binaries and “security through obscurity” (M$’s motto?) is stupid. Open source software is the future, go with it, explore how “security by design” works.

Conclusion

It is hard to write universal hiding guide for every project and each case is different. If you think it is necessary, get someone to give you good advices.

Remember that it takes only one small mistake to fail completely.

Most importantly keep your conscience clean and be good! If you are doing it for the right reasons (and have good PR campaign), people will support you (shouldn’t they?).

P.S. Feel free to add more tips or point to some errors in the comment section. I’l try to keep this post up to date.

But there is a small problem. It is written to be used as Gnome panel applet. I guess that some things will change with Gnome 3 (maybe it will get better Gnome shell integration or something?) but the lead developer has written that there will be no tray icon for Hamster:

generally though the system or notification tray is the no-go zone and one that triggered all the shunning in the panels (too many icons behaving too differently etc).

I found that there is script to use it with Ubuntu’s indicator applet but could not find one for system tray.

So I did it myself.

Inactive icon

I modified hamster-appindicator script by Alberto Milone to work with system tray (instead of appindicator). It now behaves similar to the panel applet. Only noticeable difference is that you don’t see current running task name and time in the tray (because there is only an icon and no text) and how long it has been running, but you can see it in the tooltip of the tray icon.

Right click menu

To make it more usable, I have included green and red icons that are used to indicate running task (by default I made it use green icon but you can change it in the script).

Left mouse click toggles tracker window, right click shows menu.

Opened window

Icons and script can be downloaded from https://bitbucket.org/janhouse/hamster-tray/downloads .

• Copy icons to /usr/share/icons/ or ~/.icons/
• Install latest Hamster (I tested it with latest git version): http://projecthamster.wordpress.com/building-and-running/
• Get latest hamster-tray version from https://bitbucket.org/janhouse/hamster-tray/downloads
• make it executable and run it

I’l later make hamster-tray PKGBUILD for Archlinux and put it on AUR.

If you want to point out any errors (it probably has some since I was making it in rush in the middle of the night) leave a comment. I’l try to update it at some point.

P.S. YAY! I can use Hamster again!

UPDATE:

08.04.2011.

After upgrading my Archlinux today I noticed that it did not work anymore because upgrade contained Gnome3 stuff.

So the error was:

$ hamster-tray
Traceback (most recent call last):
  File "/usr/local/bin/hamster-tray", line 51, in <module>
    from hamster.applet import HamsterApplet
  File "/usr/lib/python2.7/site-packages/hamster/applet.py", line 29, in <module>
    import gnomeapplet
ImportError: libpanel-applet-2.so.0: cannot open shared object file: No such file or directory

Quick dirty fix for the moment is to edit applet.py and remove line:

import gnomeapplet

This will probably make it not work as Gnome 2 panel applet but I guess that those applets are removed from Gnome3 anyway.

This is just a temporary fix and I will try to completely rewrite this script as soon as I get some time.

Since you probably don’t want to set up sendmail for each chroot, you could use mini_sendmail. It will work as relay and will pass messages to actual sendmail.

The problem is that there is no way to specify a custom username or hostname and this could be quite important in some cases.

In order to solve this problem I did some quick and dirty modifications and here is the patch in case you need it:

--- Makefile
+++ Makefile
@@ -7,10 +7,10 @@
 BINDIR =	/usr/local/sbin
 MANDIR =	/usr/local/man
 CC =		gcc
-CFLAGS =	-O
-#CFLAGS =	-g
-LDFLAGS =	-s -static
-#LDFLAGS =	-g -static
+#CFLAGS =	-O
+CFLAGS =	-g
+#LDFLAGS =	-s -static
+LDFLAGS =	-g -static
 LDLIBS =	$(SYSV_LIBS)

 CC :=		$(DIET) $(CC)
--- mini_sendmail.c
+++ mini_sendmail.c
@@ -65,6 +65,8 @@
 static char* argv0;
 static char* fake_from;
 static int parse_message, verbose;
+static char* helo;
+static char* user;
 #ifdef DO_MINUS_SP
 static char* server;
 static short port;
@@ -80,7 +82,7 @@
 static void usage( void );
 static char* slurp_message( void );
 #ifdef DO_RECEIVED
-static char* make_received( char* from, char* username, char* hostname );
+static char* make_received( char* from, char* user, char* helo );
 #endif /* DO_RECEIVED */
 static void parse_for_recipients( char* message );
 static void add_recipient( char* recipient, int len );
@@ -111,6 +113,7 @@
     argv0 = argv[0];
     fake_from = (char*) 0;
     parse_message = 0;
+	server = "localhost";
 #ifdef DO_MINUS_SP
     server = "127.0.0.1";
     port = SMTP_PORT;
@@ -124,6 +127,10 @@
 	    fake_from = &(argv[argn][2]);
 	else if ( strcmp( argv[argn], "-t" ) == 0 )
 	    parse_message = 1;
+	else if ( strncmp( argv[argn], "-h", 2 ) == 0 && argv[argn][2] != '\0' )
+	    helo = &(argv[argn][2]);
+	else if ( strncmp( argv[argn], "-u", 2 ) == 0 && argv[argn][2] != '\0' )
+	    user = &(argv[argn][2]);
 #ifdef DO_MINUS_SP
 	else if ( strncmp( argv[argn], "-s", 2 ) == 0 && argv[argn][2] != '\0' )
 	    server = &(argv[argn][2]);
@@ -162,14 +169,22 @@
 #endif /* DO_GETPWUID */
 	}

+	if ( user == (char*) 0 ){
+		user=username;
+	}
+
     if ( gethostname( hostname, sizeof(hostname) - 1 ) < 0 )
 	show_error( "gethostname" );

+	if ( helo == (char*) 0 ){
+		helo=username;
+	}
+
     if ( fake_from == (char*) 0 )
-	(void) snprintf( from, sizeof(from), "%s@%s", username, hostname );
+	(void) snprintf( from, sizeof(from), "%s@%s", user, helo );
     else
 	if ( strchr( fake_from, '@' ) == (char*) 0 )
-	    (void) snprintf( from, sizeof(from), "%s@%s", fake_from, hostname );
+	    (void) snprintf( from, sizeof(from), "%s@%s", fake_from, helo );
 	else
 	    (void) snprintf( from, sizeof(from), "%s", fake_from );

@@ -181,7 +196,7 @@

     message = slurp_message();
 #ifdef DO_RECEIVED
-    received = make_received( from, username, hostname );
+    received = make_received( from, user, helo );
 #endif /* DO_RECEIVED */

     (void) signal( SIGALRM, sigcatch );
@@ -209,7 +224,7 @@
 	exit( 1 );
 	}

-    (void) snprintf( buf, sizeof(buf), "HELO %s", hostname );
+    (void) snprintf( buf, sizeof(buf), "HELO %s", helo );
     send_command( buf );
     status = read_response();
     if ( status != 250 )
@@ -337,7 +352,7 @@

 #ifdef DO_RECEIVED
 static char*
-make_received( char* from, char* username, char* hostname )
+make_received( char* from, char* user, char* helo )
     {
     int received_size;
     char* received;
@@ -349,8 +364,8 @@
     tmP = localtime( &t );
     (void) strftime( timestamp, sizeof(timestamp), "%a, %d %b %Y %T %Z", tmP );
     received_size =
-	500 + strlen( from ) + strlen( hostname ) * 2 + strlen( VERSION ) +
-	strlen( timestamp ) + strlen( username );
+	500 + strlen( from ) + strlen( helo ) * 2 + strlen( VERSION ) +
+	strlen( timestamp ) + strlen( user );
     received = (char*) malloc( received_size );
     if ( received == (char*) 0 )
 	{
@@ -360,7 +375,7 @@
     (void) snprintf(
 	received, received_size,
 	"Received: (from %s)\n\tby %s (%s);\n\t%s\n\t(sender %s@%s)\n",
-	from, hostname, VERSION, timestamp, username, hostname );
+	from, helo, VERSION, timestamp, user, helo );
     return received;
     }
 #endif /* DO_RECEIVED */

Save it as some.patch. Move it inside mini_sendmail source directory and run:

patch -p0 < some.patch
make
cp mini_sendmail /to/jail/usr/bin/sendmail

You can specify username with -u and hostname (and HELO message) with -h parameter.

If you are going to use it with PHP, change sendmail_path in php.ini to something like this:

sendmail_path = /usr/bin/sendmail -s127.0.0.1 -p5555 -unoreply -hexample.com -fnoreply@example.com -t -i

This should make php connect to sendmail running on 127.0.0.1 port 5555 and send example.com as HELO and noreply as username.

Patch was made for version 1.3.6.