php – Janhouse.lv

Solving spam problems without using captcha

Some of you know that automated spam bots for the web can be pain in the ass.

And while there are many ways to deal with this problem, most of them complicate things for the regular user and are not hacky enough to satisfy me. 🙂 For example captchas, math problem solving, etc. mostly work fine but they also scare away some of the users. There are also some paid tools (Akismet?) that helps dealing with spam but it doesn’t work on all spam.

I will share information on how I stop almost all of the automated spam messages to my WordPress blog without complicating things for the user.

Knowing limitations of HTTP protocol libraries and “exploiting” them

I have been writing a lot of scripts that automate GET’ing and POST’ing things to the web. There are many libraries that help doing it and one of the most popular ones (if not the most popular) is Curl.

While Curl is great, it still lacks some features, like building a zero length multipart POST data file upload part. Basically it can’t simulate file upload field, that has no file selected for uploading. And since many spam bots use Curl and similar libraries, this can be used to identify real browser and a script.

Example usage in PHP:

if(!isset($_FILES['spamcheck']) or $_FILES['spamcheck']['name']!='' or $_FILES['spamcheck']['size']!=0 )

1	if(!isset($_FILES['spamcheck']) or $_FILES['spamcheck']['name']!='' or $_FILES['spamcheck']['size']!=0 )

Not just Curl

I haven’t used Curl for a year now because I switched to Perl’s LWP. And it has the same problem. I solved it by making my own function for building the multipart form data and it can simulate the behavior but it doesn’t happen by default.

And while in case of LWP it was rather simple, doing it in Curl (used in PHP) would probably require recompiling the Curl library and then recompiling the module for the programming language that is using that library.

Changing field names

Another trick I use is changing field names and adding additional ones that are meant to be left empty.

This method also seems to catch part of spam messages.

From the user point of view

User obviously doesn’t see those extra fields because they get hidden using CSS. The comment form looks like any other standard comment form.

Conclusion

I have been using these techniques for about a year now and haven’t had any spam problems since then. Also I have a huge log file with spam that was caught using these methods. This really does work. 🙂

Of course it won’t help in case of directly targeted spam but in that case captchas won’t help either.

And I understand that writing about this will probably contribute to fixing Curl and other libraries and eventually making this protection method useless. Well, at least they will finally fix those libraries! 😀

Jailkit, mini_sendmail and custom HELO

To be sure that a server stays safe in case when one site is compromised, I try to lock every single site in its own chroot jail. To make it a bit easier I use Jailkit.

Since you probably don’t want to set up sendmail for each chroot, you could use mini_sendmail. It will work as relay and will pass messages to actual sendmail.

The problem is that there is no way to specify a custom username or hostname and this could be quite important in some cases.

In order to solve this problem I did some quick and dirty modifications and here is the patch in case you need it:

--- Makefile
+++ Makefile
@@ -7,10 +7,10 @@
BINDIR = /usr/local/sbin
MANDIR = /usr/local/man
CC = gcc
-CFLAGS = -O
-#CFLAGS = -g
-LDFLAGS = -s -static
-#LDFLAGS = -g -static
+#CFLAGS = -O
+CFLAGS = -g
+#LDFLAGS = -s -static
+LDFLAGS = -g -static
LDLIBS = $(SYSV_LIBS)
CC := $(DIET) $(CC)
--- mini_sendmail.c
+++ mini_sendmail.c
@@ -65,6 +65,8 @@
static char* argv0;
static char* fake_from;
static int parse_message, verbose;
+static char* helo;
+static char* user;
#ifdef DO_MINUS_SP
static char* server;
static short port;
@@ -80,7 +82,7 @@
static void usage( void );
static char* slurp_message( void );
#ifdef DO_RECEIVED
-static char* make_received( char* from, char* username, char* hostname );
+static char* make_received( char* from, char* user, char* helo );
#endif /* DO_RECEIVED */
static void parse_for_recipients( char* message );
static void add_recipient( char* recipient, int len );
@@ -111,6 +113,7 @@
argv0 = argv[0];
fake_from = (char*) 0;
parse_message = 0;
+ server = "localhost";
#ifdef DO_MINUS_SP
server = "127.0.0.1";
port = SMTP_PORT;
@@ -124,6 +127,10 @@
fake_from = &amp;(argv[argn][2]);
else if ( strcmp( argv[argn], "-t" ) == 0 )
parse_message = 1;
+ else if ( strncmp( argv[argn], "-h", 2 ) == 0 &amp;&amp; argv[argn][2] != '\0' )
+ helo = &amp;(argv[argn][2]);
+ else if ( strncmp( argv[argn], "-u", 2 ) == 0 &amp;&amp; argv[argn][2] != '\0' )
+ user = &amp;(argv[argn][2]);
#ifdef DO_MINUS_SP
else if ( strncmp( argv[argn], "-s", 2 ) == 0 &amp;&amp; argv[argn][2] != '\0' )
server = &amp;(argv[argn][2]);
@@ -162,14 +169,22 @@
#endif /* DO_GETPWUID */
}

+ if ( user == (char*) 0 ){
+ user=username;
+ }
+
if ( gethostname( hostname, sizeof(hostname) - 1 ) &lt; 0 )
show_error( "gethostname" );

+ if ( helo == (char*) 0 ){
+ helo=username;
+ }
+
if ( fake_from == (char*) 0 )
- (void) snprintf( from, sizeof(from), "%s@%s", username, hostname );
+ (void) snprintf( from, sizeof(from), "%s@%s", user, helo );
else
if ( strchr( fake_from, '@' ) == (char*) 0 )
- (void) snprintf( from, sizeof(from), "%s@%s", fake_from, hostname );
+ (void) snprintf( from, sizeof(from), "%s@%s", fake_from, helo );
else
(void) snprintf( from, sizeof(from), "%s", fake_from );

@@ -181,7 +196,7 @@

message = slurp_message();
#ifdef DO_RECEIVED
- received = make_received( from, username, hostname );
+ received = make_received( from, user, helo );
#endif /* DO_RECEIVED */

(void) signal( SIGALRM, sigcatch );
@@ -209,7 +224,7 @@
exit( 1 );
}

- (void) snprintf( buf, sizeof(buf), "HELO %s", hostname );
+ (void) snprintf( buf, sizeof(buf), "HELO %s", helo );
send_command( buf );
status = read_response();
if ( status != 250 )
@@ -337,7 +352,7 @@
 #ifdef DO_RECEIVED
static char*
-make_received( char* from, char* username, char* hostname )
+make_received( char* from, char* user, char* helo )
{
int received_size;
char* received;
@@ -349,8 +364,8 @@
tmP = localtime( &amp;t );
(void) strftime( timestamp, sizeof(timestamp), "%a, %d %b %Y %T %Z", tmP );
received_size =
- 500 + strlen( from ) + strlen( hostname ) * 2 + strlen( VERSION ) +
- strlen( timestamp ) + strlen( username );
+ 500 + strlen( from ) + strlen( helo ) * 2 + strlen( VERSION ) +
+ strlen( timestamp ) + strlen( user );
received = (char*) malloc( received_size );
if ( received == (char*) 0 )
{
@@ -360,7 +375,7 @@
(void) snprintf(
received, received_size,
"Received: (from %s)\n\tby %s (%s);\n\t%s\n\t(sender %s@%s)\n",
- from, hostname, VERSION, timestamp, username, hostname );
+ from, helo, VERSION, timestamp, user, helo );
return received;
}
#endif /* DO_RECEIVED */

100

101

102

103

104

105

106

107

108

109

110

111

112

113

114

115

116

117

118

119

120

121

122

123

124

125

126

--- Makefile

+++ Makefile

@@ -7,10 +7,10 @@

BINDIR = /usr/local/sbin

MANDIR = /usr/local/man

CC = gcc

-CFLAGS = -O

-#CFLAGS = -g

-LDFLAGS = -s -static

-#LDFLAGS = -g -static

+#CFLAGS = -O

+CFLAGS = -g

+#LDFLAGS = -s -static

+LDFLAGS = -g -static

LDLIBS = $(SYSV_LIBS)

CC := $(DIET) $(CC)

--- mini_sendmail.c

+++ mini_sendmail.c

@@ -65,6 +65,8 @@

static char* argv0;

static char* fake_from;

static int parse_message, verbose;

+static char* helo;

+static char* user;

#ifdef DO_MINUS_SP

static char* server;

static short port;

@@ -80,7 +82,7 @@

static void usage( void );

static char* slurp_message( void );

#ifdef DO_RECEIVED

-static char* make_received( char* from, char* username, char* hostname );

+static char* make_received( char* from, char* user, char* helo );

#endif /* DO_RECEIVED */

static void parse_for_recipients( char* message );

static void add_recipient( char* recipient, int len );

@@ -111,6 +113,7 @@

argv0 = argv[0];

fake_from = (char*) 0;

parse_message = 0;

+ server = "localhost";

#ifdef DO_MINUS_SP

server = "127.0.0.1";

port = SMTP_PORT;

@@ -124,6 +127,10 @@

fake_from = &(argv[argn][2]);

else if ( strcmp( argv[argn], "-t" ) == 0 )

parse_message = 1;

+ else if ( strncmp( argv[argn], "-h", 2 ) == 0 && argv[argn][2] != '\0' )

+ helo = &(argv[argn][2]);

+ else if ( strncmp( argv[argn], "-u", 2 ) == 0 && argv[argn][2] != '\0' )

+ user = &(argv[argn][2]);

#ifdef DO_MINUS_SP

else if ( strncmp( argv[argn], "-s", 2 ) == 0 && argv[argn][2] != '\0' )

server = &(argv[argn][2]);

@@ -162,14 +169,22 @@

#endif /* DO_GETPWUID */

}

+ if ( user == (char*) 0 ){

+ user=username;

+ }

if ( gethostname( hostname, sizeof(hostname) - 1 ) < 0 )

show_error( "gethostname" );

+ if ( helo == (char*) 0 ){

+ helo=username;

+ }

if ( fake_from == (char*) 0 )

- (void) snprintf( from, sizeof(from), "%s@%s", username, hostname );

+ (void) snprintf( from, sizeof(from), "%s@%s", user, helo );

else

if ( strchr( fake_from, '@' ) == (char*) 0 )

- (void) snprintf( from, sizeof(from), "%s@%s", fake_from, hostname );

+ (void) snprintf( from, sizeof(from), "%s@%s", fake_from, helo );

else

(void) snprintf( from, sizeof(from), "%s", fake_from );

@@ -181,7 +196,7 @@

message = slurp_message();

#ifdef DO_RECEIVED

- received = make_received( from, username, hostname );

+ received = make_received( from, user, helo );

#endif /* DO_RECEIVED */

(void) signal( SIGALRM, sigcatch );

@@ -209,7 +224,7 @@

exit( 1 );

}

- (void) snprintf( buf, sizeof(buf), "HELO %s", hostname );

+ (void) snprintf( buf, sizeof(buf), "HELO %s", helo );

send_command( buf );

status = read_response();

if ( status != 250 )

@@ -337,7 +352,7 @@

#ifdef DO_RECEIVED

static char*

-make_received( char* from, char* username, char* hostname )

+make_received( char* from, char* user, char* helo )

{

int received_size;

char* received;

@@ -349,8 +364,8 @@

tmP = localtime( &t );

(void) strftime( timestamp, sizeof(timestamp), "%a, %d %b %Y %T %Z", tmP );

received_size =

- 500 + strlen( from ) + strlen( hostname ) * 2 + strlen( VERSION ) +

- strlen( timestamp ) + strlen( username );

+ 500 + strlen( from ) + strlen( helo ) * 2 + strlen( VERSION ) +

+ strlen( timestamp ) + strlen( user );

received = (char*) malloc( received_size );

if ( received == (char*) 0 )

{

@@ -360,7 +375,7 @@

(void) snprintf(

received, received_size,

"Received: (from %s)\n\tby %s (%s);\n\t%s\n\t(sender %s@%s)\n",

- from, hostname, VERSION, timestamp, username, hostname );

+ from, helo, VERSION, timestamp, user, helo );

return received;

}

#endif /* DO_RECEIVED */

Save it as some.patch. Move it inside mini_sendmail source directory and run:

patch -p0 < some.patch
make
cp mini_sendmail /to/jail/usr/bin/sendmail

patch -p0 < some.patch

make

cp mini_sendmail /to/jail/usr/bin/sendmail

You can specify username with -u and hostname (and HELO message) with -h parameter.

If you are going to use it with PHP, change sendmail_path in php.ini to something like this:

sendmail_path = /usr/bin/sendmail -s127.0.0.1 -p5555 -unoreply -hexample.com -fnoreply@example.com -t -i

1	sendmail_path = /usr/bin/sendmail -s127.0.0.1 -p5555 -unoreply -hexample.com -fnoreply@example.com -t -i

This should make php connect to sendmail running on 127.0.0.1 port 5555 and send example.com as HELO and noreply as username.

Patch was made for version 1.3.6.

Nginx un citi webserveri

Pirms daudziem gadiem hostēju savu blogu uz vecas Pentium 2 kastes ar, šķiet, 128MB RAM. Lieki piebilst, ka aparāts nebija no tiem ātrākajiem un varēja lieliski novērot cik ļoti prasīgas ir dažādas aplikācijas. Sākumā lietoju Windows (XP) + Apache + PHP+ MySQL. Lai uzlabotu ātrdarbību, pirmo reizi pamēģināju Linux (ja nemaldos, Slackware ar XFCE, jo terminālis bija kas svešs). Apache + PHP + MySQL uz Linux darbojās ievērojami labāk, Linux netērēja tik daudz resursus un biju priecīgs.

Vēlāk izmēģināju Lighttpd. Tas tērēja vēl mazāk resursus, bet man neiepatikās konfigurācijas sintakse un internetos sūdzējās, ka tam esot sūces, kuras, šķiet, pāris gadus nelaboja. (Pats nevienu memory leak nenovēroju.)

Vēl pēc kāda laika pamanīju, ka draugiem.lv un citas lielas lapas Lighttpd nomainīja uz Nginx. Uzreiz to neizmēģināju, jo uz jaudīgākām kastēm biju apmierināts ar Apache2, un tajā brīdī pieejamā dokumentācija bija pārsvarā krievu valodā (varbūt nemeklēju kārtīgi). Bet 2010. gada sākumā/vidū, beidzot, pienāca brīdis, kad uzliku to uz sava mājas rūtera. Nedaudz papētot tā iespējas, biju patīkami parsteigts. Tas ir ērtākais un foršākais webserveris/proxy kādu esmu līdz šim lietojis. Konfigurācija ir super ērta, ar daudzām iespējām (manurpāt, viens no lielākajiem Nginx plusiem), ir pieejama kaudze ar papildus moduļiem, tērē maz resursus un ir lielisks community support.

Izmēģinājis to uz savas kastes, uzliku uz vēl pāris serveriem un iesaku to arī jums. 🙂

Ja nemaldos, lielās Linux distribūcijas to piedāvā savos repozitorijos, bet es iesaku to nokompilēt pašiem, lai var pievienot kādu papildus moduli vai noņemt nost kādu no jums nevajadzīgajiem.

Kādu webserveri lietojat jūs un kāpēc tieši to?

Web panel for Uploader

Since some people prefer shiny user interfaces over command line, I have added web administration panel to auto uploader.

Since it is really early über pre-alpha version, it doesn’t have many options yet but it will change in time.

The idea is to create admin panel that allows you to manage Uploader using only your browser and to forget about terminal commands.

uploader-web-panel-alpha-version — Alpha version of web panel for Uploader

As you can see in the screenshot above, you can already search, remove, upload and redownload releases.

Web panel is built using PHP and jQuery. To make it work, make sure you have installed Sqlite3 for PHP (it comes installed by default from PHP 5.3.x). You should be able to install it on Debian/Ubuntu by using sudo apt-get install php5-sqlite.

It is available through SVN repository, inside trunk/web_panel/.