Thursday, August 17, 2017

Quicky secure Apache

LAMP stack which stands for Linux, Apache, MySQL/MariaDB and PHP/Python/Perl is a very popular model for serving websites.

This is my take on quickly and briefly apache on your server.

Securing Apache:

The default for Apache reveals a little too much about the server. Lets start by hiding some of that information, by editing:
sudo apt-get install libapache2-modsecurity
sudo vi /etc/apache2/conf-available/custom_security.conf
Paste the following:
ServerSignature Off
ServerTokens Prod 
TraceEnable Off
Options all -Indexes
Header unset ETag
Header always unset X-Powered-By
FileETag None
sudo a2enmod headers
sudo a2enconf custom_security.conf
sudo /etc/init.d/apache2 restart
Configuring mod_security:

sudo a2enmod security2
Configure the module and enable the OWASP ModSecurity Core Rule Set (CRS):
sudo mv /etc/modsecurity/modsecurity.conf-recommended /etc/modsecurity/modsecurity.conf

vi /etc/modsecurity/modsecurity.conf 
SecRuleEngine On
SecResponseBodyAccess Off
SecRequestBodyLimit 8388608
SecRequestBodyNoFilesLimit 131072
SecRequestBodyInMemoryLimit 262144
sudo rm -rf /usr/share/modsecurity-crs
sudo git clone /usr/share/modsecurity-crs
sudo mv /usr/share/modsecurity-crs/crs-setup.conf.example /usr/share/modsecurity-crs/crs-setup.conf
sudo vi /etc/apache2/mods-enabled/security2.conf
<IfModule security2_module>
  SecDataDir      /var/cache/modsecurity
  IncludeOptional /etc/modsecurity/*.conf
  IncludeOptional "/usr/share/modsecurity-crs/*.conf"
  IncludeOptional "/usr/share/modsecurity-crs/rules/*.conf
sudo /etc/init.d/apache2 restart
Be sure to keep an eye on the apache and mod_security logs but running:

sudo tail -f /var/log/apache2/*.log /var/log/mod_evasive/*
Automatic updates:

Last but not least, automatic updates. I know I said, this blog post is to secure apache, but one thing I am an advocate of, is secure your server. If you are a one man show, use unattended-upgrades.

Unattended-upgrades purpose is to keep the server current, up to date with the latest security (and other) updates automatically. This alone will allow you too sleep safe and tight at night. :)

sudo apt-get install unattended-upgrades
For extra security look at mod_evasive, and dont forget fail2ban.

Thursday, September 29, 2016

Puppetserver refuses to start with /tmp mounted noexec.

Im in the process of investigating and moving our code base to Puppet 4.

I came across something very interesting today, in that Puppet 4 does not like /tmp to be mounted noexec.

Now, we mount /tmp with nosuid and noexec as per the recommendation in the Securing Debian Manual.

After much hours of trouble shooting I came across "Puppet Server run issue when /tmp/ directory mounted noexec" (And I dont know why I didnt read this in the first place.)

As per the recommendation, you basically need to add to /etc/default/puppetserver.

JAVA_ARGS="-Xms2g -Xmx2g"



Tuesday, September 20, 2016

Puppetdb dies at random / silently

If you ever have it that the connection to puppet-db is refused or you find the daemon died.

Even if trawling through the puppetdb logs reveal nothing.

It might be that PuppetDB is running out of heap? Check
/var/log/puppetdb for the presence of the  file 'puppetdb-oom.hprof' for an indication that this is happening.

You may find the following to help.


Friday, August 26, 2016

Rate limiting with nginx

I just love Nginx, it's amazing at how little memory it consumes and how well it performs, much of this is owed to nginx’s use of the event driven mode.

One of my teams repeated problems that come up is the simple fact that some users and bots are a little excessive of the servers.

 http {

  limit_conn_zone  $binary_remote_addr zone=conn_limit_per_ip:50m;
  limit_req_zone   $binary_remote_addr zone=req_limit_per_ip:50m rate=1r/s;

    server {

        location / {

            limit_conn conn_limit_per_ip 10;
            limit_req zone=req_limit_per_ip burst=10 nodelay;




The directives that count here: limit_conn_zone, limit_req_zone, limit_conn and limit_req.

We first use limit_req_zone to set up at least a rate limit zone, which will then be enabled by placing them inside specific nginx location directives.

We start by setting up our first zone named ‘default’, give it 50 megabytes of memory to track our sessions, and set a rate at 1 request per second.

We then implement it in the ‘/’ location, and give it a ‘burst’ of 10.

Every time that a bot exceed the rate of 1 request per second, they have to pay a token.  Once they’ve spent all of their tokens, they are given an HTTP 503 error message. 

503 means  the server is currently unavailable (because it is overloaded or down for maintenance). Generally, this is a temporary state.

As you experience an excessive bot you will see the following in your log

2016/09/01 10:06:29 [error] 109154#109154: *42450 limiting requests, excess:
10.195 by zone "req_limit_per_ip", client: ip.of.attacher, server: default,
request: "GET
*42450 means:
This is a connection number, also available as $connection.
109154#109154 means:
This is nginx worker PID (also available as $pid) and thread identifier.
10.195 means:
This is number of requests accumulated in the bucket.  If this 
number is more than burst defined (10 in our case), further 
request will be rejected.
Number of requests in the bucket is reduced according to the rate 
defined and current time, and may not be integer.  The ".195" 
means that an additional request will be allowed in about 195 
milliseconds assuming rate 1r/s.

You can get more information at the following location(s) :


Monday, June 6, 2016

Another way to calculate PHP5-FPM max_children

This is a slight update from my previous post, but I stumbled across the following command  / method to help calculate the value of "pm.max_children" for PHP5-FPM.

echo "pm.max_children = $(( $(awk '/MemTotal:/ { printf "%d\n", ($2*0.66) }' /proc/meminfo) / $(ps --no-headers -o "rss,cmd" -C php5-fpm | awk '{ sum+=$1 } END { printf ("%d\n", sum/NR) }') ))"

Now whats interesting is that the suggested value return, is a ridiculously high value, but I think what comes out of this, is that it illustrates how much more RAM you can give  PHP5-FPM.

But (un)fortunately real world exists, and there are other existing services in play (e.g. MySQL could be installed).

So if you use the above command, I highly suggest you adjust the PHP5-FPM value sparingly, and thereafter recheck your Munin / Cacti etc


Adjusting PHP5-FPM child processes (Apache)

Every now and then, on a clients server, we get the following dreaded message:

WARNING: [pool www] seems busy (you may need to increase pm.start_servers, or pm.min/max_spare_servers), spawning 16 children, there are 4 idle, and 48 total children

So we will start off, by determining the non-swapped physical memory usage by each PHP5-FPM processes (notice its in kilo Bytes)

ps -ylC php5-fpm --sort:rss
ps --no-headers -o "rss,cmd" -C php5-fpm | awk '{ sum+=$1 } END { printf ("%d%s\n", sum/NR/1024,"M") }'

Thereafter I ran the following command to give me the total memory, of my current PHP5-FPM configuration.

ps -ylC php5-fpm --sort:rss | awk '!/RSS/ { s+=$8 } END { printf "%s\n", "Total memory used by PHP-FPM child processes: "; printf "%dM\n", s/1024 }'

In my case I still had plenty of RAM left.

Remember the appropriate value for pm.max_children can be calculated as:

pm.max_children = Total RAM of your server / Max child process size - in my case it was 67MB

pm.max_children = 70
pm.start_servers = 20
pm.min_spare_servers = 20
pm.max_spare_servers = 35
pm.max_requests = 500

Thursday, June 2, 2016

Debian upgrade Wheezy to Jessie

Good day Guys

Here is my tip / howto to upgrade Debian Wheezy to Jessie.

I highly recommend first ensuring that your server is update.

So run:

# apt-get update
# apt-get upgrade
# apt-get dist-upgrade 
# apt-get -f install
Fix any outstanding issues. 

If all is good. You can start by changing you sources.list to reflect Jessie
sed -i 's/wheezy/jessie/g' /etc/apt/sources.list
sed -i 's/wheezy/jessie/g' /etc/apt/sources.list.d/*
The second sed is to ensure all other third pary repos reflect Jessie too.
There after you are good to go.
# rm -rf /etc/apt/preferences.d/*
# apt-get update
# apt-get upgrade
# apt-get dist-upgrade
# apt-get -f install 

This can take awhile.

After the upgrade it is recommended to reboot your system: 
# reboot
After your server has come up, for safety sake, run another 

# apt-get update
# apt-get upgrade
# apt-get -f install
You should have a new and shiny new OS, and you can verify, by using either 
hostnamectl or 
lsb_release -a