Category: Ubuntu Server

Apache 2.4 Failed To Start AH00016: Configuration Failed

james 7 Comments

One of our Apache 2.4 web servers strangely stopped working and didn’t want to come back up. I tried to start it but it came back with a vague error message:

$ sudo service apache2 start
Starting web server
apache2 Action 'start' failed.
The Apache error log may have more information.

A quick look in /var/log/apache2/error.log just showed nothing useful:

$ tail var/log/apache2/error.log
AH00016: Configuration Failed, exiting

So, I ran the Apache configuration check tool on the configuration files but the syntax was okay.

$ sudo apache2ctl configtest
Syntax OK

This means that there isn’t anything wrong with the syntax of the Apache config and I generally find that this means either an SSL certificate isn’t installed incorrectly or there is something wrong with a more general configuration file. To find which host had configuration file problems – I used the strace command. This is really good because it shows all of Apache’s interactions with the file system. Therefore you can see the last file Apache opened before failing.

sudo apt-get update
sudo apt-get install strace
sudo strace -f apache2ctl start

The output from strace is quite large but you can clearly see where Apache failed.

.....
[pid  6068] munmap(0x7f45c6c38000, 4096) = 0
[pid  6068] write(11, "[Sun Oct 05 07:35:44.207805 2014"..., 189) = 189
[pid  6068] write(2, "AH00016: Configuration Failed\n", 30) = 30
[pid  6068] select(0, NULL, NULL, NULL, {0, 10000}) = 0 (Timeout)
.....

and just before this was this…

[pid  6068] munmap(0x7f45c6c38000, 4096) = 0
[pid  6068] open("/etc/ssl/private/a_website_ssl.key", O_RDONLY) = 45
[pid  6068] fstat(45, {st_mode=S_IFREG|0644, st_size=1704, ...}) = 0
[pid  6068] mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f45c6c38000
[pid  6068] read(45, "-----BEGIN PRIVATE KEY-----\nMIIE"..., 4096) = 1704
[pid  6068] close(45)

This enabled me to quickly idenfity which ssl certicate was causing the problems and disable the site which was using that key.

sudo a2dissite a_weebsite.conf

I hope that someone finds this useful.

How to install MySQL 5.6 on Ubuntu 14.04

james 6 Comments

Ubuntu 14.04 comes with both MySQL 5.5 and MySQL 5.6 and by default uses MySQL 5.5. However, you can upgrade to MySQL 5.6 by doing the following steps:

Step 1 – Take a backup!

mysqldump -u root --all-databases > /home/me/add_databases.sql

Step 2 – Remove the old MySQL
I recommend using ‘apt-get purge’ rather than ‘apt-get remove’ to uninstall the old MySQL server. Don’t worry, this doesn’t mean all your data in /var/lib/mysql will be deleted. The purge command just removes all the configuration files in /etc/mysql/ . This is important because some old MySQL configurations aren’t supported in 5.6.

If you have an unsupport configuration option (for example: table_cache was renamed table_open_cache) MySQl doesn’t silently ignore these settings… it simply doesn’t start. Oh and it doesn’t show any warnings. So you really should remove any list in /etc/mysqal/conf.d . You can read the list of settings that might have changed.

So…

apt-get purge mysql-server-5.5 mysql-client-5.5
apt-get autoremove

Step 3 – Install the new MySQL 5.6 

sudo apt-get install mysql-server-5.6 mysql-client-5.6

And you are done. Hurray.

p.s. For interest, you can see which repository these packages are in from using the ‘apt-cache policy’ command:

sudo apt-cache policy mysql-server

mysql-server:
  Installed: 5.5.38-0ubuntu0.14.04.1
  Candidate: 5.5.38-0ubuntu0.14.04.1
  Version table:
 *** 5.5.38-0ubuntu0.14.04.1 0
        500 http://archive.ubuntu.com/ubuntu/ trusty-updates/main amd64 Packages
        500 http://security.ubuntu.com/ubuntu/ trusty-security/main amd64 Packages
        100 /var/lib/dpkg/status
     5.5.35+dfsg-1ubuntu1 0
        500 http://archive.ubuntu.com/ubuntu/ trusty/main amd64 Packages

sudo apt-cache policy mysql-server-5.6

mysql-server-5.6:
  Installed: (none)
  Candidate: 5.6.17-0ubuntu0.14.04.1
  Version table:
     5.6.17-0ubuntu0.14.04.1 0
        500 http://archive.ubuntu.com/ubuntu/ trusty-updates/universe amd64 Packages
        500 http://security.ubuntu.com/ubuntu/ trusty-security/universe amd64 Packages
     5.6.16-1~exp1 0
        500 http://archive.ubuntu.com/ubuntu/ trusty/universe amd64 Packages

Kayako wget OpenSSL: error:140770FC:SSL routines:SSL23_GET_SERVER_HELLO:unknown protocol

james 3 Comments

We are using Kayako for our support ticket system. I just updated our server from Ubuntu 12.04 to Ubuntu 14.04. We started to see a problem with some crons which automate the importing and processing of emails.

This is the error that we got:

Resolving support.oursite.com (support.oursite.com)... 127.0.1.1
Connecting to support.oursite.com (support.oursite.com)|127.0.1.1|:443... connected.
OpenSSL: error:140770FC:SSL routines:SSL23_GET_SERVER_HELLO:unknown protocol
Unable to establish SSL connection.

There are several possible SSL problems that this could be:

1 – The destination site does not like the protocol – this could be caused but CURL / WGET not using a compatible version of SSL as the server. So you can try setting the version of ssl like so:

wget -secure-protocol=SSLv2 https://example.com
curl --sslv2 https://example.com

2 – The destination site does not like the cipher. This can be caused by a poor cipher configuration in Apache or anonymous ciphers is disabled on the server.

Finally – and this was my problem…

The php script that was making the request was located on the same server as the Kayako support ticket system. The url for the server was something like support.randomhacks.co.uk and the hostname of the server was also support.randomhacks.co.uk . So, when looking up the DNS record for support.randomhacks.co.uk the computer went to /etc/hosts and returned 127.0.0.1 rather than the public ip address. So, when the script tried to connect to Apache at 127.0.0.1 using https it failed because SSL certificates a limited to one ip address only and this was the public IP address.

I hope help solve you problem. I would be great to hear if you have another solution.
I hope this helps someone.

Kayako – Your PHP was not compiled with IMAP support. Ubuntu 14.04

james 6 Comments

If you have just upgrade Ubuntu from 12.04 to Ubuntu 14.04 and are running Kayako. You might have seens this bug:

Your PHP was not compiled with IMAP support.

The solution is to first install the php imap extension:

sudo apt-get install php5-imap

Then restart Apache using:

sudo apache2ctl restart

If you are still having the error then you need to explicitely enable this extension.

ln /etc/php5/conf.d/imap.ini /etc/php5/apache2/conf.d/20-imap.ini

Then finally restart Apache again:

sudo apache2ctl restart

Hope this help someone.

Bacula Error shell command: run AfterJob “/etc/bacula/scripts/delete_catalog_backup” Ubuntu 14.40

james 1 Comment

I have just upgraded from Ubuntu 12.04 to Ubuntu 14.04 on our Bacula backup server. Everything is backing up okay however it’s failing to delete the catalogue backup job after it has run. The error message is:

24-Apr 23:31 bacula-dir JobId 7388: shell command: run AfterJob "/etc/bacula/scripts/delete_catalog_backup"

It turns out that Ubuntu 14.04 updated the backup script /etc/bacula/scripts/delete_catalog_backup but didn’t make is excutable. I could see this by running:

ls -la /etc/bacula/scripts/

and saw:

-rw------- 1 root root   104 Apr 24 20:30 delete_catalog_backup
-rwxr-xr-x 1 root root   112 Sep  8  2012 delete_catalog_backup.ucf-old

To fix this you just need to run the following command:

sudo chmod 755 /etc/bacula/scripts/delete_catalog_backup

This should fix the issue but you can test this my:

sudo bconsole
run job=BackupCatalog"

Let me know if this works for you.