Applies to platform: All (especially ARM-based)
Updated: 30 August 2013
This lesson introduces some of the most effective methods to free space on the hard disk of your Endian Appliance. The situation of having a partition running out of space may hinder the correct working of any Endian UTM Appliance, hence this is a problem that should never be underestimated. While most Endian UTM Appliances on average do not suffer of this problematic very often, the new Mini Arm, which is a System on a Chip (SoC) embedded system, is more prone to this kind of situation. We show also some CLI commands and methods that help you identifying the problem.
Mini ARM Important Partitions
The /var directory is the place where all the log files, besides the configuration files, are stored. In particular, the /var/efw/ directory contains all the Appliance's configuration values and the /var/log/ directory contains the log files, produced by the services and daemons for which logging has been enabled. Hence, if there are many services running on your Endian UTM Appliance and for each of them logging is activated, the partition may quickly fill up, making the system slow and some service even unusable (for example the SMTP proxy, see below). Another reason for the /var partition to be filled up can be find in scripts that missed to delete old data from this directory. Each of these three directories is mounted on a different partition, so each can fill up independently of the other and thereforerequires a separate processing /e.g., if you free space in the /var/log/ directory, it is not true that also the /var directory will have more free space available.The most immediate check for the status of the partitions is to log in to the shell of your Endian UTM Appliance and issue the following command:
root@endian:~# df -h
Filesystem Size Used Avail Use% Mounted on
ubi0:rootfs 463M 189M 274M 41% /
/dev/mmcblk0p4 2.6G 248M 2.3G 10% /var
/dev/mmcblk0p2 97M 4.6M 88M 6% /var/efw
/dev/mmcblk0p1 291M 276M 15M 95% /var/log
tmpfs 252M 0 252M 0% /dev/shm
tmpfs 252M 8.0K 252M 1% /tmp
Note
Depending on which services are active on your Endian UTM Appliance, different strategies can be implemented to free space in the /var directory.
Remove old log files
Best practices suggest to always backup log files before cleaning them up.From the GUI, under Menubar > System > Backup you can create and download a new backup that includes log files and archives.If you prefer to proceed manually, you can copy the whole /var/log folder and its files via ssh with tools like scp, FileZilla or WinSCP. See this lesson for more information on the procedure.Once logs are safely stored, you can proceed in cleaning them up with these commands:root@endian:~ # DAYOLD=+366
root@endian:~ # /usr/bin/find /var/log/ -type f -mtime $DAYOLD | xargs /bin/rm -v
removed `/var/log/clamav/clamd.log-20120102.gz'
removed `/var/log/openvpn/openvpn.log-20120102.gz'
removed `/var/log/pyzord/pyzord.log-20120102.gz'
removed `/var/log/httpd/error_log-20120102.gz'
removed `/var/log/httpd/ssl_request_log-20120102.gz'
removed `/var/log/httpd/httpd_access-20120102.gz'
removed `/var/log/httpd/ssl_engine_log-20120102.gz'
[...]
You can review the list of files before removing them using the following command after setting the DAYOLD variable:
root@endian:~ # find /var/log/archives/ -type f -mtime $DAYOLD -exec ls -lh {} \;
Remove large (log) files
To check for the presence of large files under the /var or /var/log directories, issue the following command from the shell of your Endian UTM Appliance:
root@endian:~# du /var -hc --max-depth=4 | sort -rn | head -10
The output is the list of the ten directories which occupy more space on disk, ordered by size. You can now inspect these directories to check for files that you can delete, for example using the following two small scripts.
The first one combines the find and awk utilities:
root@endian:~# find . -type f -size +30000k -exec ls -lh {} \; | awk '{ print $9 ": " $5 }'
that searches in the current directory tree (i.e., the current directory and all its child directories) for files bigger than 30 megabytes. To search in a given directory, replace the . after find with the full path (e.g., find /var).
The second one retrieves the number of files that are saved under the root directory:
root@endian:~# cd / && for i in *; do echo -n "$i: "; find $i -type f | wc -l; done
To inspect a given directory tree, simply replace / with the path you wish to examine (e.g. cd /var/). Both the above scripts prove useful even in case of inspecting possible cases of inode consumption (see section below).
SMTP Proxy
In most cases the space in the /var/ directory can be filled up by the SMTP Proxy, when active and configured as spam filter. In some circumstances, for example after certain types of mail bomb/floods have been received and are inspected, the antispam daemon may fill up its temporary directories with the extracted mail contents or with quarantine files. When the temporary directory -by default /var/tmp/-is full the SMTP Proxy can not process any more e-mail and in practice stops working.
You can check whether this is the case by issuing the following command, though even the one in the previous section may be useful:
root@endian:~# du -sh /var/amavis/
The output of this command is the size of the /var/amavis directory. Repeat the same command for the /var/tmp/clamav and /var/amavis/virusmails/ which are most likely those containing a large number of processed files, possibly of a large size.
You can then remove all the files and directories therein, issuing
root@endian:~# rm -rf /var/amavis/virusmails/*
root@endian:~# rm -rf /var/amavis/amavis-*
root@endian:~# rm -rf /var/tmp/clamav-*
After you created some room for the SMTP Proxy to work, you should restart the service with:
root@endian:~# jobcontrol restart smtpscan
Older systems may not accept the above mentioned jobcontrol command: Only in that case, you should use the legacy restartsmtpscan script.
SARG and Inodes Consumption
SARG, the Squid Analysis Report Generator, is a tool that analyses, parses, and processes the HTTP proxy's logs to create reports showing various facts about the HTTP proxy's use. Since it can create a large number of small files, it can quickly fill up the space and the inodes of the /var/ partition. The solution in this case is to delete old SARG reports.
Inodes are special structures on a Unix/Linux filesystem, which carry various information and metadata about the files stored on disk. Each inode contains information for one file on the disk, so the presence of a huge number of files on a partition is an indicator of a large use of inodes too. When the number of available inodes of a partition is exhausted, no more files can be stored on that partition. This problems can be very subtle and difficult to tackle: A partition indeed, using the df -h command mentioned above, may look like having several mega- or gigabytes free, but the creation of a file on it is unsuccessful. The same df utility however proves useful in this case, in a slight different form:
root@endian:~# df -iThe output will be something like:
Filesystem Inodes IUsed IFree IUse% Mounted onWhen the inodes' use percentage (IUse) becomes very high, Linux can not store more files on the disk, though apparently there is room for them.
/dev/mmcblk0p4 344960 1603 343357 1% /var
/dev/mmcblk0p2 25600 128 25472 1% /var/efw
/dev/mmcblk0p1 76800 107 76693 1% /var/log
Very useful and functional.