Optimizing system performance of flash-based Linux systems

Here are a few tips to significantly improve interactive performance on basic flash-based devices running Linux, such as the Raspberry Pi, other ARM or x86-based development boards, simple netbooks, mini PCs and media boxes, and mobile device such as tablets. These tips apply primarily to Debian and derived distributions such as Raspbian, Ubuntu, Linux Mint, but in general terms apply to most Linux distributions or even Android to some extent.

Typical Linux distributions configured for HDD-based server applications, not optimized for flash-based systems

When the root filesystem used on the device is on a flash memory card or another type of simple flash memory (but not a higher performance recent SSD), the flash storage can quickly become an enormous performance bottleneck, especially given the fact that Linux distributions are typically configured out of the box to continuously access and write to the file system for all kind of logging activity and temporary files used by many applications. While this configuration is suitable for HDD-based PCs or servers, it is not all suitable for a flash-based device.

Possible measures to reduce and optimize flash disk access

As listed below, several types of optimization are possible, each of which can make the system significantly faster, and together they can make a difference of night and day, turning an unusable sluggish system in a fairly quick usable one.

• Reducing system logging activity. Out of the box, Linux distributions tend to be configured with full logging that produces a significant amount of ongoing write access the disk. A lot of data such as kernel messages is often generously logged into two or three different logs. Although logging has its purpose for diagnosing a system problem in a mission-critical system, this does usually not apply to a flash-based device so almost all logging can be safely disabled.
• Using ramdisk filesystems for temporary storage. Most of the time temporary files can easily be stored in RAM, avoiding the significant overhead of storing and modifying a temporary file on flash storage. This involves mounting /tmp and similar directories on a ramdisk (tmpfs), and coaxing applications to store their internal cache or temporary storage directories on a ramdisk. Of course, it is helpful if the the device has a reasonable amount of RAM (512MB is already sufficient for extensive use of ramdisks, while 1GB or more is convenient).
• In general when local cache storage of an application is configurable, an example of which is the web content cache used by a web browser such as Firefox, it can be helpful to eliminate local storage as much as possible (set the size to zero). While obviously more content being kept in RAM by the application of its own volition would be good (this may happen), reloading from the network (internet) is often preferable to the high overhead and bottlenecks caused by the continuous flash disk access for local cache storage.
• The filesystem used, mostly commonly ext4, can be extensively tweaked to provide much better performance on flash storage. Measures taken include using write-back modes with longer "sync" delays instead of ordered data modes with relatively short delays before writing to disk, resulting in much more effective write caching, which can take the edge out of a flash write access bottleneck. Another obvious trick is to eliminate unnecessary bookkeeping such as file access time (use of the " noatime" mount option), and other performance improvements such as forgoing entirely on features such as journalling and huge file support that cater for larger systems and maximal stability on externally powered systems such as PCs. On a battery-powered device, write-back mode/write caching can often be extended without a disproportionate decrease in reliability and stability.

The relevant configuration settings changes are described below. Most of this section has been copied from the recent netbook blog article. Note that superuser priviledges (for example, using sudo) are required for editing most of the configuration files mentioned and any command lines.

Reducing system logging activity

System logging is often configured to be pretty active out of the box with most Linux distributions. Much of the logging that takes place is not a requirement for a (often single-user) flash-based device and can be disabled without consequences. Although kernel logs can be useful, the dmesg command is also able to log kernel messages for the current boot. The system logger is usually rsyslog, the rules for which may be stored in the /etc/rsyslog.d/ directory (for example, in a file named 50-default.conf). Disabling most logging can be accomplished by commenting out the rules by putting a '#' in front of them, for example:

    #auth,authpriv.*        /var/log/auth.log
    #*.*;auth,authpriv.none -/var/log/syslog
    #cron.*                 /var/log/cron.log
    #daemon.*               -/var/log/daemon.log
    #kern.*                 -/var/log/kern.log
    #lpr.*                  -/var/log/lpr.log
    #mail.*                 -/var/log/mail.log
    #user.*                 -/var/log/user.log

Of course, for security purposes or in a multi-user system it might be preferable to keep some of these logs, such as auth.log. Some kernels can be very noisy due to frequent messages or non-fatal errors or warnings which can affect performance when logging is enabled. In this case, it seems reasonable to disable kernel logging via rsyslog because dmesg already produces a similar log.

Using ramdisks (tmpfs) for temporary files

It is quite easy to move directories where temporary files are stored (primarily /tmp) to a ramdisk, and it can make a significant difference in performance. The following lines added to /etc/fstab cause /tmp and /var/tmp to be stored in ramdisks:

    tmpfs    /tmp       tmpfs    defaults    0 0
    tmpfs    /var/tmp   tmpfs    defaults    0 0

Optimizing cache directories

Applications like browsers and window managers that use a disk cache may conform to the XDG Base Directory specification standard. In that case, the environment variable XDG_CACHE_HOME defines the directory where local temporary cache files are stored. By setting this variable to a ramdisk location, it is possible to significantly speed-up the performance of certain browsers that are otherwise affected by heavy writing to the disk-cache on the flash device. This can be accomplished by creating a new file in /etc/profile.d/, for example /etc/profile.d/xdg_cache_home.sh, that will be executed at the start of every shell.

    #!/bin/bash
    export XDG_CACHE_HOME="/dev/shm/.cache"

Note this may not affect the main internet web content cache with certain browsers (such as Firefox), speeding up other types of cached information used by Firefox instead, while it does cause the main internet content cache to be stored on the ramdisk in the case of the lightweight browser Midori. In the case of Firefox, it can be beneficial to reduce the internet cache as much as possible (down to 8MB or zero), since extra network access (as long as it fast enough and not associated with extra cost) likely to be faster than the constant writing to the internet content cache on the flash card that otherwise happens. In the case of Midori, the internet content cache directory on the ramdisk can build up in size, affecting free RAM, which can be fixed by instructing the browser to empty the internet cache on exit.

Optimizing the filesystem (ext4) using write-back mode and other settings

Resources exist on the web on how to improve filesystem performance with ext4. The following line in /etc/fstab illustrates a optimized set of mount options for the ext4 root filesystem that should make a big difference in performance (UUID=nnnn or /dev/sdXn is the partition device used, which depends on the system):

UUID=nnnn / ext4 noatime,journal_async_commit,data=writeback,barrier=0,nobh,errors=remount-ro 0 1

You should also change the physical flag for journal_data_writeback mode, stored in the filesystem itself:

    tune2fs -o journal_data_writeback /dev/sdXn

where sdX is the SD card device and sdXn is the partition where the filesystem is stored. These changes should improve performance a lot, allowing it to reach an acceptable level.

Although these filesystem options have the potential to jeopardize stability and recoverability somewhat in case of system crash or power interruption, when a device is battery-powered the risk is much less.

Disabling X Window System error logging

Finally, the X Window System maintains a logfile called .xsession-errors in your home directory that gets filled with warnings and errors messages from the X server. In some cases this log file can fill up quickly and affect system performance. To disable it, edit the file /etc/X11/Xsession and edit the relevant lines to look like this (somewhere in the middle of the file), In this case the log-generating line has been commented out and replaced with a one that routes messages to /dev/null:

    #exec >> "$ERRFILE" 2>&1
    exec >> /dev/null 2>&1

Conclusion

In summary, the configuration changes above, which are relative to standard configuration settings in typical Linux distributions, can help transform a flash-based Linux system from very slow, continuously stalling behaviour to a reasonably consistent fairly quick response, make it much more usable.

Sources: SmartLogic

Updated November 2, 2014 (spelling).

Project -- Revitalizing an old Asus Eee PC

I was recently given a disused Asus Eee PC model 701SD with a view to making it usable again, because it was very slow. The Eee PC 701SD comes with a Celeron M processor up to 900 MHz, 512 MB DDR2, 8GB of early (2008 era) SSD storage and a 7" 800x480 screen (although the netbook is physically larger than 7" with a big border including speakers around the screen), and Windows XP installed. The device has 802.11b/g WiFi, an Ethernet port and VGA output.

This Asus Eee PC model provides convenient hardware upgrade options

Although some Asus Eee PC models originally came with a version of Linux installed, this particular model (dating from about 2008) came with Windows XP installed on the internal early Phison 8GB SSD storage device. However, as I received it performance was slow probably because of severe speed limitations associated with the early SSD model. Also, limited RAM and the fact that the storage space was almost full contributed to bad performance.

One of the first things I did was to assess to what extent the hardware could be physically upgraded. There is a convenient panel on the bottom of the device that gives easy access to the DDR2 SO-DIMM module and the SSD storage device. I replaced the 512MB DDR2 module with a 1GB one (333 MHz DDR2-667), which should be a big boost.

However, I am not certain the memory is running at the optimal speed. The BIOS seems to configure it running at about 150 MHz (DDR-300 effective) which may reflect a power saving state. Asus includes a Hybrid Engine driver in Windows XP that may regulate the RAM frequency as well as the CPU frequency. Because of this, it is possible that RAM is stuck at a low speed when running Linux.

The internal SSD is easily removable and it looks like it is connected using some kind of IDE-like interface. I have read the device may use a CompactFlash-compatible interface so that installing a more recent CompactFlash storage device may significantly increase performance. There is supposed to be room and board space beyond the cover area to accommodate a fairly large device.

Windows XP slow, but somewhat improved after optimization

I flashed the BIOS to the lastest available version, and after cleaning up the XP installation by removing unnecessary applications and cleaning up temporary files, I updated most of the Windows device drivers for the various hardware devices. However, the drivers that are still listed on the Asus website do not seem to be the most recent versions in many cases. I was able to breath new life into the WiFi chip by downloading an updated driver from Realtek. Overall, the slowness of Windows XP seemed to be improved somewhat.

Linux Mint Mate seems a good match for a netbook

For running Linux, I picked Linux Mint 17 Mate. I have good experiences with the more demanding Cinnamon variant of Linux Mint, and although Cinnamon does not put high demands on hardware, the Mate desktop is supposed to be considerably more lightweight and does look and function well. The main drawback of Mate would be limitations caused by the continuing use of the GTK+ 2.x libraries instead of the current GTK+ 3.x, although I have yet to encounter such limitations on this system.

I believe there is little against porting desktops such as Mate to GTK+ 3.x, because the perceived heavy overhead associated with GTK+3/Gnome is much more associated with the Gnome desktop environment rather than the underlying low-level GTK+ 3.x libraries. I have in the past happily run GTK+ 3.x applications in a GTK+ 2.x desktop environment on a relatively slow ARM-based device with few repercussions for speed or memory use.

I installed Mint on an SD-card, which can conveniently be inserted into the netbook' s card reader. Although the BIOS can be instructed to directly boot from the SD card or an USB stick, I let the USB stick-based installer install grub on the boot record of the internal SSD, so that Linux and Windows XP are now selectable at boot without going into the BIOS, as long as the SD card with  Linux is kept inside the SD card slot.

Despite the netbook's relatively small 7" 800x480 screen, Linux Mint Mate looks pretty good, and menus generally fit on the screen after setting the DPI to the lower value of 80. Performance was already much better than under Windows XP.

Optimizations to reduce and speed up disk (write) access

However, system performance was clearly affected by delays associated with regular and excessive disk access. After some research on the web, I came with the following set of mount options for the ext4 root filesystem in order to improve performance (modified in /etc/fstab):

UUID=nnnn / ext4 noatime,journal_async_commit,data=writeback,barrier=0,nobh,errors=remount-ro 0 1

I also changed the physical flag for journal_data_writeback mode, stored in the filesystem itself:

    tune2fs -o journal_data_writeback /dev/sdXn

where sdX is the SD card device. These changes certainly seem to improve performance a lot, allowing it to reach an acceptable level, even though the used SD card (8GB standard Class 10 HC 8GB dating from 2013) is not very new or particularly fast.

Although these filesystem options have the potential to jeopardize stability and recoverability somewhat in case of system crash or power interruption, the fact that the device is battery-powered provides considerable insurance.

Using ramdisk for tmp directories

I moved the /tmp and /var/tmp directories to a ramdisk by adding the following lines to /etc/fstab:

    tmpfs    /tmp       tmpfs    defaults    0 0
    tmpfs    /var/tmp   tmpfs    defaults    0 0

Moving application cache directories to ramdisk

Applications like browsers and window managers that use a disk cache may conform to the XDG Base Directory Specification standard. In that case, the environment variable XDG_CACHE_HOME defines the directory where local temporary cache files are stored. By setting this variable to a ramdisk location, it is possible to significantly speed-up the performance of certain browsers that are otherwise affected by heavy writing to the disk-cache on the flash device. This can be accomplished by creating a new file in /etc/profile.d/, for example /etc/profile.d/xdg_cache_home.sh, that will be executed at the start of every shell.

    #!/bin/bash
    export XDG_CACHE_HOME="/dev/shm/.cache" 

Note this may not affect the main internet web content cache with certain browsers (such as Firefox), speeding up other types of cached information instead, while it does cause the main internet content cache to be stored on the ramdisk in the case of the lightweight browser Midori. In the case of Firefox, it can be beneficial to reduce the internet cache as much as possible (down to 8MB or zero), since extra network access (as long as it fast enough and not associated with extra cost) likely to be faster than the constant writing to the internet content cache on the flash card that otherwise happens. In the case of Midori, the internet cache directory on the ramdisk can build up in size, affecting free RAM, which can be fixed by instructing the browser to empty the internet cache on exit.

Reducing/eliminating system logging

System logging is also configured to be pretty active out of the box with Linux Mint 17 Mate. The system logger is rsyslog, and the rules it uses are stored at /etc/rsyslog.d/50-default.conf in Mint. Although Mint does not enforce synchronous log updates (the dash in front of the log file means syncing on update is omitted), several logs are still being kept. Although kernel logs can be useful, the dmesg command is also able to log kernel messages for the current boot. It seems dmesg itself also logs kernel messages in /var/log in addition to rsyslog. Disabling most logging can be accomplished by commenting out the rules by putting a '#' in front of them.

    #auth,authpriv.*        /var/log/auth.log
    #*.*;auth,authpriv.none -/var/log/syslog
    #cron.*                 /var/log/cron.log
    #daemon.*               -/var/log/daemon.log
    #kern.*                 -/var/log/kern.log
    #lpr.*                  -/var/log/lpr.log
    #mail.*                 -/var/log/mail.log
    #user.*                 -/var/log/user.log

Of course, for security purposes or in a multi-user system it might be preferable to keep some of these logs, such as auth.log. Some kernels can be very noisy due to frequent messages or non-fatal errors or warnings which can affect performance when logging is enabled. In this case, it seems reasonable to disable kernel logging via rsyslog because dmesg already produces a similar log.

Disabling X Window System error logging

Finally, the X Window System maintains a logfile called .xsession-errors in your home directory that gets filled with warnings and errors messages from the X server. In some cases this log file can fill up quickly and affect system performance. To disable it, edit the file /etc/X11/Xsession and edit the relevant lines to look like this (somewhere in the middle of the file), In this case the log-generating line has been commented out and replaced with a one that routes messages to /dev/null:

    #exec >> "$ERRFILE" 2>&1
    exec >> /dev/null 2>&1

Conclusion

In summary, the configuration changes above, which are relative to standard configuration settings in typical Linux distributions, can help transform a flash-based Linux system from very slow, continuously stalling behaviour to reasonably consistent fairly quick response, make it much more usable.

Sources: SmartLogic

Running Linux distributions from a flash card or stick -- customization required for good performance!

Linux on flash storage is ubiquitous

There are quite a few Linux-based systems running with the main (root) file system on cheap flash memory (not a full SSD), ranging from development boards to the highest-volume consumer mobile devices sold today.

Full Linux distributions being run on such devices include the popular Raspberry Pi educational development board, and there are communities dedicated to running full Linux distributions on other ARM-based developments boards or consumer devices such as tablets and media boxes.

Many consumer devices can be made to run full Linux, but tinkering often required

In fact, if a consumer device such as a cheap tablet has external interfaces such as a USB port and HDMI and the ability to boot an alternative OS after flashing the bootloader or directly from an SD card, it is probably possible to run a pretty much full-featured Linux desktop on it, including mouse, keyboard and a big PC monitor. Although this used to be relatively slow (especially for a full GUI environment), recent ARM SoCs have become faster and can provide a more pleasant experience.

However, on anything but a development board, this is not usually 'plug-and-play', and technical expertise and experimentation is often required, with the different ARM chip platforms being in different states of development regarding running full Linux on them, and development efforts for a typical platform being fragmented. The often widespread variation between devices using the same chip platform in the use of peripheral chips such as WiFi chips, use of different LCD screens, different RAM configurations and other factors add to the complexity. If a functional kernel with enough working device drivers is available, any typical Linux root filesystem such as something based on Debian compiled for ARM can be run on it.

Or course, billions of mobile devices such as smartphones and tablets running Android already run Linux-like systems (including an actual kernel) on cheap flash-based storage. There is also significant cross-over between Android kernel driver source code and device drivers for full Linux, with many kernels and drivers being directly usable or requiring limited adaptation.

Intel/AMD x86 systems too

Running Linux on a cheap or older x86 system using simple flash storage for the root filesystem is not uncommon either. This includes early netbooks such as the Asus Eee PC series, which can still be pretty usable when properly upgraded, configured and optimized with a lightweight Linux distribution. Additionally the cheapest and smallest x86 mini-PCs often have basic flash storage (although low-cost SSDs offer high performance for a higher price).

Optimization required and does wonders for performance

The fact that standard Linux distributions with their default configurations and settings have a strong legacy in full-featured hard disk drive-based systems, including heavy duty applications like servers, and therefore are not at all suited for running on a small, flash-based single user device in their default configuration, is often overlooked.

Even popular development boards such as the Raspberry Pi for a long time shipped with a Debian-based distribution that was still largely configured for heavier use, with extensive logging and temporary files and caches from applications being continuously stored and modified on the local flash disk (as they would on HDD-based systems), which is an obvious way to make a system very slow and unresponsive given the nature of flash drives (especially writing to a flash drive can be costly in terms of time taken, with excessive write access also being detrimental to the stability and lifetime of the flash memory).

Several major file system and OS configuration optimizations possible

As will be described in subsequent posts, it is not at all difficult to largely eliminate excessive flash disk access (especially writes), resulting in a dramatically better user experience and performance. Measures include largely eliminating system logging, clever use of RAM-disks for every possible kind of temporary storage, and configuring a Linux filesystem such as ext4 with settings such as write-back mode resulting in more effective write caching, eliminating unneccessary bookkeeping such as access time, and other tweaks such as forgoing on features such as journalling and huge file support that cater for larger systems and maximal stability on externally powered systems such as PCs.

In fact, because many of the devices mentioned are battery powered or can easily be modified to be battery powered, there is actually potential to run a fairly stable system even with heavy optimizations such as write-back settings and limited journalling that would normally impact stability on HDD-based desktop systems. Apart from a common filesystem like ext4, many older or new filesystems can in principle be configured to work well on flash memory.