Oracle Linux on Btrfs for the Raspberry Pi

Enterprise comes to the microserver. By Charles Fisher

Oracle Linux 7 has been released for the Raspberry Pi 3. The release packages Btrfs as the root filesystem on the UEK-branded Linux 4.14 Long Term Support (LTS) kernel. A bootable disk image with a minimal install is provided along with a standard ISO installer.

CentOS appears to support only the "Mustang" Applied Micro X-Gene for AArch64, and it provides the older AArch32 environment for all models of the Raspberry Pi. Oracle Linux is a compelling option among RPM distributions in supporting AArch64 for the Pi Model 3.

This is not to say that Oracle AArch64 Linux is without flaw, as Oracle warns that this is "a preview release and for development purposes only; Oracle suggests these not be used in production." The non-functional WiFi device is missing firmware and documentation, which Oracle admits was overlooked. No X11 graphics are included in the image, although you can install them. The eponymous database client (and server) are absent. Oracle has provided a previous example of orphaned software with its Linux for SPARC project, which was abandoned after two minor releases. There's no guarantee that this ARM version will not suffer the same fate, although Oracle has responded that "our eventual target is server class platforms". One possible hardware target is the Fujitsu A64FX, a new server processor that bundles 48 addressable AArch64 cores and 32GB of RAM on one die, asserted to be the "fastest server processor" that exists.

AArch64 on the Pi

You'll need a Raspberry Pi Model 3 to run Oracle Linux. The 3B+ is the best available device, and you should choose that over the predecessor Model 3B and all other previous models. Both Model 3 boards retain the (constraining) 1GB of RAM—a SODIMM socket would be far more practical. The newer board has a CPU that is 200MHz faster and a Gigabit-compatible Ethernet port (that is limited to 300Mbit due to the USB2 linkage that connects it). A Model A also exists, but it lacks many of the ports on the 3B. More important, the Model 3 platform introduces a 64-bit CPU.

ARM was very tardy to 64-bit addressing capabilities compared to other well known microprocessor families, announcing this extension in 2011. Intel's first attempt to migrate the x86 market to the abortive Itanium 64-bit architecture shipped in 2001, ultimately yielding to AMD64, which debuted in 2003. MIPS and SPARC made this transition much, much earlier (1991 and 1995, respectively).

Despite the late arrival, the ARM AArch64 CPU architecture is now the dominant mobile platform. All supported iPhones are now required to run it, and most modern Android devices have migrated to it. ARM has guarded a competitive edge in power efficiency, as vast changes in its instruction set ideology and implementation have allowed ARM to maintain its market leadership in mobile.

The Acorn/Advanced RISC Machine (ARM) began with the retroactively named AArch32 assembly and machine language that was designed by Furber and Wilson for the Archimedes desktop computer, where tremendous power efficiency "was a complete accident". Desktop performance remained an architectural focus for a decade after the birth of ARM.

Apple's decision to base the (failed) Newton on ARM opened a new market of mobile device applications that prompted a new instruction set for ultra-compact assembly—Thumb 1 and 2. These are distinct from AArch32 and AArch64, and they focus on code density and minimal footprint for mobile devices. Thumb 2 is a 16-/32-bit variable length instruction set, which is dynamically translated to AArch32. Many other ARM extensions exist, but Thumb appears to have both great flexibility and persistence across ARM implementations.

When mobile devices neared the 4Gb RAM limit of 32-bit ARM, the designers decided to break from the past. The primary mistakes in AArch32 have long been known:

Design errors, like having r15 as the program counter or making every instruction conditional, are problems for CPU architects rather than programmers, and it's no surprise that they disappeared in the 64-bit version of the ARM architecture. They must have made it awfully hard to implement superscalar, out-of-order execution.

The changes in AArch64 bring it much closer to the spirit of MIPS, the most notable of which are:

• Conditional Execution has been removed, facilitating out-of-order processing in multiple pipelines.
• The R15/PC register can now be manipulated only by a small number of jumping and branching instructions, vastly simplifying branch prediction.

These performance improvements, along with the increase of pointer sizes to 64-bits, come at the cost of code density—programs compiled for native AArch64 will be larger than the equivalent for AArch32. Despite these enhancements, the majority of Intel desktop processors of the last decade easily will beat the Pi in most benchmarks (but they will not do so with a 10-Watt power supply). I examine the code density impact of AArch64 in greater detail below.

Installation

Most Raspberry Pi users rely on flash memory, which comes in two grades. Multi-Level Cell (MLC) media is cheap and offers large amounts of storage, but it can decay very quickly (cells typically are destroyed after 5,000 write operations). Most retail flash media (SD cards, USB flash drives) are MLC, and will not have a long lifetime under high I/O usage, despite the "wear-leveling" electronics that attempt to distribute writes over the whole device evenly. Single-Level Cell (SLC) media is more expensive and offers smaller amounts of storage, but it drastically increases the number of write operations until cell failure (100,000). Both types of memory can be "rehabilitated" by heating them, but this is not feasible for memory in most plastic packaging. If you anticipate large amounts of I/O, plan to purchase the correct grade of flash.

One great benefit of the new Model 3 is the ability to boot from USB. A standard hard disk drive is now a boot option. SLC media is also more plentiful and inexpensive in the USB flash format than as microSD cards. Choose the format that fits your expected I/O usage.

Once you have selected and obtained your media, you're ready to download and uncompress the following file:

$ xz -dkv rpi3-ol7.6-image-20181116.img.xz
rpi3-ol7.6-image-20181116.img.xz (1/1)
  100 %   266.4 MiB / 5120.0 MiB = 0.052  55 MiB/s   1:32

The full size of the boot image is 5GB—your boot media must be at least this size:

$ ll rpi3-ol7.6-image-20181116.img*
-rw-r--r-- 1 root root 5368709120 Jan 13 18:52
rpi3-ol7.6-image-20181116.img
-rw-r--r-- 1 root root  279309592 Jan 13 18:52
rpi3-ol7.6-image-20181116.img.xz

Insert your boot media and ensure that it is detected, but not mounted:

# dmesg | tail -3
[  378.540649] mmc0: new high speed SDHC card at address 0002
[  378.544104] mmcblk0: mmc0:0002 00000 7.83 GiB 
[  378.548395]  mmcblk0: p1

Finally, write the image to the raw media:

# dd if=rpi3-ol7.6-image-20181116.img of=/dev/mmcblk0 bs=4M

Assuming there are no write errors, the media is now prepared to boot the Raspberry Pi Model 3.

Operation

Load the media into the Pi, connect an HDMI cable to a monitor, and attach an ethernet cable. After the power supply is connected, the Pi will boot (there is no power switch).

The Pi might fail to boot with older USB peripherals. When a well-worn IBM 89P8800 USB 1.1 keyboard was attached, the firmware issued the message Timeout poll on interrupt endpoint and refused to boot. Trying a slightly newer Lenovo 41A5248 keyboard, the boot proceeded but was greatly delayed. Both keyboards worked without error once the OS was running. It might be wise to boot initially without any non-essential USB hardware.

Once the boot is complete, a login: prompt should be displayed. The user root with the password oracle will allow you to select a new root password, then drop to Bash.

The /proc/cpuinfo file will list four processor cores with the following descriptions:

[root@rpi3 ~]# cat /proc/cpuinfo
processor   : 0 ... 1 ... 2 ... 3
BogoMIPS    : 38.40
Features    : fp asimd evtstrm crc32 cpuid
CPU implementer   : 0x41
CPU architecture: 8
CPU variant : 0x0
CPU part    : 0xd03
CPU revision      : 4

The root filesystem is on Btrfs, which is a change from the XFS that you normally see on the x86_64 (AMD64) version of Oracle/Red Hat/CentOS/Scientific Linux 7. The /boot filesystem is on EXT4, likely due to bootloader considerations:

[root@rpi3 ~]# mount | egrep 'btrfs|ext4'
/dev/mmcblk0p4 on / type btrfs
(rw,noatime,ssd,space_cache,subvolid=5,subvol=/)
/dev/mmcblk0p2 on /boot type ext4 (rw,noatime,data=ordered)

Observe also the ssd mount option above. Btrfs detected this option automatically and was previously unsafe with a "a negative impact on usability and lifetime" of the flash media, but it's now appropriate in the 4.14 kernel. Observe that the autodetected SSD mount option was not specified in /etc/fstab (I've removed the UUIDs and LABELs):

[root@rpi3 ~]# sed -r 's/^(UUID|LABEL)[^ ]*[ ]*//' /etc/fstab
#Generated by RootFS Build Factory
/boot/efi vfat noatime 0 0
/boot ext4 noatime 0 0
swap swap noatime 0 0
/ btrfs noatime 0 0
tmpfs   /tmp       tmpfs   rw,nodev,nosuid,size=128M    0 0

The root filesystem is on the p4 partition of the SD card:

[root@rpi3 ~]# fdisk -l
 
Disk /dev/mmcblk0: 7948 MB, 7948206080 bytes, 15523840 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk label type: dos
Disk identifier: 0x000164f6
 
        Device Boot    Start      End   Blocks   Id  System
/dev/mmcblk0p1          2048   526335   262144    c  W95 FAT32
 ↪(LBA)
/dev/mmcblk0p2        526336  1550335   512000   83  Linux
/dev/mmcblk0p3       1550336  2074623   262144   82  Linux 
 ↪swap / Solaris
/dev/mmcblk0p4       2074624  10463231  4194304   83  Linux

Note above that I'm running on an 8GB SD card, but the last third of the card is unused, as it does not lie in a partition. You can add the unused space to the root filesystem by first expanding the partition:

[root@rpi3 ~]# growpart /dev/mmcblk0 4
CHANGED: partition=4 start=2074624 old: size=8388608 
↪end=10463232 new: size=13449183,end=15523807

And then expanding the Btrfs filesystem into the newly allocated space:

[root@rpi3 ~]# btrfs filesystem resize max /
Resize '/' of 'max'

The root filesystem now occupies the rest of the flash device:

[root@rpi3 ~]# fdisk -l
 
Disk /dev/mmcblk0: 7948 MB, 7948206080 bytes, 15523840 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk label type: dos
Disk identifier: 0x000164f6
 
        Device Boot  Start      End   Blocks   Id  System
/dev/mmcblk0p1        2048   526335   262144    c  W95 FAT32
 ↪(LBA)
/dev/mmcblk0p2      526336  1550335   512000   83  Linux
/dev/mmcblk0p3     1550336  2074623   262144   82  Linux 
 ↪swap / Solaris
/dev/mmcblk0p4     2074624 15523806  6724591+  83  Linux

Btrfs is an extremely powerful filesystem, similar in capabilities to ZFS. It's capable of transparent compression, mirroring, error detection and it has self-healing capabilities. (Watch for a future article on in-depth Btrfs coverage.) Oracle Linux on the Raspberry Pi provides a useful learning environment for many new tools and features, and the addition of Btrfs is chief among them.

A number of missing utilities are present in minimal installs on x86_64. In no particular order of preference, some are ethtool, less, man and nmtui. Assuming internet connectivity to Oracle, a yum install man will bring in less as a dependency (and let you begin reading all the Btrfs manual pages). The yum whatprovides command is useful for searching the contents of uninstalled packages for a particular utility.

Busybox is an alternative to some of the native Oracle AArch64 packages. Those unfamiliar with Busybox might review my previous container article published in Linux Journal that details its use. The 1.28.1 release offers several ARM binaries of Busybox (listed below):

[root@rpi3 ~]# for x in busybox-arm*
               do ls -l $x; file $x; ./$x | head -1; done

-rwxr-xr-x 1 root root 1132724 Jan 10 17:23 busybox-armv5l
busybox-armv5l: ELF 32-bit LSB executable, ARM, version 1 
 ↪(SYSV)...
BusyBox v1.28.1 (2018-02-15 14:34:02 CET) multi-call binary.
-rwxr-xr-x 1 root root  836560 Jan 10 17:23 busybox-armv7m
busybox-armv7m: ELF 32-bit LSB shared object, ARM, version 1 
 ↪(SYSV)...
BusyBox v1.28.1 (2018-02-15 14:34:02 CET) multi-call binary.
-rwxr-xr-x 1 root root 1079156 Jan 10 17:23 busybox-armv7r
busybox-armv7r: ELF 32-bit LSB executable, ARM, version 1 
 ↪(SYSV)...
BusyBox v1.28.1 (2018-02-15 14:34:02 CET) multi-call binary.
-rwxr-xr-x 1 root root 1078504 Jan 10 17:23 busybox-armv8l
busybox-armv8l: ELF 32-bit LSB executable, ARM, version 1 
 ↪(SYSV)...
BusyBox v1.28.1 (2018-02-15 14:34:02 CET) multi-call binary.

Notice above that the busybox-armv7m binary is substantially smaller than all the rest. This binary is apparently composed of Thumb 2 code; the ARM 7 M and R architectures appear to exclude both AArch32 and AArch64: "ARMv7-M... No ARM instruction set support (Thumb only)." Thumb might explain the smaller size, but its use may come at some cost to performance.

Oracle includes an AArch64 compiler environment, but this isn't likely to be capable of emitting ARMv7-M code. Oracle doesn't provide a glibc.aarch32 or glibc.thumb2 package in the same way that it provides a glibc.i686 on AMD64, nor are there any 32-bit libraries in /usr/lib. ARM itself provides Thumb-capable GNU compilers, as do other sources. Using Thumb as a compiler target will conserve memory at the potential cost of performance. This might be a reasonable choice for standing dæmons that are not CPU-intensive.

One glaring lack in Oracle Linux on the Raspberry Pi is the missing WiFi device. The kernel dmesg has a clue to the problem:

brcmfmac: brcmf_fw_map_chip_to_name: using
        brcm/brcmfmac43455-sdio.bin for chip 0x004345(17221) 
         ↪rev 0x000006
usbcore: registered new interface driver brcmfmac
brcmfmac mmc1:0001:1: Direct firmware load for
                         brcm/brcmfmac43455-sdio.bin failed 
                          ↪with error -2

You can find one source of the missing firmware at this link, although you also can find it within Raspbian. Installing the firmware will cause a wlan0 device to appear, but all my attempts to configure it have failed. It doesn't appear to be functional in the current release, despite the brcmfmac kernel module:

[root@rpi3 ~]# cd /usr/lib/firmware/brcm/
[root@rpi3 ~]# ll brcmfmac43455*

-rw-r--r-- 1 root root 600487 Jan 14 19:33 
 ↪brcmfmac43455-sdio.bin
-rw-r--r-- 1 root root  14036 Jan 14 19:49 
 ↪brcmfmac43455-sdio.clm_blob
-rw-r--r-- 1 root root   2054 Jan 14 19:41 
 ↪brcmfmac43455-sdio.txt

It appears that the WiFi and Bluetooth devices on the Raspberry Pi work through the SD card's SDIO interface. Once those files are in place, reboot, and the WiFi driver should appear in the dmesg. Note this will use a small amount of additional memory:

mmc1: new high speed SDIO card at address 0001
brcmfmac: brcmf_fw_map_chip_to_name: using
        brcm/brcmfmac43455-sdio.bin for chip 0x004345(17221) 
         ↪rev 0x000006
brcmfmac: brcmf_c_preinit_dcmds: Firmware version = wl0:
      Feb 27 2018 03:15:32 version 7.45.154 (r684107 CY) 
       ↪FWID 01-4fbe0b04
Bluetooth: Generic Bluetooth SDIO driver ver 0.1

There is no mention at all of the WiFi hardware on the Raspberry Pi within Oracle's documentation on the AArch64 release, which Oracle claims was an oversight. This also appears to be an issue in CentOS, where it is at least discussed at some length.

Code Density

As 1GB of RAM included on the Pi is constraining, you should have some idea of the penalty AArch64 imposes.

Below is a script that I've used to size all the ELF binaries in Raspbian Linux running on the original Raspberry Pi, storing this in the file a32.txt:

for x in /bin/*
do [ -f "$x" ] &&
   case "$(file "$x")" in
        *ELF*) stat -c %n\ %s "$x";;
   esac
done > a32.txt

Moving this file to Oracle Linux running on the Raspberry Pi Model 3 B+, I run the following to find the size differences:

while read p s
do [ -f "$p" ] &&
   case "$(file "$p")" in
        *ELF*) echo $p $s $(stat -c %s "$p");;
   esac
done < a32.txt | awk '
    {a+=$2; b+=$3; print $1,$2,$3,$3/$2}
END {print a,b,b/a}' > a64.txt

For this small sample of 66 files, I found the results shown in Table 1.

Table 1. Results of Size Differences of 66 Files

These programs take nearly twice the space in Oracle Linux as they do in Raspbian. This somewhat explains the CentOS decision to remain on AArch32 with smaller 32-bit binaries. Oracle's pursuit of AArch64 is likely due to similar platforms that it supports or may support in the future.

Should Oracle elect to provide a Thumb2 development environment in the same way that it supports 32-bit x86, then Oracle could produce even smaller binaries than are found in Raspbian while still running a 64-bit kernel, at some cost to performance. This assumes that all target platforms support Thumb2; by all appearances, the Fujitsu A64FX does not.

It might be useful to examine commonly run server dæmons, system libraries and extract the text/data/bss segment sizes within all of these programs to see greater detail on the AArch64 penalty paid here. Those with large ARM deployments are encouraged to do so.

Conclusion

It's refreshing to have a new Linux distribution where legacy support is slashed in a way that never would be tolerated within Intel/AMD64 environments. There is substantial complexity and inertia in the maintenance of the systems of decades past.

Still, the relative silence in the documentation on questions of (the overlooked WiFi) hardware support and the legacy Thumb and AArch32 instruction sets is unsettling. Operating system vendors should be clear on what their products can and cannot do with the target hardware. While there are issues with Oracle AArch64 Linux where that clarity is lacking, it must be conceded that this is a pre-production release, and the desired clarity and supported server-grade AArch64 target platforms may not yet exist. To reiterate, one possible hardware target is the Fujitsu A64FX, which designers are asserting as the fastest processor in the world. Amazon also recently began running ARM workloads in its EC2 cloud with its Graviton processor, but Gravitons are not expected to outperform the Fujitsu A64FX, and the relationship between Amazon and Oracle is not warm. Oracle also may be developing its own AArch64 specifically tuned for Oracle's database. Oracle previously has maintained SPARC in this capacity, and continues to dominate the TPC benchmark with it; the company also may decide to do this with its own ARM processor.

On the subject of the Oracle Database, the absence of any discussion or mention of it also is a cause for substantial concern on the longevity of the platform.

In any case, Oracle AArch64 Linux likely will be used by many pursuing low-power, large memory applications. The Raspberry Pi may be able to provide a development environment for those larger systems. It's encouraging to see ARM move into the enterprise space, and the prospect of a legacy-free computing environment without all the problems (Meltdown), scandals (ME/PSP) and firmware concerns of Intel is quite refreshing.

Resources

• Oracle Linux for ARM Downloads
• Raspberry Pi (Wikipedia)
• Btrfs—the Next Generation Filesystem for Linux
• CentOS and AArch64
• Oracle Linux for Sparc Downloads
• "Fujitsu's A64FX ARM Chip Waves The HPC Banner High" by Timothy Prickett Morgan
• R4000 Microprocessor from 1991 (Wikipedia)
• UltraSPARC Microprocessor from 1995 (Wikipedia)
• "ARM Creators Sophie Wilson and Steve Furber, Part Two: the accidental chip" by Chris Bidmead, The Register
• Acorn Archimedes (Wikipedia)
• Apple Newton (Wikipedia)
• Thumb and Thumb-2 (Wikipedia)
• "RISC instruction sets I have known and disliked" by Jack Whitham
• Conditional Execution (Wikipedia)
• R15/PC register (Wikipedia)
• Types of Flash Memory (Wikipedia)
• "Making flash memory more reliable with 800°C heat pulses" by Mark Tyson
• Installation of Raspberry Pi 3 Image—Release Notes for Oracle Linux 7 Update 6 (aarch64)
• Btrfs Gotchas
• "ZFS for Linux" by Charles Fisher, LJ, March 2018
• BusyBox
• "Infinite BusyBox with systemd" by Charles Fisher, LJ, March 2015
• BusyBox 1.28.1 Release
• What are the differences between ARMv7-A, ARMv7-R and ARMv7-M? (arm Developer)
• GNU ARM Embedded Toolchain Downloads (arm Developer)
• armbian firmware
• SDIO cards (Wikipedia)
• CentOS on ARMv7hl boards
• The Raspberry Pi 3 B+ in Fedora (nullr0ute's blog)
• Fujitsu A64FX Post-K Supercomputer: World's Fastest ARM Processor (YouTube)
• "Amazon's homegrown 2.3GHz 64-bit Graviton processor was very nearly an AMD Arm CPU" by Chris Williams, The Register
• SPARC SuperCluster with T3-4 Servers

About the Author