ata over ethernet
One quiet morning I took a look at the user space tools for ATA over Ethernet. It is a practical way to provide blocks from one system to another over Ethernet. It has been around for a long time. It does not require authentication and requires LAN security (so create physical networks/VLANs where needed).
We'll be working on a setup that looks a bit like this:
,----------, ,----------, ,----------+ server +--------------------------------+ ethernet | | `----------' ,-------+ switch | | | `-----+----' | | | | _.---'''''''''---._ | \ | =..__ __,.. ,-----+----, \ ,---------, | | `'''''''' | ,----------+ server | `-----+ client | +----+ sdb | | `----------' `---------' | +..__ __,.+ | | `'''''''' | | | _.---'''''''''---._ | _.---'''''''''---._ | =..__ __,.. | =..__ __,.. | | `'''''''' | | | `'''''''' | +----+ sdb | +----+ sdc | | +..__ __,.+ | +..__ __,.+ | `'''''''' | `'''''''' | | | _.---'''''''''---._ | _.---'''''''''---._ | =..__ __,.. | =..__ __,.. | | `'''''''' | | | `'''''''' | +----+ sdc | +----+ sdd | | +..__ __,.+ | +..__ __,.+ | `'''''''' | `'''''''' | | | _.---'''''''''---._ | _.---'''''''''---._ | =..__ __,.. | =..__ __,.. | | `'''''''' | | | `'''''''' | +----+ sdd | `----+ sde | | +..__ __,.+ +..__ __,.+ | `'''''''' `'''''''' | | _.---'''''''''---._ | =..__ __,.. | | `'''''''' | `----+ sde | +..__ __,.+ `''''''''
On your servers, you'll need vblade
and something to act as your block device, I was using real devices, but you can use what you want. In the following example I defined /dev/sdb /dev/sdc /dev/sdd /dev/sde as shelf 1, slots 0 and 1, then shelf 2 slots 0 and 1.
server# /usr/sbin/vbladed 1 0 eth0 /dev/sdb server# /usr/sbin/vbladed 1 1 eth0 /dev/sdc server# /usr/sbin/vbladed 2 0 eth0 /dev/sdd server# /usr/sbin/vbladed 2 1 eth0 /dev/sde
You could put this at the end of /etc/rc.local too, so it runs at system start if you prefer.
You'll need to duplicate this on your other server, if you wish to have two servers:
server# /usr/sbin/vbladed 3 0 eth0 /dev/sdb server# /usr/sbin/vbladed 3 1 eth0 /dev/sdc server# /usr/sbin/vbladed 4 0 eth0 /dev/sdd server# /usr/sbin/vbladed 4 1 eth0 /dev/sde
On the client system you'll need aoe-tools
, I did the following:
client# modprobe aoe client# aoe-discover client# aoe-stat
(It's worth putting aoe in /etc/modules
if you wnat
aoe-stat
will show what has been discovered, where eX.Y are the shelf and slot numbers.
The block devices will appear in /dev/etherd/eX.Y
and you're free to use these as any other block device, like sd
devices.
It's possible now to use these devices as part of software raid, to then provide the devices to LVM.
client# mdadm --create /dev/md0 --level=1 --raid-devices=2 /dev/etherd/e{1,3}.0 client# mdadm --create /dev/md1 --level=1 --raid-devices=2 /dev/etherd/e{1,3}.1 client# mdadm --create /dev/md2 --level=1 --raid-devices=2 /dev/etherd/e{2,4}.0 client# mdadm --create /dev/md3 --level=1 --raid-devices=2 /dev/etherd/e{4,4}.1 client# watch -n 1 cat /proc/mdstat
If you're running a single server then you'll need to use e{1,2} rather than {1..4}.
So, you can now watch the arrays build. What we've done is create a raid device across the virtual shelves, so should one shelf fail, then we can have redundancy on another. In reality, if you have two servers providing block storage, you'd balance over these. Use aoe-sancheck
to show if you have mismatched shelf numbers. Each server should have unique shelf numbers.
To setup an LVM device using these:
client# pvcreate /dev/md0 client# pvcreate /dev/md1 client# pvcreate /dev/md2 client# pvcreate /dev/md3
View the created physical volumes:
client# pvdisplay
Create the volume group
client# vgcreate ataoegroup /dev/md{0,1,2,3}
and vgdisplay
to see its status.
Create the logical volume
client# lvcreate -L4G -n ataoevol ataoegroup
Now create a FS on the device and mount it:
client# mkfs.ext3 /dev/ataoegroup/ataoevol && mount -t ext3 /dev/ataoegroup/ataoevol /mnt
Last modified: Sun, 21 Jun 2015 09:53:36 BST