pefs crypto primitives (updated)

Supported data encryption algorithms: AES and Camellia (with 128, 192 and 256 bits key sizes). Adding another block cipher with 128 block size should be trivial.

File names are always encrypted using AES-128 in CBC mode with zero IV. Encrypted file name consists of a unique per file tweak, checksum and name itself:
XBase64(checksum || E(tweak || filename))

Checksum is VMAC of encrypted tweak and file name:
checksum = VMAC(E(tweak || filename))

Both checksum and tweak have 64 bit length.

Main reason for not providing alternatives to name encryption algorithm is to keep design simple. Data encryption is different from name encryption here: encrypted data, unlike encrypted file name, is not parsed in any way by pefs and user expects to be able to use secure/fast/best-name cipher.

Name has such structure to work around some of CBC shortcomings. Random tweak value is placed at the beginning of the first encrypted block. That gives us unique encrypted file names and eliminates the need of dealing with initial IV (IV is zero and name is padded with zeros).

Encrypt-then-Authenticate construction is used. In addition to being most secure variant it allows checking if the name was encrypted by the given key without performing decryption. VMAC was chosen because of it performance characteristics and its ability to produce 64 bit MAC (without truncation of original result like in HMAC case). 64 bit size is almost mandatory here because larger MAC would result in much larger file name and it can hardly improve security. But the real reason is that no real "authentication" performed. It's designed to be just a cryptographic checksum (sounds incorrect but I can't find a better wording), so that breaking VMAC wouldn't result in breaking encrypted data, besides name checksum doesn't authenticate encrypted data. Checksum's main purpose is to be able to find a key the file is encrypted with.

Encrypted directory/socket/device name also contains tweak but it's used solely to randomize first CBC block and keep name structure uniform.

Idea behind tweak is to get unique per file ciphertext. Block ciphers (AES, Camellia) operate in XTS mode. 64 bit tweak value concatenated with 64 bit file offset form tweak used by XTS. All encryption operations performed on 4096 bytes sectors ("block" in XTS notion). Incomplete sectors are also encrypted according to XTS standard. But encryption of sectors smaller then 128 bits is not defined for XTS, in such situation CTR mode is used with tweak value generated according to XTS. If full 4096 byte sector is zero (all 4096 bytes are zero) before decryption it is not decrypted and treated as hole is sparse file.

4 different keys are used for cryptographic operations: one for name encryption, one for VMAC and two keys for data encryption as required by XTS. These keys are derived from 512 bit user supplied key using HKDF algorithm based on HMAC-SHA512 (IETF draft). The kernel part expects cryptographically strong key from userspace. This key is generated with PBKDF on using HMAC-SHA512 from passphrase.

Standard implementations of ciphers are used, but I do not use opencrypto framework, so there is no hardware acceleration available. opencrypto is not used mainly because it lacks full support for XTS mode (OpenBSD version is not able to encrypt incomplete sectors). opencrypto is rather heavy weight (extra initialization and memory allocations) so using may even worsen performance (hardware initialization costs for encrypting short chunks with different keys).

Besides pefs supports multiply keys, mixing files encrypted with different keys in single directory, transparent(unencrypted) mode, key chaining (adding a series of keys by entering just one of them) and more. I'm going to write about it soon.

pefs benchmark

pefs is a stacked cryptographic filesystem for FreeBSD. It has started as a Goggle Summer of Code'2009.

I've just come across performance comparison of eCryptfs against plain ext4 filesystem on Ubuntu, benchmark I was going to perform on my own.

I run dbench benchmarks regularly while working on pefs. But use it mostly as a stress test tool. I haven't reached the point I can start working on improving performance yet. But measuring pefs overhead is going to be interesting.

Unfortunately I fail to interpret dbench results from the article. They've used dbench 4, while I'm using dbench 3 from ports. But never the less result of 4-8 Mb/s looks too strange for me.

I've benchmarked 4 and 16 dbench clients on zfs, pefs with salsa20 encryption (256 bit key) on top of same zfs partition and pefs with aes encryption (128 bit key, ctr mode). I executed benchmark for 3 times in each setup.

First of all, cipher throughput:
salsa20 ~205.5 Mb/s
aes128 ~81.3 Mb/s

Benchmark results:





In both cases (4 and 16 clients) CPU was limiting factor, disks where mostly idle. This explains such divergence in zfs results, I've actually benchmarked zfs arc cache performance. Because of unpredictable zfs inner workings one can get the best aes128 result surprisingly close to the worst salsa20 one (salsa20 is ~2.5 times faster than aes128).

The graph comparing average values:


Conclusion is that pefs is 2x times slower. But that shouldn't be solely because of encryption. From my previous testing I can conclude that it's mostly filesystem overhead:

• Current pefs implementation avoids data caching (to prevent double caching and restrain one's paranoia). I had version using buffer management for io (bread/bwrite) it's performance was awful, something like 20-30 Mb/s with salsa20 encryption.


• Sparse files (add file resizing here too) are implemented very poorly: it requires exclusive lock and fills gap with zeros. While this "gap" is likely to be filled by application really soon.


• Lookup operation is very expensive. It calls readdir and decrypts name for each directory entry.

eCryptfs IOzone benchmark also shows 2x difference

Layer2 dummynet

Haven't posted about progress with lyear2 filtering for a while. One notable improvement is addition of ethernet address masks to dummynet.

Just configure a pipe. New masks available: src-ether and dst-ether (and a shortcut for specifying both of them: ether)
# ipfw pipe 1 config bw 1Mb mask ether


And use it:
# ipfw add 1100 pipe 1 src-ether 00:11:11:11:11:11 dst-ether 00:22:22:22:22:22 out via bridge0 layer2
# ipfw add 1200 pipe 1 dst-ether 00:11:11:11:11:11 src-ether 00:22:22:22:22:22 out via bridge0 layer2


# ipfw pipe show
00001: 1.000 Mbit/s 0 ms 50 sl. 2 queues (64 buckets) droptail
mask: ff:ff:ff:ff:ff:ff -> ff:ff:ff:ff:ff:ff tag: 0x0000
BKT _Source Ether Addr_ _Dest. Ether Addr__ Tag Tot_pkt/bytes Pkt/Byte Drp
40 00:11:11:11:11:11 00:22:22:22:22:22 0 2 196 0 0 0
43 00:22:22:22:22:22 00:11:11:11:11:11 0 2 196 0 0 0

Besides, masking packet by tag is also there:
# ipfw add 200 pipe 1 ip from any to any tagged 1-1000 via bridge0 layer2

As several tags per packet supported, it is necessary to specify desired tag range, tag, or any tag:
# ipfw add 200 pipe 1 ip from any to any tagged any via bridge0 layer2

ipfw: layer2 lookup tables

I had an opportunity to spend some extra time improving layer2 filtering.
I've extended lookup tables in ipfw to support several layer2 addresses for a single layer3 address/mask. It means that it's possible to assign mac addresses to network (in case ip's are dynamically distributed by dhcp or whatever). Besides, wildcard ip address 'any' is supported, and entries with wildcard ip can be used for layer2 filtering.

For example:

ipfw table 1 add 192.168.1.0/24 ether 00:11:11:11:11:11
ipfw table 1 add 192.168.1.0/24 ether 00:22:22:22:22:22
ipfw table 1 add 192.168.1.0/24 ether 00:33:33:33:33:33

# equivalent to: ipfw table 2 add any ether ...
ipfw table 2 add ether 00:11:11:11:11:11
ipfw table 2 add ether 00:22:22:22:22:22
ipfw table 2 add ether 00:33:33:33:33:33
ipfw table 2 add ether ff:ff:ff:ff:ff:ff

ipfw add 1000 allow ip from 'table(2)' to 'table(2)' layer2

# layer3
ipfw add 2000 allow ip from 'table(1)' to 'table(1)'

Layer2 filtering with pf

Instead of trying to describe all the changes regarding layer2 filtering in pf I'd better provide some examples.

Ethernet address can be specified for host or interface name:
pass in on bridge0 from 10.0.0.1 ether 00:11:11:11:11:11 to 10.0.0.2 ether 00:22:22:22:22:22
pass in on bridge0 from ($int_if:network) ether 00:11:11:11:11:11 to any

Ethernet addresses are supported in table entries:
table <test> persist {10.0.0.1 ether 00:11:11:11:11:11, 10.0.0.2 ether 00:22:22:22:22:22}
pass on bridge0 from <test> to <test> keep state (ether)

Ethernet stateful filtering is handled specially. Per rule flag is added to conditionally enable ethernet stateful filtering (disabled by default):
pass log on bridge0 from <test> to <test> keep state (ether)

With keep state (ether) option enabled pf uses real source and destination ethernet addresses from the first packet to create the state and uses these addresses afterwards to match the state.

Filtering on bridge

There used to be a flaw in using ipfw on bridge interface. It's impossible to distinguish incoming packets on member interface from incoming packets on bridge itself. For example consider two rules:
add 1 allow ip from any to any in recv bridge
add 2 allow ip from any to any in recv member

First rule will never match. The logic is ok here (if you are aware of ipfw's handling of interface options). But what do you expect if you disable filtering on member interfaces and perform filtering on bridge only. You expect rule 1 to match all incoming packets on bridge. It gets extremely annoying when using stateful filtering.

First time I came across this issue several years ago. But didn't figure out how to fix it. At that time I've decided to switch to pf.

Actually ipfw is the only firewall that allow rules like
allow ip from any to any out recv if1 xmit if2
Such tricks are possible because ipfw gets input interface from mbuf of a packet. pf for example relies on pfil to provide interface.

I've added a hack into if_bridge to work around it. It contradicts traditional ipfw behaviour a little but seems to be much more useful. I think patches are useful enough and can be commited into FreeBSD:

perforce.freebsd.org/changeView.cgi?CH=143921
perforce.freebsd.org/changeView.cgi?CH=144238

Incompatibility and some new features

I've made some changes that break backward compatibility. But I've tried not to break anything intentionally but to do a cleanup work.

First of all most of sysctl's responsible for layer2 filtering were replaced by per interface flags.

net.link.ether.ipfw and net.link.bridge.ipfw are replaced by l2filter interface flag. So sysctl net.link.ether.ipfw=1 became ifconfig if1 l2tag.

net.link.bridge.ipfw_arp was renamed to net.link.bridge.pfil_layer2_arp

Introduced l2tag interface flag. It's purpose is to add mbuf tag containing source and destination layer2 addresses to every packet passing through interface. Note that l2tag filtering against layer2 addresses is performed in layer3.

When invoked from layer2 ipfw no longer touches layer2 headers. So they following rule won't work anymore:
ifpw allow ip from 10.1.1.1 to any src-ether 00:11:11:11:11:11 layer2

ipfw mac option was replaced by to two options: src-ether and dst-ether. ipfw still accepts mac option but translates it into src-ether and dst-ether.

Lookup tables support layer2 addresses now:
ipfw table 1 add 10.1.1.1 ether 00:11:11:11:11:11
ipfw allow ip from table(1) to any

ipfw mac-type was renamed to ether-type. Support for mac-type preserved.

Stateful filtering remains somewhat special. The problem here is that l2tag is added to a packet only in input path (when invoked from ether_demux). Such decision was intentional, mainly because it's impossible to get tag added in output path without serious layer violations or entire pfil framework and packet handling redesign. That's why a packet that has no l2tag attached, will pass against layer2 dynamic rule.

Dynamic rules (state created by the rule) do not check both source and destination layer2 address, but just the addresses specified by the rule created it. For example
ifpw allow ip from 10.1.1.1 to any src-ether 00:11:11:11:11:11 keep-state
will create dynamic rule that checks only source ethernet address of a packet, but not destination.