File names are always encrypted using AES-128 in CBC mode with zero IV. Encrypted file name consists of a unique per file tweak, checksum and name itself:
XBase64(checksum || E(tweak || filename))
Checksum is VMAC of encrypted tweak and file name:
checksum = VMAC(E(tweak || filename))
Both checksum and tweak have 64 bit length.
Main reason for not providing alternatives to name encryption algorithm is to keep design simple. Data encryption is different from name encryption here: encrypted data, unlike encrypted file name, is not parsed in any way by pefs and user expects to be able to use secure/fast/best-name cipher.
Name has such structure to work around some of CBC shortcomings. Random tweak value is placed at the beginning of the first encrypted block. That gives us unique encrypted file names and eliminates the need of dealing with initial IV (IV is zero and name is padded with zeros).
Encrypt-then-Authenticate construction is used. In addition to being most secure variant it allows checking if the name was encrypted by the given key without performing decryption. VMAC was chosen because of it performance characteristics and its ability to produce 64 bit MAC (without truncation of original result like in HMAC case). 64 bit size is almost mandatory here because larger MAC would result in much larger file name and it can hardly improve security. But the real reason is that no real "authentication" performed. It's designed to be just a cryptographic checksum (sounds incorrect but I can't find a better wording), so that breaking VMAC wouldn't result in breaking encrypted data, besides name checksum doesn't authenticate encrypted data. Checksum's main purpose is to be able to find a key the file is encrypted with.
Encrypted directory/socket/device name also contains tweak but it's used solely to randomize first CBC block and keep name structure uniform.
Idea behind tweak is to get unique per file ciphertext. Block ciphers (AES, Camellia) operate in XTS mode. 64 bit tweak value concatenated with 64 bit file offset form tweak used by XTS. All encryption operations performed on 4096 bytes sectors ("block" in XTS notion). Incomplete sectors are also encrypted according to XTS standard. But encryption of sectors smaller then 128 bits is not defined for XTS, in such situation CTR mode is used with tweak value generated according to XTS. If full 4096 byte sector is zero (all 4096 bytes are zero) before decryption it is not decrypted and treated as hole is sparse file.
4 different keys are used for cryptographic operations: one for name encryption, one for VMAC and two keys for data encryption as required by XTS. These keys are derived from 512 bit user supplied key using HKDF algorithm based on HMAC-SHA512 (IETF draft). The kernel part expects cryptographically strong key from userspace. This key is generated with PBKDF on using HMAC-SHA512 from passphrase.
Standard implementations of ciphers are used, but I do not use opencrypto framework, so there is no hardware acceleration available. opencrypto is not used mainly because it lacks full support for XTS mode (OpenBSD version is not able to encrypt incomplete sectors). opencrypto is rather heavy weight (extra initialization and memory allocations) so using may even worsen performance (hardware initialization costs for encrypting short chunks with different keys).
Besides pefs supports multiply keys, mixing files encrypted with different keys in single directory, transparent(unencrypted) mode, key chaining (adding a series of keys by entering just one of them) and more. I'm going to write about it soon.