Encoding scheme for digcap.dav

You need to inhibit or bypass the psh shell.
You are asking for the keys to Hikvision's kingdom.
To do that - you'd need to have penetrated Hikvision's layers of integrity and anti-tampering protection, to modify the underlying firmware.
 
That is totally incorrect. Only digicap.dav header is XOR'd in newer firmware. Files inside the dav are encrypted using 3DES-ECB. The keys are in the uImage.

Newer kernels provide /dev/hikded device and IOCTL interfaces used by /bin/ded to encrypt/decrypt files. Still, this does not protect anything from anything. You can still decrypt/repack/encrypt lzma files, update their MD5 checksums stored in new_10.bin and boot any firmware.

I've decrypted new_10.bin and tried to match the md5 checksums with files within the cramfs (app.tar.lzma, logo.tar.lzma, etc...) without success, any tips? :)
 
Did you match the MD5 values with the encrypted files from cramfs.img ? That's how it works.

I've performed MD5 on encrypted file e.g. logo.tar.lzma then checked if the checksum is within new_10.bin and couldn't find a match, do I need to decrypt the file, perform checksum and then match?
 
Assuming we are talking about 76xx NVR firmware here - it's only the encrypted files listed in the decrypted new_10.bin that have their MD5 values checked.
I've never seen a logo.tar.lzma listed in new_10.bin