Introduction This is a relatively challenging reverse engineering problem. Thankfully, the hint provided is very helpful. The link to the writeup for the challenge eBook DRM is https://ubcctf.github.io/2022/03/utctf-ebook-drm/. Strongly recommend a read as it is a very well written write up.
Initial Analysis Unzipping the challenge file, I was given 2 files. obsessive-checking and flag_book.txt.bin. Based the write-up above, I figured that this challenge also requires some way to proxy library functions to extract the key from obsessive-checking to decrypt the flag book file.