Just my 2 cents for this "old" topic.
If you want to use your own viewer (not integrated), then you need to use the "User Define Type" and choose an extension for your mime type.
If you want to use the integrated viewer of the CMOD client, then you need to choose of the integrated Data Type choice (PDF, AFP, Line, TIFF, ...) but NOT the User Define choice.
Why it did work on 8.4 and not on 8.5 for you... I don't know, maybe an old bug that was corrected in 8.5 ?? don't know, just speculation !! :-D
Sincerely yours,
Alessandro