Hi Stephen.
Changing expiration type of an App Group is not supported.
However, you can build a new AG with the expiration Type of Document, then group it together with the other AG at the folder level. You can insert documents from ODWEK into the new AG, and continue to do server-side loads into the old AG.
Remember that Document expiration comes with a price -- incompatibility with certain features (I think Enhanced retention management is one of them) and the fact that running expiration (arsmaint -d) will take considerably longer than the Load or Segment expiration types, as CMOD needs to examine each row, and if that row is eligible for deletion, change the indexes to reflect that.
You might consider loading data into that 'new' AG, extracting the data once a day/week/month with arsdoc, and loading it into the existing AG, and clearing out the new one.
Let us know how you solve your issue!
-JD.