To install FTS server or not?

[tjspencer2]

I am replacing a system that is all about text and line report data with my CMOD platform. 

I have only stored indexed PDF data in CMOD today as statements, and so just capturing this kind of data and report types is new to me.

Further, the way many of their users search their system is to select a list of reports and then do a full text search on a phrase across those reports. Yikes!

In short, I am wanting to understand what text searching I can do without an FTS Server and what is the line I step over that requires an FTS server to be installed.

As I appreciate CMOD there are some built-in full text search capabilities for searching text documents out of the box without FTS Server beyond just index values, But I don’t quite understand those just yet.

And as I understand those capabilities, they may only applicable when you have returned documents using an indexed search in a result list and then want to further search the results by full text in those documents.

How do I determine if I need FTS Server or not?

Thanks in advance!

[Justin Derrick]

It really just depends on how good or bad the performance is for your existing users, and what you think the future holds.

If you're doing server-side full text searching (where the CMOD server opens the documents, performs the search, then sends the results back) then you might also be concerned about the impact to the server if a 'rogue query' happens where a user unwittingly searches thousands or millions of documents.

If users are doing client-side searching, there might be a performance problem for end users, as each item is dragged across the network (and in the case of large objects, multiple round-trips to get all of the document segments).

All of that to say...  Maybe. 

You may want to try this out in a test environment first.  Load a bunch of data, and do some worst-case tests to see what the differences are in performance and server-side load.  If you're feeling especially nice, you can report back and let us know about your results. 

-JD.

[tjspencer2]

So I did some more research and determined that if users require searching non-indexed fields that I will definitely need FTS Server to search.  80% of the reports I will be ingesting are minus indexes and users are used to wild card searching text strings across a host of folders.

The question is, do I need to place the FTS Server on a separate AIX LPAR (in my example) with its own database or install it on my library server.  And I think your answer is that it depends on whether they’re doing client-side or server-side searching.

When you say server side versus client side, I’m not sure what you mean.  Are you calling a server side search one where I’ve installed FTS on my library server and a client side search one where I’ve installed an FTS server separately on separate hardware?

And when I do this and benchmark it, I will share the results :-)

[Justin Derrick]

Okay, let me start with explaining the difference between client-side search, and server-side search, and full text indexing ("FTI").

Client side search is available on the Thick Client, and retrieves the requested documents from the server, and performs the actual search using the local PC's CPU & memory.  This is often slow for reports that have large-object-support turned on, because the client has to make many (sometimes dozens) of requests against the server for EACH individual document it's searching.  The load on the server is relatively low.

Server Side Search sends the search request to the server, and the server itself retrieves the documents to a temporary directory on the server, expands them, and searches the contents for the string provided by the client.  This is usually faster than a client side search (because there's no network lag) consumes a LOT of CPU and memory on the server, especially if the user decides to search more than a few files.  If someone decided to search 1000 100,000 page documents, the server could be busy for hours.

Full Text Indexing is a separate process that reads each and every document in an application group, and builds an index of every word in the document.  When FTI searches are submitted, CMOD hands the query to the FTI server, then waits for it to return a list of documents that contain that phrase.

As for architecture -- again, that's up to you to determine based on how you do things...  If your CMOD server has lots of memory & CPU & storage, yes, the FTI can peacefully co-exist on your CMOD server.  It might make sense to move FTI to another platform where CPU/Memory/Storage is cheaper (Wintel/Lintel), or expanding capacity is easier (cloud-ish infrastrutcutre / VMs).

Hopefully that clears things up a little. 

-JD.

[tjspencer2]

Well of course - thick clients - being client side searching 

We're not using any thick clients - only IBM Content Navigator for searching and viewing here - so all our searches will be server side.

And of note, we have both internal and external customers being served up from this same platform

I for sure don't want to kill my external client statement retrievals by adding this new capability

I guess we'll just have to do some load testing and benchmarking and report back here with our results

Any other thoughts?

Any other thoughts?

[jsquizz]

my opinion- in the 6-7 CMOD installs ive seen ranging from all sizes, ive never seen it used.

[Lars Bencze]

Hi, this thread is old, but I do beleive that using the Folder setting "Maximum Hits" to limit the number of documents to, say, somewhere between 10 and 50 in those Folders where you want to enable the "Text Search" functionality would help prevent the worst performance penalties.
I'm sure the more experienced OnDemand guys in here can confirm or deny that this setting indeed sets the limit on the SQL/search field results, before the text search phase is initiated. Thus the number of documents that are actually opened and searched for text is kept low.

If anyone has a test environment, feel free to try it out - run a search that would return far more hits than the Maximum Hits setting, and include a Text Search value as well.

[Justin Derrick]

Lars, you're absolutely right.  But as always, there's a snag. 

If you limit end users to a relatively low number of search results, it may be impossible for them to find what they're looking for.  There's no good way to go to the 'next page' of results in CMOD when you're limited a query, that I know of.  (Might be possible through the API...)

-JD.