OnDemand User Group

Support Forums => Report Indexing => Topic started by: Justin Derrick on August 07, 2015, 05:14:28 AM

Title: How do you index reports with no index fields in them?
Post by: Justin Derrick on August 07, 2015, 05:14:28 AM

Hey folks.

I've got a report that's basically a pipe-delimited dump of a database table.  There's no usable header or index values inside it. 

I know I can pick up date fields from the file name, or write a script to build a generic index file, but I'm just wondering if anyone has any other methods for indexing reports without any usable fields inside them.

Thanks for your advice!  :)

-JD.

Title: Re: How do you index reports with no index fields in them?
Post by: Maciej Mieczakowski on August 10, 2015, 03:11:17 AM

It all depends how the data shall be accessible after all and what customer/end users want to achieve ;) I have been using such approaches:

 1. Create a one big generic index file (scripting) and load one data file (report) into system. Index file consist of multiple indexes that points to that single report. So many rows added into DB, one object file sent to TSM. It weren't big reports up to few hundreds, but works as expected for end users.

P.S Now in CMOD 9.5 it would be maybe even better to use full text search server instead as it gather all data from reports:) But it was not available in previous CMOD ver and I have not yet tested it :) Maybe someone who uses it can share its experience, as it might be a good tool

2. Modify report (if possible) and add a header if data is well organized (for instance 100). Sometimes it's easy to use sed, awk, regex or simply even vi to deal with it if you uses UNIX for CMOD. After such pre-processing let system index that.

3. Use index values that has nothing in common with a data that's planned to be loaded. That's the easiest approach. And as you mentioned I also used as indexes the values that were taken from data file name.

As segment date I uses a load date if possible, and let system fill it in while loading. By Adding default value in application for segment date field ('t') system catch a current date when data is processed. So there is not even a need to add this value to index file if generic index file is used.

All depends what customer want to achieve :)

Title: Re: How do you index reports with no index fields in them?
Post by: Justin Derrick on August 10, 2015, 08:37:44 AM

All items were excellent advice.  I like the idea of building a generic index for this data.

Thanks!

-JD.

Title: Re: How do you index reports with no index fields in them?
Post by: ewirtz on August 17, 2015, 12:45:40 AM

Hi Justin,
it can be done as well with an index exit. you need the wanted indexes and a pseudo index field (varchar) that contains the possible index fields with pipe delimiter. By parsing the varchar string you can provide the correct indexes. Your can minimize the overhead  by setting this filed to '' at the end.

regards

Egon

Title: Re: How do you index reports with no index fields in them?
Post by: jeffs42885 on August 17, 2015, 09:18:14 AM

Take one of the "pipes" and map it to a dummy field, and set something like report_date to t or something along those lines, I had to do that once with a pipe delimited file out of input accel.