Segmentation fault while loading PDF using PDF indexer

[Anand]

Hi

After upgrading Ondemand server & PDF indexer component from 8.4.0.3 to 9.0.0.6 MP in AIX server, getting Segmentation fault while trying to load PDF files using PDF indexer. We are running Ondemand using root user and checked the ulimit. All values set for unlimited only.

Indexer process completing successfully and creating proper *.ind and *.out.

Issue only for PDF using PDF indexer, all other types (AFP, TEXT and PDF using generic indexer) are loading without issues.

Error msg:
ARS4315I Processing file >/home/drcoana/TEST.PDF<
ARS4334I Load Version <9.0.0.6>  Operating System <AIX> <6.1>  OS Userid <root>  Install Location </usr/lpp/ars/>
ARS4335I Server Version <9.0.0.6>  Operating System <AIX> <6.1>  Database <DB2> <09.07.0011>
ARS4302I 02/16/16 15:07:21 -- Indexing started, 66728 bytes to process
ARS4901I TRIGGER1=ul(5.99,0.99),lr(6.71,1.16),*,'FACTORPRO'
ARS4901I FIELD1=ul(0.99,0.99),lr(1.44,1.16),0
ARS4901I FIELD3=ul(1.34,1.49),lr(1.92,1.66),0
ARS4901I FIELD4=ul(7.38,0.99),lr(7.58,1.16),0
ARS4901I INDEX1='ReportID',FIELD1
ARS4901I INDEX3='ReportDate',FIELD3
ARS4901I INDEX4='pccode',FIELD4
ARS4901I INDEXSTARTBY=1
ARS4901I parmdd=./TEST.PDF.parm
ARS4901I inputdd=/home/drcoana/TEST.PDF
ARS4901I outputdd=./TEST.PDF.out
ARS4901I indexdd=./TEST.PDF.ind
ARS4901I resobjdd=./TEST.PDF.res
ARS4902I Number of input pages = 9
ARS4922I ARSPDOCI completed code 0
ARS4308I 02/16/16 15:07:21 Indexing completed
ARS4312I 02/16/16 15:07:21 -- Loading started, 65525 bytes to process
Segmentation fault

ARSLOAD Trace:

26345606:1 02/16/2016 15:07:21:840035 ERROR arsload.c(4793)ArcLOADP_LoadFile:Current state msg_buf=ARS4312I 02/16/16 15:07:21 -- Loading started, 65525 bytes to process
 rc=1
26345606:1 02/16/2016 15:07:21:840045 INFO arsload.c(4793)ArcLOADP_LoadFile:Current state msg_buf=ARS4312I 02/16/16 15:07:21 -- Loading started, 65525 bytes to process
 
26345606:1 02/16/2016 15:07:21:840050 FLOW arsload.c(768)ArcLOADP_StatusMsg1:Return
26345606:1 02/16/2016 15:07:21:840080 INFO arsload.c(5044)ArcLOADP_LoadFile:Load Started
26345606:1 02/16/2016 15:07:21:840087 FLOW arsload.c(905)ArcLOADP_Run:Enter
26345606:1 02/16/2016 15:07:21:840093 FLOW arsload.c(849)ArcLOADP_OSArgsInit:Enter
26345606:1 02/16/2016 15:07:21:840120 FLOW arsload.c(875)ArcLOADP_OSArgsInit:Return os_argv=4580849456
26345606:1 02/16/2016 15:07:21:840125 FLOW arsadmin.c(5035)ArcADMIN_MainRun:Enter
26345606:1 02/16/2016 15:07:21:840140 INFO arsadmin.c(5962)ArcADMIN_MainRun:Current state u_argv[1]=load title_str=/usr/lpp/ars/bin/arsadmin -s ./TEST.PDF.out -i ./TEST.PDF.ind -h localhost -u admin -g FMFPROOPS-STD -a FP4751SG-FMS -P 30957 -U 0 -Z  -x /home/drcoana/26345606.sysload
26345606:1 02/16/2016 15:07:21:840148 INFO arsadmin.c(5989)ArcADMIN_MainRun:Current state title_str=/usr/lpp/ars/bin/arsadmin -s ./TEST.PDF.out -i ./TEST.PDF.ind -h localhost -u admin -g FMFPROOPS-STD -a FP4751SG-FMS -P 30957 -U 0 -Z  -x /home/drcoana/26345606.sysload
26345606:1 02/16/2016 15:07:21:840194 FLOW arsadmin.c(2605)ArcADMINP_SetNodeId:Enter
26345606:1 02/16/2016 15:07:21:840202 FLOW arsadmin.c(2744)ArcADMINP_SetNodeId:Return rc=0
26345606:1 02/16/2016 15:07:21:840964 INFO arsadmin.c(7111)ArcADMIN_MainRun:Current state index_file=./TEST.PDF.ind resource_file=(null)


Need help to figure out the issue.

[Justin Derrick]

Looks like a bug to me.  There isn't any terribly useful messages as far as I can see.  Probably best to open a PMR.

[kbsiva]

Hi - Just to rule out everything. Did you check the PATH, LIBPATH, ARS_INSTALL_DIR etc? Also did you check if the ARS_TMP directory has permissions and not filled up?

Thanks,
Siva

[Anand]

kbsiva, Checked all the ENV variables. All set as recommended. Checked ARSTMP , HOME dir path permission and space. All okay.

Raised PMR case with IBM. Waiting for their reply.

[Anand]

We found out that for those AGs which we are getting core dump while loading, few metadata fields went missing (while seeing in Admin client and exporting the AG using ARSXML but available in ARSAGFLD table) after upgrade. Upon further analysis, we found that for those missing fields, the MASK value in ARSAGFLD table are having -ve value eg. -32756,-32766 etc.  MASK column is of SMALLINT type in 8.4.0.x version and converted to INT type on later versions. We suspect that the MASK values were altered during the upgrade casting the column from SMALLINT to INT. We provided the details to IBM for confirmations. Waiting for their reply. Anyone faced similar scenario while upgrading from 8.4.0.x version.