Out of memory with large count of records with BLOBs

[sglodek]

Hello,

I use ZEOSLib 7.2.6 stable in Delphi 2010 to create a report (FastReports Release 6) with about 1.000 ID-cards with a photography on the card. Database is Firebird 2.5. The report selects about 1.000 records each with a picture in a blob filed. The picture-size is about 250kbytes as .jpg-file, so the report / datasource will use 250MBytes for the pictures.

During report-creation there is an "out of memory" exception. I sent a small example-project to fast-report and they told me that the error occures in the datasource and not in the report-component. It crashes after loading abut 500 ID-cards.

The computer has 16GByte of memory and Delphi2010-applications (as 32bit-applications) should be able to handle 4GBytes.

Are there any special settings for ZConnection or ZQurey to handle queries with many records and contained large blobs?

I tried the following without any change in number of successfully loaded records:
- ZQuery-Options doCachedLobs
- ZQuery fetchRow with values of 10, 100, 500

The sql is like this (simlified here but there are only some more text-fields in the original query):
select lastname, picture from persons;

A Delphi TDatasource-component is linked to the TZQuery and the report uses this TDatasource to get the records from the database.

Do you need more informations to give me a hint for solving the problem?

Many thanks,

Siegbert

[marsupilami]

Hello Sigbert,

I may be wrong but with Firebird BLOBs should not be cached, when setting doCacedLobs to false. In this case, BLOBs should always be fetched from the database if you need them. There couold be some inprovements for this use case in the current SVN version of Zeos 7.2.
If this doesn't work out, you could try to see what happens if you do the job in a master - detail kind of way: Have e separate Query for the BLOB data and fetch the data for the current record in the AfterScroll event of TZQuery:

Code: Select all

procedure MasterQ.AfterScroll(DataSet: TDataSet);
begin
  BlobQ.Close;
  BlobQ.ParamByName('PersonID').Value := DataSet.FieldByName('PersonID').Value;
  BlobQ.Open;
end;

This should free the memory and only load the BLOBs you need.

Best regards,

Jan

[Fr0sT]

What happens if you just fetch all records in dataset without FR?
Do memory counters show correct values when you fetch 100, 200, 500 records without FR?
What memory counters show when you create report with maximal count of records that won't produce an error?

[aehimself]

I am using Zeos to fetch 10k+ records with XMLs stored as BLOBs, with often being 14MB+ each without memory issues. I'm compiling 64bit apps, though.

[EgonHugeist]

@aehimself,
you are using 7.3(halfed memory for all column except lob's. They are shared since i know Zeos), sigbert is using the last "stable" release of 7.2.
Sigbert, what do you think about an upgrade to 7.3 and report the findings?