[patch_done] Delphi 2010 + ZEOS 7.0.0 + MySQL + cxGrid

[DoctorZLO]

that's a good choise but i don't know if they will help us.
I've uploaded one patch for a missing null-byte reservation but that patch will not solve your problem, i think. Rev. 1346

Today tested the 1359 version ... An error has remained ...

Do you have the same trouble if you use the normal DBGrid?

Everything works fine even if you open all 50 000 records... Does not work if you try to open a cxGrid or MemTableEh ... If the field type VarChar ...

Hmm where to start is a good question. I'm still thinking it is a memory problem. So open ZdbcCache again and goto:

Code: Select all

  Creates this object and assignes the main properties.
  @param ColumnsInfo a collection with column information.
%u7d
constructor TZRowAccessor.Create%u28ColumnsInfo%u3a TObjectList%u29;
var
  I%u3a Integer;
  Current%u3a TZColumnInfo;
begin
  FBuffer %u3a= nil;
  FColumnCount %u3a= ColumnsInfo.Count;
  FColumnsSize %u3a= 0;
  SetLength%u28FColumnNames, FColumnCount%u29;
  SetLength%u28FColumnCases, FColumnCount%u29;
  SetLength%u28FColumnTypes, FColumnCount%u29;
  SetLength%u28FColumnLengths, FColumnCount%u29;
  SetLength%u28FColumnOffsets, FColumnCount%u29;
  SetLength%u28FColumnDefaultExpressions, FColumnCount%u29;
  FHasBlobs %u3a= False;

  for I %u3a= 0 to FColumnCount - 1 do
  begin
    Current %u3a= TZColumnInfo%u28ColumnsInfo%u5bI%u5d%u29;
    FColumnNames%u5bI%u5d %u3a= Current.ColumnName;
    FColumnCases%u5bI%u5d %u3a= Current.CaseSensitive;
    FColumnTypes%u5bI%u5d %u3a= Current.ColumnType;
    FColumnLengths%u5bI%u5d %u3a= GetColumnSize%u28Current%u29;
    FColumnOffsets%u5bI%u5d %u3a= FColumnsSize;
    FColumnDefaultExpressions%u5bI%u5d %u3a= Current.DefaultExpression;
    Inc%u28FColumnsSize, FColumnLengths%u5bI%u5d + 1%u29;
    if FColumnsSize > SizeOf%u28TZByteArray%u29-1 then
      raise EZSQLException.Create%u28SRowBufferWidthExceeded%u29;
    FHasBlobs %u3a= FHasBlobs
      or %u28FColumnTypes%u5bI%u5d in %u5bstAsciiStream, stUnicodeStream, stBinaryStream%u5d%u29;
  end;
  FRowSize %u3a= FColumnsSize + RowHeaderSize;
end;

set a breakpoint on the line Inc(FColumnsSize, FColumnLengths + 1);

and check the FColumnsSize. An overload should normaly raise an exception.

This piece of code does not throw errors ... And here is an error!

Code: Select all

function TZAbstractRODataset.GetFieldData(Field: TField; Buffer: Pointer;
  NativeFormat: Boolean): Boolean;
begin
  try
  if Field.DataType in [ftWideString] then
    NativeFormat := True;
  Result := inherited GetFieldData(Field, Buffer, NativeFormat);
  except
    ShowMessage('FColumnsSize: ' + IntToStr(RowAccessor.ColumnsSize));
  end;
end;

I'm a little refined its value to get the FColumnsSize... Here is the image value of the variable.

Another possibility. Try to limit the selected rows like: select * from .. where .. limit 10). And check the issue.

If the limit is 100 or 500 records it works fine ... Did I mention that the error comes at the request of a large number of records ... With 20,000 entries error!

[EgonHugeist]

These are good answers and do underline my memory overload thoughts.

Ok the erreor doesn't happen with the DBGrid + all columns + all rows. And the error doesn't happen if you limt the rows or the columns. Now it's the question, what exactly happens. I don't believe that the function GetFieldData is the troublemaker.

So we have to exclude all possibillities step by step because i can not test it by my selves.

Try to use the TZReadOnlyQuery. That component needs only the half memory. Does it work now?

I have the impression that this cxGridand MemTable does double cache the values too which does the dbGrid not. Which means Zeos allocates the whole memory of the Columnsize+RowHeader x2 for the internal and cached ResultSets. The cxGrid/MemTableEh does that too eventually? I don't know. Which means the memory increases x3 or more because i don't now anything about how the two components behave. That you can watch if you open the taskmanager and check what happens with the used memory.

Maybe it would be helpfull if you add that thread as link to your request of the cx-company?

Michael

[ism]

Possible utf8 string is measured as ansi, with no ascii characters value can be in 2 times more

[EgonHugeist]

ism

You are right. That could be a possibilty. On the other hand: i know that for mysql three times mir memory well be allocated because one UTF8 char can have 1-3 bytes. But i don't know that for 100% sure yet. I hope that works for mysql. Where i not 100% sure is the point if that allocation is only done for the display-size or/and for the rowbuffer. That i've to check. In my mind there is a little bug on MySql. Only a display thing and that i'll ignore actually. Create a table with a varchar(100) field. If you open that table your displaysize is now 300. This was done to avoid exactly these issues on UTF8. But there are other charactersets like the "euc" encodeds (chinese, japanese..) which can have 1-6 bytes per char.

Let's strikeout each possibilty step by step.

Michael

[DoctorZLO]

Try to use the TZReadOnlyQuery. That component needs only the half memory. Does it work now?

TZReadOnlyQuery gives the same ...

I have the impression that this cxGridand MemTable does double cache the values too which does the dbGrid not. Which means Zeos allocates the whole memory of the Columnsize+RowHeader x2 for the internal and cached ResultSets. The cxGrid/MemTableEh does that too eventually? I don't know. Which means the memory increases x3 or more because i don't now anything about how the two components behave.

Maybe ... I can only say that the component MyDAC shows no errors ...

I will say that the project works fine on Delphi 2007 ... Where there is no unicode ...

That you can watch if you open the taskmanager and check what happens with the used memory.

Well I'll do it with three components: MyDAC, ZQuery and ZReadOnlyQuery...

All components before open table:


ZQuery after open table: (Opened with errors)


ZReadOnlyQuery after open table: (Opened with errors)


MyQuery after open table (Opened with no errors)


As you said, and memory consumption - a big ... In ZReadOnlyQuery and MyQuery the same flow but ZReadOnlyQuery an error ...

Maybe it would be helpfull if you add that thread as link to your request of the cx-company?

cx-company asked to create a new theme, I have created will now have to wait .. Link: http://www.devexpress.com/Support/Cente ... ls/Q406146

[EgonHugeist]

DoctorZLO,

Hmm no good answer for me personallly. I'll send you a private email..
Btw. the Dev express theme is private and we can't read it...

Michael

[DoctorZLO]

Replied to you ... Both get an answer tell me what I knew

[EgonHugeist]

DoctorZLO,

i hope i got it. Patch done Rev. 1377. Can you test this and tell me if that error is gone?

If that error is solved then i've to say that this component uses a differnt kind to read the string-data buffer than the other grid-components do...

Can you confirm this?

Michael

[miab3]

@EgonHugeist

Are you sure about this patch?

(1377) Length(AnsiString(RowAccessor.GetString(ColumnIndex, Result))));

It seems to me that for me it causes some strange effects, repeated insertions at the ends of text fields in Delphi XE2.

Michal Abramczyk

[EgonHugeist]

miab3,

oh can you tell me more about this? Does that mean there are chars left of another row?

Now if added Fillchar(Buffer, ..., #0) which overrides the Buffer completly. And this stupid component does not react viny nilly too.
Michal is that strange effect gone? Rev 1381

Michael

[miab3]

@EgonHugeist

Probably missed the # 0 at the end.
It is good but would not it be easier, faster way?

Michal

[EgonHugeist]

miab3,

hum that's a good question. Just download a trail of this DevExpress component and see what happens. I don't know why that component does react so terribly crazy. Normaly that should work with the old code too.
Imagine the RowBuffer is an Byte array. All byte where written sequential which means including the #0(ansi), #0#0#(wide) terminator. I can not say what is different here, because the normal dbGrid does not have that strange behavior.

I do not like this patch. I've tryed several modes. In my branch the dbc and component string is set to type String whatever String is a type from. So i need to boil him down to Ansi if the Field is a stString-type. Normaly we can write in the buffer what ever we want but that component has an different behavior here. I'll check this later on because i agree i dont like the length-check+Fillchar.

Either we are wrong with our way or that comoponent. But DoctorZLO writes that that with that MyDAC component all is working fine.

So let's see it as improvement until we know a better way to solve this issue...

Michael

[miab3]

@EgonHugeist

For me the extra characters at the end of the field shown in a simple DBGrid(in rev 1377).

Michal Abramczyk

[EgonHugeist]

@miab3,

what do you think about this?:

Code: Select all

        {$IFDEF DELPHI12_UP}
        ftString:
          begin
            //FillChar(Buffer^, RowAccessor.GetColumnDataSize(ColumnIndex), #0);
            //Ansi := AnsiString(RowAccessor.GetString(ColumnIndex, Result));
            //System.Move(PAnsiChar(Ansi)^, Buffer^, Length(Ansi));
            FillChar(Buffer^, Field.Size, 0);
            StrCopy(PAnsiChar(Buffer), PAnsiChar(AnsiString(RowAccessor.GetString(ColumnIndex, Result))));
            Result := not Result;
          end;
        {$ENDIF}

Both procedures are pure assemble code. And i reduced the FillChar to Field.Size which should be enough for me. Does that patch makes you happy and can you test it please? I do not want to run into the same "chars left" issue again..

It solves perfectly the posted issue too. Btw. it works too without the FillChar function. So can you test both designs?

Waiting for replay.

Michael

[miab3]

@EgonHugeist

I tested:

{$IFDEF DELPHI12_UP}
ftString:
begin
StrCopy(PAnsiChar(Buffer), PAnsiChar(AnsiString(RowAccessor.GetString(ColumnIndex, Result))));
Result := not Result;
end;
{$ENDIF}

and it seems that is well.

Michal