Zeoslib Portal

Posted: **09.11.2021, 09:17**

aehimself wrote: ↑08.11.2021, 21:08 @Jan, can you please check? As changes are not on GitHub yet I can not confirm, but something is still not in-sync it seems. I attach my latest version of ZMemTable.pas, can you please compare the differences?

Somehow LoadFomStream and SaveToStream were added to the sources twice. I added your file. GitHub should be in sync now.

Posted: **09.11.2021, 09:47**

Thank you Jan, as always :)

Posted: **09.11.2021, 13:32**

Hi aehimself,

I guess you're wrong about the ControlsCodePage.
See how your extended example works:

Code: Select all

uses 
...ZDatasetUtils...

procedure TForm1.Button12Click(Sender: TObject);
var ms:TMemoryStream;
begin
  ZMemTable2.ControlsCodePage:=cCP_UTF16;

  ZMemTable2.Close;
  ZMemTable2.FieldDefs.Clear;
  ZMemTable2.FieldDefs.Add('ID', ftInteger, 0, True);
  ZMemTable2.FieldDefs.Add('Name', ftString, 50);
  ZMemTable2.FieldDefs.Add('Test', ftWideMemo);
  ZMemTable2.Open;
  ZMemTable2.Insert;
  ZMemTable2.Fields[0].AsInteger:=1;
  ZMemTable2.Fields[1].AsString:='Kovács';
  ZMemTable2.Insert;
  ZMemTable2.Fields[0].AsInteger:=2;
  ZMemTable2.Fields[1].AsString:='Péter';
  ZMemTable2.Insert;
  ZMemTable2.Fields[0].AsInteger:=3;
  ZMemTable2.Fields[1].AsString:='Jakab';
  ZMemTable2.Insert;
  ZMemTable2.Fields[0].AsInteger:=4;
  ZMemTable2.Fields[1].AsString:='Gipsz';
  ZMemTable2.FieldByName('Test').AsString := 'Hello4';
  ZMemTable2.Post;
  ms := TMemoryStream.Create;
 Try
   ms.Position := 0;
   ZMemTable2.SaveToStream(ms);
   ZMemTable2.Close;
   ZMemTable2.ControlsCodePage:=cGET_ACP;
   ms.Position := 0;
   ZMemTable2.LoadFromStream(ms);
 Finally
   FreeAndNil(ms);
 End;

end;

Michał

Posted: **09.11.2021, 15:01**

@ Michal,

On my end it did not produce strange displays, but it actually revealed the Stream Read error. See, I was simply reading x bytes out of the stream. I didn't know that if you pass 0, it means to read out everything.
Whenever the code attempted to read 0 bytes, it read everything, putting AStream to EOF. Next read operation caused Stream read error, as there was nothing more to read.
I'll issue a pull request with the 0-length check in a minute.

Thank you for the test case!

Edit: Done.

Edit-edit: I got the "compression" to work! It managed to cut the data size of Michal's test case from 347 to 152. That's about 143% :D Unfortunately it only works with WITH_TVALUEBUFFER. I'm not that good in pointer magic to code the same. I'm sorry :(

Posted: **10.11.2021, 08:52**

Hi aehimself,

In your last version of ZMemTable, empty cells are filled with zeros for numeric fields and with data from an
adjacent cell for string fields(after SaveSoFile-LoadFromFile)
Michał

Posted: **10.11.2021, 13:31**

@Michal,

You mean the "compression" version?

Posted: **10.11.2021, 13:57**

Hi aehimself,

Yes. but I don't know if and not earlier. I am enclosing an example(D10.3.3-Win32).
At the top, the original at the bottom after clone-save-load.

dup1.png

Code: Select all

SET SQL DIALECT 3;

CREATE TABLE TEST_PERSON (
    ID          INTEGER,
    NAME        CHAR(20),
    SURNAME     CHAR(20),
    BIRTH_DATE  TIMESTAMP,
    NOTE        BLOB SUB_TYPE 0 SEGMENT SIZE 80
);

Michał

Posted: **10.11.2021, 14:03**

I found issues with the code not preserving .IsNull, which is already fixed on my local branch but I never met this value duplication. Can you be so kind to provide the fielddef creation / data input code again?

Edit:

I used this code to try to replicate the situation:

Code: Select all

 ZMemTable1.FieldDefs.Add('ID', ftInteger);
 ZMemTable1.FieldDefs.Add('NAME', ftString, 20);
 ZMemTable1.FieldDefs.Add('SURNAME', ftString, 20);
 ZMemTable1.FieldDefs.Add('BIRTH_DATE', ftDateTime);
 ZMemTable1.FieldDefs.Add('NOTE', ftWideMemo);

 ZMemTable1.Open;

 ZMemTable1.Append;
 ZMemTable1.FieldByName('ID').AsInteger := 2;
 ZMemTable1.FieldByName('NAME').AsString := 'Ola';
 ZMemTable1.FieldByName('SURNAME').AsString := 'Olecka';
 ZMemTable1.Post;

 ZMemTable1.Append;
 ZMemTable1.FieldByName('ID').AsInteger := 3;
 ZMemTable1.FieldByName('SURNAME').AsString := 'Alecka';
 ZMemTable1.FieldByName('BIRTH_DATE').AsDateTime := EncodeDate(2001, 1, 1);
 ZMemTable1.Post;

 ZMemTable1.Append;
 ZMemTable1.FieldByName('NAME').AsString := 'Wowa';
 ZMemTable1.Post;

After this, saving to stream and loading it back. There was a buffer corruption issue, causing ID 3 to change to something insane. After fixing that, saving and loading the stream produced the exact same results:

Capture.PNG

I'll wait for your test case before I check in my recent changes.

Posted: **10.11.2021, 15:07**

Hi,

After entering your table, the duplication of the string is gone but the numeric ones are still set to zero.

dup2.png

Michał

Posted: **10.11.2021, 15:15**

Checkin is available on my fork, included in the same pull request.

Posted: **10.11.2021, 15:25**

Hi aehimself,

It seems that with this version of yours from a moment ago, both bugs are gone.
But it keeps testing.

Michał

Posted: **10.11.2021, 15:51**

I'm happy to hear :) Please keep me informed if it works or not; especially LCL as I have no possibility to test that one.

If I feel brave enough I can try to implement the compression if the IDE does not support WITH_TVALUEBUFFER. Should be able to test it with my handy D7.

Posted: **10.11.2021, 18:40**

Hi aehimself,

There is one more bug with ftDate.
Wrong writes or reads from the stream.
Wrong writes or reads (0 probably).
You can see it in dbgrid.

Code: Select all

procedure TForm1.Button20Click(Sender: TObject);
var ms:TMemoryStream;
    cp:TZControlsCodePage;
begin

  ZMemTable2.Close;
  ZMemTable2.FieldDefs.Clear;
  ZMemTable2.FieldDefs.Add('DATA', ftDate);
  ZMemTable2.Open;
  ZMemTable2.Insert;
  ZMemTable2.Fields[0].value:='12/12/12';
  ZMemTable2.Insert;
  ZMemTable2.Fields[0].value:='11/11/11';
  ZMemTable2.Post;

  ms := TMemoryStream.Create;
 Try
   ms.Position := 0;
   ZMemTable2.SaveToStream(ms);
   ZMemTable2.Close;
   ms.Position := 0;
   ZMemTable2.LoadFromStream(ms);
 Finally
   FreeAndNil(ms);
 End;
end;

Michał

Posted: **10.11.2021, 20:31**

It's a .GetData bug, probably like at Blobs. Use ftDateTime instead, it doesn't seem to be affected. I'll look into it but I won't touch anything else than MemTable until Michael confirms if this behavior is normal or not.

Edit: Strange but no, it's not the same. ftDate, ftTime and ftDateTime are all coded down correctly, but it does not work.

ZAbstractRODataSet.pas : 2859 (TZAbstractRODataset.GetFieldData)

Code: Select all

                    else PDateTime(Buffer)^ := DT

DT correctly receives it's data but the assignment above does absolutely nothing to Buffer, it stays [0, 0, 0, 0]. The reason is, we are assigning a TDateTime (Double, which is 8 bytes) to a 4 byte buffer. 4 bytes, that is coming from TDateField.GetDataSize, which returns SizeOf(Integer).
This is actual memory corruption, so if Zeos is using DataSize and GetData internally this must be looked at.

Am am summoning Michael here. Are we missing an override in TZDateField...?

Edit-edit: Can be. Overriding TZDateField's (and TZTimeField's) GetDataSize to return this:

Code: Select all

function TZDateField.GetDataSize: Integer;
begin
  Result := SizeOf(TDateTime);
end;

does fix the issue. On the other hand I'm not sure how the internal design of TZFields work, so - as I mentioned - I'm not touching this :)

Posted: **10.11.2021, 20:44**

Hi aehimself,

I came across this by cloning zTable, so the data comes from the zeos table where it behaves normally.

Michał

Zeoslib Portal

zMemtable

Re: zMemtable

Re: zMemtable

Re: zMemtable

Re: zMemtable

Re: zMemtable

Re: zMemtable

Re: zMemtable

Re: zMemtable

Re: zMemtable

Re: zMemtable

Re: zMemtable

Re: zMemtable

Re: zMemtable

Re: zMemtable

Re: zMemtable