Somehow LoadFomStream and SaveToStream were added to the sources twice. I added your file. GitHub should be in sync now.
zMemtable
-
- Platinum Boarder
- Posts: 1939
- Joined: 17.01.2011, 14:17
Re: zMemtable
Re: zMemtable
Thank you Jan, as always :)
Delphi 12.1, Zeos 8 from latest GIT snapshot
Using:
- MySQL server 8.0.18; libmariadb.dll 3.3.8
- Oracle server 11.2.0, 12.1.0, 19.0.0; oci.dll 21.13
- MSSQL 2012, 2019; sybdb.dll FreeTDS_2435
- SQLite 3.45.2
Using:
- MySQL server 8.0.18; libmariadb.dll 3.3.8
- Oracle server 11.2.0, 12.1.0, 19.0.0; oci.dll 21.13
- MSSQL 2012, 2019; sybdb.dll FreeTDS_2435
- SQLite 3.45.2
Re: zMemtable
Hi aehimself,
I guess you're wrong about the ControlsCodePage.
See how your extended example works:
Michał
I guess you're wrong about the ControlsCodePage.
See how your extended example works:
Code: Select all
uses
...ZDatasetUtils...
procedure TForm1.Button12Click(Sender: TObject);
var ms:TMemoryStream;
begin
ZMemTable2.ControlsCodePage:=cCP_UTF16;
ZMemTable2.Close;
ZMemTable2.FieldDefs.Clear;
ZMemTable2.FieldDefs.Add('ID', ftInteger, 0, True);
ZMemTable2.FieldDefs.Add('Name', ftString, 50);
ZMemTable2.FieldDefs.Add('Test', ftWideMemo);
ZMemTable2.Open;
ZMemTable2.Insert;
ZMemTable2.Fields[0].AsInteger:=1;
ZMemTable2.Fields[1].AsString:='Kovács';
ZMemTable2.Insert;
ZMemTable2.Fields[0].AsInteger:=2;
ZMemTable2.Fields[1].AsString:='Péter';
ZMemTable2.Insert;
ZMemTable2.Fields[0].AsInteger:=3;
ZMemTable2.Fields[1].AsString:='Jakab';
ZMemTable2.Insert;
ZMemTable2.Fields[0].AsInteger:=4;
ZMemTable2.Fields[1].AsString:='Gipsz';
ZMemTable2.FieldByName('Test').AsString := 'Hello4';
ZMemTable2.Post;
ms := TMemoryStream.Create;
Try
ms.Position := 0;
ZMemTable2.SaveToStream(ms);
ZMemTable2.Close;
ZMemTable2.ControlsCodePage:=cGET_ACP;
ms.Position := 0;
ZMemTable2.LoadFromStream(ms);
Finally
FreeAndNil(ms);
End;
end;
Re: zMemtable
@ Michal,
On my end it did not produce strange displays, but it actually revealed the Stream Read error. See, I was simply reading x bytes out of the stream. I didn't know that if you pass 0, it means to read out everything.
Whenever the code attempted to read 0 bytes, it read everything, putting AStream to EOF. Next read operation caused Stream read error, as there was nothing more to read.
I'll issue a pull request with the 0-length check in a minute.
Thank you for the test case!
Edit: Done.
Edit-edit: I got the "compression" to work! It managed to cut the data size of Michal's test case from 347 to 152. That's about 143% :D Unfortunately it only works with WITH_TVALUEBUFFER. I'm not that good in pointer magic to code the same. I'm sorry :(
On my end it did not produce strange displays, but it actually revealed the Stream Read error. See, I was simply reading x bytes out of the stream. I didn't know that if you pass 0, it means to read out everything.
Whenever the code attempted to read 0 bytes, it read everything, putting AStream to EOF. Next read operation caused Stream read error, as there was nothing more to read.
I'll issue a pull request with the 0-length check in a minute.
Thank you for the test case!
Edit: Done.
Edit-edit: I got the "compression" to work! It managed to cut the data size of Michal's test case from 347 to 152. That's about 143% :D Unfortunately it only works with WITH_TVALUEBUFFER. I'm not that good in pointer magic to code the same. I'm sorry :(
Delphi 12.1, Zeos 8 from latest GIT snapshot
Using:
- MySQL server 8.0.18; libmariadb.dll 3.3.8
- Oracle server 11.2.0, 12.1.0, 19.0.0; oci.dll 21.13
- MSSQL 2012, 2019; sybdb.dll FreeTDS_2435
- SQLite 3.45.2
Using:
- MySQL server 8.0.18; libmariadb.dll 3.3.8
- Oracle server 11.2.0, 12.1.0, 19.0.0; oci.dll 21.13
- MSSQL 2012, 2019; sybdb.dll FreeTDS_2435
- SQLite 3.45.2
Re: zMemtable
Hi aehimself,
In your last version of ZMemTable, empty cells are filled with zeros for numeric fields and with data from an
adjacent cell for string fields(after SaveSoFile-LoadFromFile)
Michał
In your last version of ZMemTable, empty cells are filled with zeros for numeric fields and with data from an
adjacent cell for string fields(after SaveSoFile-LoadFromFile)
Michał
Re: zMemtable
@Michal,
You mean the "compression" version?
You mean the "compression" version?
Delphi 12.1, Zeos 8 from latest GIT snapshot
Using:
- MySQL server 8.0.18; libmariadb.dll 3.3.8
- Oracle server 11.2.0, 12.1.0, 19.0.0; oci.dll 21.13
- MSSQL 2012, 2019; sybdb.dll FreeTDS_2435
- SQLite 3.45.2
Using:
- MySQL server 8.0.18; libmariadb.dll 3.3.8
- Oracle server 11.2.0, 12.1.0, 19.0.0; oci.dll 21.13
- MSSQL 2012, 2019; sybdb.dll FreeTDS_2435
- SQLite 3.45.2
Re: zMemtable
Hi aehimself,
Yes. but I don't know if and not earlier. I am enclosing an example(D10.3.3-Win32).
At the top, the original at the bottom after clone-save-load.
Michał
Yes. but I don't know if and not earlier. I am enclosing an example(D10.3.3-Win32).
At the top, the original at the bottom after clone-save-load.
Code: Select all
SET SQL DIALECT 3;
CREATE TABLE TEST_PERSON (
ID INTEGER,
NAME CHAR(20),
SURNAME CHAR(20),
BIRTH_DATE TIMESTAMP,
NOTE BLOB SUB_TYPE 0 SEGMENT SIZE 80
);
You do not have the required permissions to view the files attached to this post.
Re: zMemtable
I found issues with the code not preserving .IsNull, which is already fixed on my local branch but I never met this value duplication. Can you be so kind to provide the fielddef creation / data input code again?
Edit:
I used this code to try to replicate the situation:
After this, saving to stream and loading it back. There was a buffer corruption issue, causing ID 3 to change to something insane. After fixing that, saving and loading the stream produced the exact same results:
I'll wait for your test case before I check in my recent changes.
Edit:
I used this code to try to replicate the situation:
Code: Select all
ZMemTable1.FieldDefs.Add('ID', ftInteger);
ZMemTable1.FieldDefs.Add('NAME', ftString, 20);
ZMemTable1.FieldDefs.Add('SURNAME', ftString, 20);
ZMemTable1.FieldDefs.Add('BIRTH_DATE', ftDateTime);
ZMemTable1.FieldDefs.Add('NOTE', ftWideMemo);
ZMemTable1.Open;
ZMemTable1.Append;
ZMemTable1.FieldByName('ID').AsInteger := 2;
ZMemTable1.FieldByName('NAME').AsString := 'Ola';
ZMemTable1.FieldByName('SURNAME').AsString := 'Olecka';
ZMemTable1.Post;
ZMemTable1.Append;
ZMemTable1.FieldByName('ID').AsInteger := 3;
ZMemTable1.FieldByName('SURNAME').AsString := 'Alecka';
ZMemTable1.FieldByName('BIRTH_DATE').AsDateTime := EncodeDate(2001, 1, 1);
ZMemTable1.Post;
ZMemTable1.Append;
ZMemTable1.FieldByName('NAME').AsString := 'Wowa';
ZMemTable1.Post;
I'll wait for your test case before I check in my recent changes.
You do not have the required permissions to view the files attached to this post.
Delphi 12.1, Zeos 8 from latest GIT snapshot
Using:
- MySQL server 8.0.18; libmariadb.dll 3.3.8
- Oracle server 11.2.0, 12.1.0, 19.0.0; oci.dll 21.13
- MSSQL 2012, 2019; sybdb.dll FreeTDS_2435
- SQLite 3.45.2
Using:
- MySQL server 8.0.18; libmariadb.dll 3.3.8
- Oracle server 11.2.0, 12.1.0, 19.0.0; oci.dll 21.13
- MSSQL 2012, 2019; sybdb.dll FreeTDS_2435
- SQLite 3.45.2
Re: zMemtable
Hi,
After entering your table, the duplication of the string is gone but the numeric ones are still set to zero.
Michał
After entering your table, the duplication of the string is gone but the numeric ones are still set to zero.
Michał
You do not have the required permissions to view the files attached to this post.
Re: zMemtable
Checkin is available on my fork, included in the same pull request.
Delphi 12.1, Zeos 8 from latest GIT snapshot
Using:
- MySQL server 8.0.18; libmariadb.dll 3.3.8
- Oracle server 11.2.0, 12.1.0, 19.0.0; oci.dll 21.13
- MSSQL 2012, 2019; sybdb.dll FreeTDS_2435
- SQLite 3.45.2
Using:
- MySQL server 8.0.18; libmariadb.dll 3.3.8
- Oracle server 11.2.0, 12.1.0, 19.0.0; oci.dll 21.13
- MSSQL 2012, 2019; sybdb.dll FreeTDS_2435
- SQLite 3.45.2
Re: zMemtable
Hi aehimself,
It seems that with this version of yours from a moment ago, both bugs are gone.
But it keeps testing.
Michał
It seems that with this version of yours from a moment ago, both bugs are gone.
But it keeps testing.
Michał
Re: zMemtable
I'm happy to hear :) Please keep me informed if it works or not; especially LCL as I have no possibility to test that one.
If I feel brave enough I can try to implement the compression if the IDE does not support WITH_TVALUEBUFFER. Should be able to test it with my handy D7.
If I feel brave enough I can try to implement the compression if the IDE does not support WITH_TVALUEBUFFER. Should be able to test it with my handy D7.
Delphi 12.1, Zeos 8 from latest GIT snapshot
Using:
- MySQL server 8.0.18; libmariadb.dll 3.3.8
- Oracle server 11.2.0, 12.1.0, 19.0.0; oci.dll 21.13
- MSSQL 2012, 2019; sybdb.dll FreeTDS_2435
- SQLite 3.45.2
Using:
- MySQL server 8.0.18; libmariadb.dll 3.3.8
- Oracle server 11.2.0, 12.1.0, 19.0.0; oci.dll 21.13
- MSSQL 2012, 2019; sybdb.dll FreeTDS_2435
- SQLite 3.45.2
Re: zMemtable
Hi aehimself,
There is one more bug with ftDate.
Wrong writes or reads from the stream.
Wrong writes or reads (0 probably).
You can see it in dbgrid.
Michał
There is one more bug with ftDate.
Wrong writes or reads from the stream.
Wrong writes or reads (0 probably).
You can see it in dbgrid.
Code: Select all
procedure TForm1.Button20Click(Sender: TObject);
var ms:TMemoryStream;
cp:TZControlsCodePage;
begin
ZMemTable2.Close;
ZMemTable2.FieldDefs.Clear;
ZMemTable2.FieldDefs.Add('DATA', ftDate);
ZMemTable2.Open;
ZMemTable2.Insert;
ZMemTable2.Fields[0].value:='12/12/12';
ZMemTable2.Insert;
ZMemTable2.Fields[0].value:='11/11/11';
ZMemTable2.Post;
ms := TMemoryStream.Create;
Try
ms.Position := 0;
ZMemTable2.SaveToStream(ms);
ZMemTable2.Close;
ms.Position := 0;
ZMemTable2.LoadFromStream(ms);
Finally
FreeAndNil(ms);
End;
end;
Re: zMemtable
It's a .GetData bug, probably like at Blobs. Use ftDateTime instead, it doesn't seem to be affected. I'll look into it but I won't touch anything else than MemTable until Michael confirms if this behavior is normal or not.
Edit: Strange but no, it's not the same. ftDate, ftTime and ftDateTime are all coded down correctly, but it does not work.
ZAbstractRODataSet.pas : 2859 (TZAbstractRODataset.GetFieldData)
DT correctly receives it's data but the assignment above does absolutely nothing to Buffer, it stays [0, 0, 0, 0]. The reason is, we are assigning a TDateTime (Double, which is 8 bytes) to a 4 byte buffer. 4 bytes, that is coming from TDateField.GetDataSize, which returns SizeOf(Integer).
This is actual memory corruption, so if Zeos is using DataSize and GetData internally this must be looked at.
Am am summoning Michael here. Are we missing an override in TZDateField...?
Edit-edit: Can be. Overriding TZDateField's (and TZTimeField's) GetDataSize to return this:
does fix the issue. On the other hand I'm not sure how the internal design of TZFields work, so - as I mentioned - I'm not touching this :)
Edit: Strange but no, it's not the same. ftDate, ftTime and ftDateTime are all coded down correctly, but it does not work.
ZAbstractRODataSet.pas : 2859 (TZAbstractRODataset.GetFieldData)
Code: Select all
else PDateTime(Buffer)^ := DT
This is actual memory corruption, so if Zeos is using DataSize and GetData internally this must be looked at.
Am am summoning Michael here. Are we missing an override in TZDateField...?
Edit-edit: Can be. Overriding TZDateField's (and TZTimeField's) GetDataSize to return this:
Code: Select all
function TZDateField.GetDataSize: Integer;
begin
Result := SizeOf(TDateTime);
end;
Last edited by aehimself on 10.11.2021, 21:08, edited 3 times in total.
Delphi 12.1, Zeos 8 from latest GIT snapshot
Using:
- MySQL server 8.0.18; libmariadb.dll 3.3.8
- Oracle server 11.2.0, 12.1.0, 19.0.0; oci.dll 21.13
- MSSQL 2012, 2019; sybdb.dll FreeTDS_2435
- SQLite 3.45.2
Using:
- MySQL server 8.0.18; libmariadb.dll 3.3.8
- Oracle server 11.2.0, 12.1.0, 19.0.0; oci.dll 21.13
- MSSQL 2012, 2019; sybdb.dll FreeTDS_2435
- SQLite 3.45.2
Re: zMemtable
Hi aehimself,
I came across this by cloning zTable, so the data comes from the zeos table where it behaves normally.
Michał
I came across this by cloning zTable, so the data comes from the zeos table where it behaves normally.
Michał