zMemtable

[aehimself]

This is odd, my code works fine even on Delphi 7:

Code: Select all

 fs := TFileStream.Create('C:\LocalWork\a.bin', fmCreate);
 Try
  ZMemTable1.SaveToStream(fs);
 Finally
  FreeAndNil(fs);
 End;

 ZMemTable1.Close;

 fs := TFileStream.Create('C:\LocalWork\a.bin', fmOpenRead);
 Try
  ZMemTable1.LoadFromStream(fs);
 Finally
  FreeAndNil(fs);
 End;

Capture.PNG

I got 'Access violation' error when trying to SaveToStream. I think the problem is due to the statement (Self.Fields[a] As TBlobField).SaveToStream(ms) -- I got error when the field type is ftMemo.

Please check, TMemoField should be a child of TBlobField. Otherwise the code wouldn't even end up in that branch; as there should be a check like this before:

Code: Select all

If Self.Fields[b] Is TBlobField Then [...]

@Jan,
It would help to be able to debug these somewhere. I'm a bit confused, as if the code works on D7, it should really work on D2006 as well. I'd personally prefer the VPN solution - easy to implement and provides considerable security.
As for the format I implemented an import - export functionality in my DB client not that long ago. Currently binary, TSV and CSV is supported, but what I wanted to say is; if we know the format, it can be done.
Also, I sent an other pull request with the latest version of my ZMemTable.pas; which was tested on D7. Since GitHub sync is off, the first two commits are included but they can be ignored :)

It seems to compile in D2006 and DX2 but the SaveToFile file is 2 times larger and with LoadFromFile it gets 'Stream read error'.

The size - unfortunately it's out of my control. The issue is, if a string field is set to have the size of 200 and it contains only a letter 'a', GetData returns the full 200 bytes.
There's a commented out section in the code which strips the trailing zeroes to reduce size but it messes something up in numeric fields.
Stream Read error usually means that there is nothing else to read. Are you sure that the stream is reset to position 0 before LoadFromStream is called? Can you reproduce the issue with TMemoryStream?

[miab3]

Hi aehimself,

Still "Stream read error" reading from disk Delphi XE2-Win32.
I use this method:
I am connecting to a database (right now Firebird) with a lot of different tables.
It clones them sequentially from ZTable to ZMemTable, writing to and reading from disk.

As for the size, the table that was written to the disk in your old version in D10.3.3-Win32
and occupied 13MB now in DXE2-Win32 is 25MB.

Michał

[miab3]

Hi aehimself, Jan,

This could be a question of new field types in D2006, D2007 and DXE2 compared to D7?:

Code: Select all

procedure TZAbstractRODataset.CheckFieldCompatibility(Field: TField; AFieldDef: TFieldDef);
const
  {EH: hint all commented types are the fields the RowAccessor can't handle -> avoid stack killing moves in Get/SetFieldData()
  this Error trapping is made for User-added fields like calculated's ....}
  BaseFieldTypes: array[TFieldType] of TFieldType = (
    //generic TFieldTypes of FPC and Delphi(since D7, of course):
    ftUnknown, ftString, ftSmallint, ftInteger, ftWord, // 0..4
    ftBoolean, ftFloat, ftCurrency, ftBCD, ftDate,  ftTime, ftDateTime, // 5..11
    ftBytes, ftVarBytes, ftInteger{ftAutoInc}, ftBlob, ftMemo, ftBlob{ftGraphic}, ftMemo{ftFmtMemo}, // 12..18
    ftBlob{ftParadoxOle}, ftBlob{ftDBaseOle}, ftBlob{ftTypedBinary}, ftUnknown{ftCursor}, ftString{ftFixedChar}, ftWideString, // 19..24
    ftLargeint, ftUnknown{ftADT}, ftUnknown{ftArray}, ftUnknown{ftReference}, ftDataSet, ftBlob{ftOraBlob}, ftMemo{ftOraClob}, // 25..31
    ftUnknown{ftVariant}, ftUnknown{ftInterface}, ftUnknown{ftIDispatch}, ftGuid, ftTimeStamp, ftFMTBcd // 32..37
{$IFDEF FPC} //addition types for FPC
    , ftWideString{ftFixedWideChar}, ftWideMemo // 38..39
    {$IFDEF WITH_FTLONGWORD}
    , ftTimeStamp{ftOraTimeStamp}, ftDateTime{ftOraInterval} //40..41
    , ftLongWord, ftShortint, ftByte, ftExtended //42..45
    {$ENDIF}
{$ELSE !FPC}
{$IF CompilerVersion >= 18} //additional Types since D2006 and D2007
    , ftWideString{ftFixedWideChar}, ftWideMemo, ftDateTime{ftOraTimeStamp}, ftDateTime{ftOraInterval} // 38..41
{$IF CompilerVersion >= 20} //additional Types since D2009
    , ftLongWord, ftShortint, ftByte, ftExtended, ftUnknown{ftConnection}, ftUnknown{ftParams}, ftBlob{ftStream} //42..48
{$IF CompilerVersion >= 21} //additional Types since D2010
    , ftDateTime{ftTimeStampOffset}, ftUnknown{ftObject}, ftSingle //49..51
{$IFEND CompilerVersion >= 21}
{$IFEND CompilerVersion >= 20}
{$IFEND CompilerVersion >= 18}
{$ENDIF FPC}
  );
  CheckTypeSizes = [ftBytes, ftVarBytes, ftBCD, ftReference, ftFmtBCD];
begin
  with Field do
  begin
    if (BaseFieldTypes[DataType] <> BaseFieldTypes[AFieldDef.DataType]) then
      DatabaseErrorFmt(SFieldTypeMismatch, [DisplayName,
        FieldTypeNames[DataType], FieldTypeNames[AFieldDef.DataType]], Self);
    if (DataType in CheckTypeSizes) and (Size <> AFieldDef.Size) then
        DatabaseErrorFmt(SFieldSizeMismatch, [DisplayName, Size,
          AFieldDef.Size], Self);
  end;
end;

Maybe it's also worth explaining the new types in Lazarus as mentioned in:
https://sourceforge.net/p/zeoslib/tickets/537/

Michał

[kjteng]

Hi aehimself,

Still "Stream read error" reading from disk Delphi XE2-Win32.
I use this method:
I am connecting to a database (right now Firebird) with a lot of different tables.
It clones them sequentially from ZTable to ZMemTable, writing to and reading from disk.

As for the size, the table that was written to the disk in your old version in D10.3.3-Win32
and occupied 13MB now in DXE2-Win32 is 25MB.

Michał

1. Stream read error - I got this error message only when running in debugging mode...a bit strange.
2. The size of file is bloated because string data are stored with the maximum size (eg for varchar(255), a string of two characters will be padded with 253 #0 on the right). In my version, I have avoided this by dealing string data separately and use AsString (rather than getdata or copy from blobstream). Perhaps aehimself can look into this.

[kjteng]

This is odd, my code works fine even on Delphi 7:
...

I got 'Access violation' error when trying to SaveToStream. I think the problem is due to the statement (Self.Fields[a] As TBlobField).SaveToStream(ms) -- I got error when the field type is ftMemo.

Please check, TMemoField should be a child of TBlobField. Otherwise the code wouldn't even end up in that branch; as there should be a check like this before:

Code: Select all

If Self.Fields[b] Is TBlobField Then [...]

Screen shot for your reference. Perhaps we can deal with string/memo field separately and make use of the AsString property (this would also reduces the size of string data with no trailing #0)

Screenshot (34).png

Screenshot (33).png

[miab3]

Hi aehimself,

I noticed (in DXE2-Win32) that two errors appear with LoadFromFile: 'Stream read error' and 'Out of memory' depending on TMemTable - ControlCodePage cCP_UTF16, cDynamic, cGET_ACP.

ADD: I think the encoding information should be saved and used in stream and file?

Michał

[marsupilami]

@aehimself: I fixed a compilation problem on Lazarus when ZMEMTABLE_ENABLE_STREAM_EXPORT_IMPORT is disabled. Not sure if you need that b variable or not though.

Also there seems to be a problem when enabling ZMEMTABLE_ENABLE_STREAM_EXPORT_IMPORT:
ZMemTable.pas(544,13) Error: identifier idents no member "ReadData"

It seems that TStream has no ReadData method on FPC 3.2.2.

Best rgeards,

Jan

[aehimself]

@Jan,

Something went wrong when the commits were applied, most of the new things did not go through. I'm happy you found the issue too though, why it failed to compile on FPC :)
No, variable "B" is not needed, it was removed in the changes which got lost somewhere. Also, ReadData was also changed.

I opened a new pull request to apply these.

[miab3]

Hi aehimself, Jan,

Duplicate SaveToStream and LoadFromStream in svn7702.
I still have 'Stream read error' in DXE2-Win32.
For example, in such a table FB2.5 (100000 Records):

Code: Select all

SET SQL DIALECT 3;

CREATE TABLE ELEMENTY100 (
    NRELEMENTU   INTEGER NOT NULL,
    NRKON        INTEGER,
    NRDOK        INTEGER,
    NRFRAGDOK    INTEGER,
    NAZFRAGDOK   VARCHAR(10) COLLATE PXW_PLK,
    NRSTRONY     SMALLINT,
    NRNASTRONIE  SMALLINT,
    NAZELEMENTU  VARCHAR(30) COLLATE PXW_PLK,
    TYP          VARCHAR(2) COLLATE PXW_PLK,
    HAKI         VARCHAR(2) COLLATE PXW_PLK,
    NRPRO        VARCHAR(4) COLLATE PXW_PLK,
    GATUNEKS     VARCHAR(12) COLLATE PXW_PLK,
    IGRUP        SMALLINT,
    IWGRUP       SMALLINT,
    ICALKO       SMALLINT,
    FI           SMALLINT,
    DLUGOSC      SMALLINT,
    DLUGTECH     SMALLINT,
    A1           SMALLINT,
    A            SMALLINT,
    B            SMALLINT,
    C            SMALLINT,
    D            SMALLINT,
    E            SMALLINT,
    F            SMALLINT,
    G            SMALLINT,
    A2           SMALLINT,
    X            SMALLINT,
    Y            SMALLINT,
    V            SMALLINT,
    S            SMALLINT,
    T            SMALLINT,
    U            SMALLINT,
    R            SMALLINT,
    UWAGI1       BLOB SUB_TYPE 1 SEGMENT SIZE 80 COLLATE PXW_PLK,
    DATA1        TIMESTAMP,
    ZROBIONO     SMALLINT,
    WYSLANO      SMALLINT,
    IWTOKU       SMALLINT,
    TECHNO       VARCHAR(1) COLLATE PXW_PLK,
    KTO          VARCHAR(2) COLLATE PXW_PLK,
    PRZEL        SMALLINT,
    FLAGI        INTEGER
);

Michał

[aehimself]

I noticed (in DXE2-Win32) that two errors appear with LoadFromFile: 'Stream read error' and 'Out of memory' depending on TMemTable - ControlCodePage cCP_UTF16, cDynamic, cGET_ACP.

Out of memory is easy to reproduce... just create a string field with the length 2000, append 1 000 000 empty rows and try to save it on Win32. This would need the process to have 2 000 000 000 bytes (~2 GB) of memory for the stream only, with which the process easily attempt to exceed the 2 GB Win32 memory limit. Unless I manage to trim the data as I planned first this will stay, but it's only a matter of fields / records to get up to this limit anyway.
Recompile your tool in Win64, that'll solve the memory issue.

No matter how hard I try I simply can not reproduce the Stream read error issues.

ADD: I think the encoding information should be saved and used in stream and file?

ControlsCodePage only affects in which encoding the RDBMS is sending textual data AFAIK, which will be converted back to Unicode/ ANSI by Zeos anyway. This is why no matter this setting, you never get scrambled text with FieldByName.AsString. As a MemTable has no connection it should be irrelevant in this case.
I really would like @Jan and @Michael to confirm this, though.

Nonetheless, this is a valid question which I completely overlooked. Thank you very much for the heads-up!

Edit: I'm an idiot. If we are talking about MemTables, send me the file you are attempting to load when you get the Stream read error. The layout should be in the stream so no external data should be needed.
Question is, if it goes south during saving or loading...

Also please make sure you export with the latest SVN code. As far as I can see the latest pull request is included, all builds are back to normal. Now ZMemTable.pas should be the same what I have on my dev branch.

[miab3]

Hi aehimself,

1. None of my tables exceed 100MB.
2. In versions from svn this error also occurs in D10.3.3-Win32 and in Win64 it does not want to load from a file.
3. I meant the distinction between cCP_UTF16, cDynamic, cGET_ACP, cCP_UTF8 which produce different streams and files
4. Currently for D10.3.3 I went back to your original pre-svn version.

Michał

[kjteng]

Hi aehimself,
I downloaded a copy of zmemTable.pas which contains the following twice in the source file.
{$IFDEF ZMEMTABLE_ENABLE_STREAM_EXPORT_IMPORT}

Procedure TZAbstractMemTable.LoadFromStream(AStream: TStream);
....
{$ENDIF}
{$IFDEF ZMEMTABLE_ENABLE_STREAM_EXPORT_IMPORT}

Procedure TZAbstractMemTable.SaveTostream(AStream: TStream);
....
{$ENDIF}

I think v need to remove one of them.

[marsupilami]

ControlsCodePage only affects in which encoding the RDBMS is sending textual data AFAIK, which will be converted back to Unicode/ ANSI by Zeos anyway. This is why no matter this setting, you never get scrambled text with FieldByName.AsString. As a MemTable has no connection it should be irrelevant in this case.

Unfortunately you are wrong. ControlsCodepage is the Codepage that is used for displaying text in Controls. On Unicode Delphi this should be UTF16. On FPC this will be UTF8 in a lot of cases but can be different. On ANSI Delphi this usually should be the system codepage. It also can be another codepageif you want to process data in a different character set.

The codepage that is used for communication with the database is the ClientCodepage setting.

[aehimself]

The codepage that is used for communication with the database is the ClientCodepage setting.

Damn, I mixed them up once again. I remember how much time I spent attempting to understand all encoding logic in Zeos but unfortunately I was unable to find the thread.
Thank you though, this is why I wanted someone with more knowledge to confirm :)

ControlsCodepage is the Codepage that is used for displaying text in Controls

In this case I think my point is still valid, I suppose. If a server application (which was built with Delphi) is saving the stream and sends it over the network to a client (which was built in Lazarus) changing the code page of linked controls will cause display issues.

The question is, is there ANY setting which is affecting Zeos's internal binary data storage format. If there is such, it must be saved and reloaded as Michal suggested.

[aehimself]

Duplicate SaveToStream and LoadFromStream in svn7702

I downloaded a copy of zmemTable.pas which contains the following twice in the source file.
{$IFDEF ZMEMTABLE_ENABLE_STREAM_EXPORT_IMPORT}

Procedure TZAbstractMemTable.LoadFromStream(AStream: TStream);
....
{$ENDIF}
{$IFDEF ZMEMTABLE_ENABLE_STREAM_EXPORT_IMPORT}

Procedure TZAbstractMemTable.SaveTostream(AStream: TStream);
....
{$ENDIF}

@Jan, can you please check? As changes are not on GitHub yet I can not confirm, but something is still not in-sync it seems. I attach my latest version of ZMemTable.pas, can you please compare the differences?