zMemtable

[kjteng]

May I know if it is possible to load/save to/from stream. I ask this questions because I have applications (currently using tbufdataset) which store the whole bufdataset in a field (of a table). I am thinking of replacing it with tzmemtable.

[marsupilami]

Hello kjteng,

I seem to remember that this is not possible currently. We think about adding this as a feature in the future.

Best regards,

Jan

[aehimself]

Hello,

As far as I know there is no direct method to load / save the contents from / to a stream from any Zeos dataset. However writing your own is pretty easy.

With a TZQuery I already managed to export to / import from a JSON file.
I guess if you use a stream, just write the column names and then all the field data as TBytes. Loading it back is as easy as reading the column names out, appending and reentering the saved field data.

I also advise to take a look how WebServiceProxy works. I personally never used it but as far as I understand it's purpose is to serialize / deserialize field data so it can be sent over HTTP for example.
Jan might correct me out on this one if I'm not right.

[miab3]

Hi,

There is currently no such possibility.
You can use another dataset as an intermediary.
Here are examples of using the ZMemTable:

Code: Select all

procedure TForm1.Button2Click(Sender: TObject);
begin
ZMemTable1.CloneDataFrom(ZTable1);
end;

procedure TForm1.Button4Click(Sender: TObject);
begin
ZMemTable1.Close;
ZMemTable1.FieldDefs.Clear;
ZMemTable1.FieldDefs.Assign(ZTable1.FieldDefs);
ZMemTable1.Open;
end;

procedure TForm1.Button6Click(Sender: TObject);
begin
ZMemTable1.Close;
ZMemTable1.FieldDefs.Clear;
ZMemTable1.FieldDefs.Add('Lp', ftinteger);
ZMemTable1.FieldDefs.Add('Naz',ftString,20);
ZMemTable1.Open;
end;

Michał

[aehimself]

Edit ZMemTable.pas and add two public procedures:

Code: Select all

    Procedure SaveToStream(AStream: TStream);
    Procedure LoadFromStream(AStream: TStream);

Implementation is as follows:

Code: Select all

Procedure TZAbstractMemTable.LoadFromStream(AStream: TStream);
Var
 a, count, b, len: Integer;
 s: String;
 ftype: TFieldType;
 required: Boolean;
 c: Char;
 tb: TBytes;
Begin
 Self.CheckInactive;

 Self.FieldDefs.Clear;

 Self.DisableControls;
 Try
   // How many fields there are
   AStream.ReadData(count);

   For a := 0 To count - 1 Do
   Begin
     // Name of field
     AStream.ReadData(len);
     SetLength(s, len);
     For b := 1 To len Do
     Begin
       AStream.ReadData(c);
       s[b] := c;
     End;
     // Field type as Integer
     AStream.ReadData(b);
     ftype := TFieldType(b);
     // Size of field
     AStream.ReadData(b);
     // Is the field required?
     AStream.ReadData(required);

     Self.FieldDefs.Add(s, ftype, b, required);
   End;

   Self.Open;

   // How many records there are
   AStream.ReadData(count);

   For a := 0 To count - 1 Do
   Begin
     Self.Append;

     For b := 0 To Self.FieldCount - 1 Do
     Begin
       // Length of data
       AStream.ReadData(len);

       // Data itself
       SetLength(tb, len);
       AStream.Read(tb, len);

       Self.Fields[b].AsBytes := tb;
     End;

     Self.Post;
   End;

   Self.First;
 Finally
   Self.EnableControls;
 End;
End;

Procedure TZAbstractMemTable.SaveToStream(AStream: TStream);
Var
 bm: TBookMark;
 a, b: Integer;
 tb: TBytes;
Begin
 Self.CheckActive;

 bm := Self.GetBookmark;
 Try
   Self.DisableControls;
   Try
     // Write how many fields there are
     AStream.WriteData(Self.FieldDefs.Count);
     For a := 0 To Self.FieldDefs.Count - 1 Do
     Begin
       AStream.WriteData(Length(Self.FieldDefs[a].Name));
       For b := 1 To Length(Self.FieldDefs[a].Name) Do
         AStream.WriteData(Self.FieldDefs[a].Name[b]);
       AStream.WriteData(Integer(Self.FieldDefs[a].DataType));
       AStream.WriteData(Self.FieldDefs[a].Size);
       AStream.WriteData(Self.FieldDefs[a].Required);
     End;

     Self.First;
     AStream.WriteData(Self.RecordCount);
     While Not Self.Eof Do
     Begin
       For a := 0 To Self.FieldCount - 1 Do
       Begin
         tb := Self.Fields[a].AsBytes;

         AStream.WriteData(Length(tb));
         AStream.WriteData(tb, Length(tb));
       End;

       Self.Next;
     End;
   Finally
     Self.EnableControls;
   End;
 Finally
   Self.GotoBookmark(bm);
 End;
End;

The only drawback is that it will only work on newer Delphi versions, who already knows what TBytes is and has Field.AsBytes.

Tested with a small demo:

Code: Select all

procedure TForm1.FormCreate(Sender: TObject);

 Procedure Add(Const inName: String);
 Begin
  ZMemTable1.Append;
  ZMemTable1.FieldByName('ID').AsInteger := ZMemTable1.RecordCount;
  If Not inName.IsEmpty Then ZMemTable1.FieldByName('Name').AsString := inName;
  ZMemTable1.Post;
 End;

begin
  ms := TMemoryStream.Create;

  ZMemTable1.FieldDefs.Add('ID', ftInteger, 0, True);
  ZMemTable1.FieldDefs.Add('Name', ftString, 50);

  ZMemTable1.Open;
  Add('Kovács');
  Add('Péter');
  Add('Jakab');
  Add('Gipsz');

  ZMemTable1.SaveToStream(ms);
  ms.Position := 0;
  ZMemTable1.Close;
  ZMemTable1.LoadFromStream(ms);
end;

Handles unicode perfectly, the connected DBGrid shows the exact same data as it did before.

I'm not going to send a pull request as the TBytes thing has to be ported back to earlier Delphi / Lazarus versions. It's a good starting point, though.

[miab3]

Hello aehimself,

Wow. Well, then only SaveToFile and LoadFromFile are missing.

Michał

[aehimself]

SaveToFile:

Code: Select all

Var
 fs: TFileStream;
Begin
  fs := TFileStream.Create('filename', fmWrite);
  Try
    Self.SaveToStream(fs);
  Finally
    fs.Free;
  End;
End;

LoadFromFile:

Code: Select all

Var
 fs: TFileStream;
Begin
  fs := TFileStream.Create('filename', fmRead);
  Try
    Self.LoadFromStream(fs);
  Finally
    fs.Free;
  End;
End;

these should work :)

Edit: I just wrote this by heart, I'm not sure if FileStream constructor has the proper parameters. Also, this will save things in binary format, not XML or some other well-known, easily parseable.
One more thing which pops in my mind, in the past when I prepared a test case for Michael, his IDE did not handle loading numeric fields as bytes correctly ad he manually had to parse my data. I have no Lazarus / older Delphi installed so I can not test, unfortunately.

[kjteng]

Thanks all for the replies and suggestion. aehimself's methods look promissing and I will give it a try.

[miab3]

Hi aehimself,

Works (D10.3.3), only minor fix:

Code: Select all

procedure TForm1.Button13Click(Sender: TObject);
var fs: TFileStream;
begin
  fs := TFileStream.Create('t1.bin', fmCreate);
  Try
    ZMemTable1.SaveToStream(fs);
  Finally
    fs.Free;
  End;
end;

procedure TForm1.Button14Click(Sender: TObject);
var fs: TFileStream;
begin
  ZMemTable1.Close;
  fs := TFileStream.Create('t1.bin', fmOpenRead);
  Try
    ZMemTable1.LoadFromStream(fs);
  Finally
    fs.Free;
  End;
end;

One note. Only works properly on Win32 builds.

Michał

[aehimself]

Let me guess, it’s because of NativeInt in TStream.WriteData. I thought I can save writing some custom methods by using the built in ones, but hey.

Jan,
As the implementation of.SaveToStream and .LoadFromStream is going to get a lot more messy, I’d like to issue a pull request once I’m done with the custom data writing / reading. Because of reasons above I also don’t want the code to be “live” as it’ll most probably break backwards compatibility and has to be reviewed first.
Can you put these modifications in a different “branch” or should I simply use a conditional directive?

[miab3]

Hi aehimself,

Let me guess, it’s because of NativeInt in TStream.WriteData. I thought I can save writing some custom methods by using the built in ones, but hey.

Probably yes.
A disk dump of a large table with different types is ~ 50% larger for Win64.
On top of that, SaveToStream / LoadFromStream for Win64 makes bugs even in your example.
It would be nice to have disk format compatibility for both Win32 and Win64.
I would stick to a common format for many systems (even at the expense of size).
Well, disk read / write is a bit slow (probably stream generation), 3 times slower than cloning with a dataset.

Michał

[aehimself]

Please try again with these methods, working fine in both 32 and 64 bit:

Code: Select all

Procedure TZAbstractMemTable.LoadFromStream(AStream: TStream);

 Function ReadBool: Boolean;
 {$IFDEF FPC}
 Type
  PBoolean = ^Boolean;
 {$ENDIF}
 Var
  tb: Array Of Byte;
 Begin
  SetLength(tb, SizeOf(Boolean));
  AStream.Read(tb[0], Length(tb));

  Result := PBoolean(@tb[0])^;
 End;

 Function ReadInt: Integer;
 Var
  tb: Array Of Byte;
 Begin
  SetLength(tb, SizeOf(Integer));
  AStream.Read(tb[0], Length(tb));

  Result := PInteger(@tb[0])^;
 End;

 Function ReadBytes: TBytes;
 Begin
  SetLength(Result, ReadInt);

  AStream.Read(Result[0], Length(Result));
 End;

 Function ReadString: String;
 Var
  a: Integer;
  c: Char;
 Begin
  SetLength(Result, ReadInt);

  For a := 1 To Length(Result) Do
  Begin
    AStream.ReadData(c);
    Result[a] := c;
  End;
 End;

Var
 a, b: Integer;
Begin
 Self.CheckInactive;

 Self.FieldDefs.Clear;

 Self.DisableControls;
 Try
   // Recreate FieldDefs
   For a := 0 To ReadInt - 1 Do
     Self.FieldDefs.Add(ReadString, TFieldType(ReadInt), ReadInt, ReadBool);

   // Activate the MemTable so we can write the data back
   Self.Open;

   // Now read each field of each record, one by one
   For a := 0 To ReadInt - 1 Do
   Begin
     Self.Append;

     For b := 0 To Self.FieldCount - 1 Do
       Self.Fields[b].AsBytes := ReadBytes;

     Self.Post;
   End;

   Self.First;
 Finally
   Self.EnableControls;
 End;
End;

Procedure TZAbstractMemTable.SaveToStream(AStream: TStream);

 Procedure WriteBool(Const ABoolean: Boolean);
 {$IFDEF FPC}
 Type
  PBoolean = ^Boolean;
 {$ENDIF}
 Var
  tb: Array Of Byte;
 Begin
  SetLength(tb, SizeOf(Boolean));
  PBoolean(@tb[0])^ := ABoolean;

  AStream.Write(tb[0], Length(tb));
 End;

 Procedure WriteInt(Const ANumber: Integer);
 Var
  tb: Array Of Byte;
 Begin
  SetLength(tb, SizeOf(Integer));
  PInteger(@tb[0])^ := ANumber;

  AStream.Write(tb[0], Length(tb));
 End;

 Procedure WriteBytes(Const ABytes: TBytes);
 Begin
  WriteInt(Length(ABytes));

  AStream.Write(ABytes[0], Length(ABytes));
 End;

 Procedure WriteString(Const AText: String);
 Var
  a: Integer;
 Begin
  WriteInt(Length(AText));

  For a := 1 To Length(AText) Do
   AStream.WriteData(AText[a]);
 End;


Var
 bm: TBookMark;
 a: Integer;
Begin
 Self.CheckActive;

 bm := Self.GetBookmark;
 Try
   Self.DisableControls;
   Try
     // Write how many fields there are
     WriteInt(Self.FieldDefs.Count);
     For a := 0 To Self.FieldDefs.Count - 1 Do
     Begin
       WriteString(Self.FieldDefs[a].Name);
       WriteInt(Integer(Self.FieldDefs[a].DataType));
       WriteInt(Self.FieldDefs[a].Size);
       WriteBool(Self.FieldDefs[a].Required);
     End;

     Self.First;
     WriteInt(Self.RecordCount);
     While Not Self.Eof Do
     Begin
       For a := 0 To Self.FieldCount - 1 Do
         WriteBytes(Self.Fields[a].AsBytes);

       Self.Next;
     End;
   Finally
     Self.EnableControls;
   End;
 Finally
   Self.GotoBookmark(bm);
 End;
End;

TBytes issue is still present, I guess I need to look into GetData and SetData to be able to avoid this...
The previous test dataset is 119 bytes on 32 and 64 bit too, I guess they should be cross-platform-compatible now :)

Edit: Added PBoolean to FPC... at least Zeos defines it elsewhere, too

[miab3]

Hi aehimself,

Something is wrong with the new LoadFromStrem.
Even on Win32 your test does not pass.

Michał

[aehimself]

Eh, last minute optimizations. I really should stop doing that. If you leave a variable for everything that makes sure everything is being read back in order:

Code: Select all

Var
 a, b, len, ftype, fsize: Integer;
 fname: String;
Begin
 Self.CheckInactive;

 Self.FieldDefs.Clear;

 Self.DisableControls;
 Try
   // Recreate FieldDefs
   len := ReadInt;
   For a := 0 To len - 1 Do
   Begin
     fname := ReadString;
     ftype := ReadInt;
     fsize := ReadInt;
     Self.FieldDefs.Add(fname, TFieldType(ftype), fsize, ReadBool);
   End;

   // Activate the MemTable so we can write the data back
   Self.Open;

   // Now read each field of each record, one by one
   len := ReadInt;
   For a := 0 To len - 1 Do
   Begin
     Self.Append;

     For b := 0 To Self.FieldCount - 1 Do
       Self.Fields[b].AsBytes := ReadBytes;

     Self.Post;
   End;

   Self.First;
 Finally
   Self.EnableControls;
 End;
End;

I'll try to look into how to avoid using TBytes to keep older FPC / Delphi compatibility.

[miab3]

Hi aehimself,

Now it runs in D10.3.3 Win32 / Win64 and loads data from each other

Michał