TZMemTable suggestions

[stoffman]

I would like to suggest and request 2 changes to TZMemTable that will make it much more useful and future proof

1. Make the first byte of the file a version number of the format, if in future version when the file layout will change it would be much much easier to write a code that supports older versions of the format (a simple if/else)

2. Expose a file_metadata field of type string that will be save to the file as well and it is *application specific* content. Developers may add this way to the file an arbitrary data such as:
a. the user name that created the file
b. timestamp of the creation
c. the original SQL query
d. description of the content
e. etc.. etc.. etc...

Again. this is a string that is application specific, it would be up to the application to parse this string after it was read back from the file

[marsupilami]

Hello Stoffman,

2. Expose a file_metadata field of type string that will be save to the file as well and it is *application specific* content. Developers may add this way to the file an arbitrary data such as:
a. the user name that created the file
b. timestamp of the creation
c. the original SQL query
d. description of the content
e. etc.. etc.. etc...

Again. this is a string that is application specific, it would be up to the application to parse this string after it was read back from the file

As far as I understand, the feature is intended to not be the only user of a stream. I think you could write your application specific data right in front of the stream:

Code: Select all

procedure WriteDataToStream(const AppData: String; Dataset: TZMemTable; SomeStream: SomeStream: TStream);
var
  StrLen: Cardinal;
  AppDataUtf8: UTF8String;
begin
  AppDataUtf8 := UTF8Encode(AppData);
  StrLen := length(AppDataUtf8);
  SomeStream.Write(StrLen, SizeOf(StrLen));
  SomeStream.Write(AppDataUtf8[1], StrLen);
  Dataset.SaveToStream(SomeStream);
end;

Note: I didn't test this code. But I think it ilustrates how the feature is intended to be used.

With best regards,

Jan

[stoffman]

Hi Jan,

The main drawback for implementing it this way is making the file *unportable* between applications.

I'm not inventing anything new here, if you take a look at the specifications of the JPEG (exif) file format you will see the UserComment Tag, you can basically store any information along side with the image.
While taking a picture with a camera it may store the camera owner's name, editing the image in Photoshop will disregard it as well as Chrome when viewing it.

Adding this metadata field directly in the format, ensure that anyone who develop an application with TZMemTable will be able to read it no matter who produce the file.

http://www.exif.org/Exif2-2.PDF
https://stackoverflow.com/questions/108 ... -exif-tags

[marsupilami]

Duh - you are going big there Do you really think there are usecases where data will be interchanged between Zeos applications from different vendors? And if there is that use case, wouldn't it make more sense to reuse something that already is standardized? Maybe take a look at OData instead of inventing our own binary file format for this use case?

[aehimself]

I am with Jan on this one. User comment in a memtable is not applicable in most applications and will only confuse the majority of the users. A need for this feels really application-specific, and this way it makes sense to have it's own format.

If you don't want to see it in the application you also can create a descendant of TZMemTable and add the property / additional code for yourself.

[stoffman]

Do you really think there are usecases where data will be interchanged between Zeos applications

First, I can easily see this used within the same organization. And not having to write low level parsing routine would be a great usability feature. Also not having such a feature would exclude the use of ZMemTable between vendors. Relative to the amount of work and the added functionality it is such a great feature to have.

will only confuse the majority of the users

I'm not sure how it can confuse users, care to give an example?


Also, what about the file format version number suggestion?

[Fr0sT]

Version is better to be 2-component: major and minor; minor versions are compatible at least backwards (newer software could read older data). However, in my opinion the format that is supposed for massive interchange should be more standard, probably XML, MsgPack or ProtoBuf.

Regarding header: extendable and customizable header is nice but the question is how to insert this feature into the code. Ideally it should be self-describing and order-independent like JSON or HTTP headers:

Code: Select all

Version: 1.3
FieldCount: 5
Fields: Field1...5
...any custom field that base class just skips

but how all this will fit into SaveTo/ReadFromStream methods?
Going further, it seems that separate reader/writer should be introduced instead of method of memtable, thus user will have more control and ability to customize, and could use any datasets as well.
But - is it really needed to invest time into?

BTW, current format could break if reader and writer are compiled with different TDataType enums - either writer has more extended range that reader hasn't or ordinal values vary (FPC vs Delphi?)