Zeos 8 - draft of Release Notes

[marsupilami]

Hello,

I finally took the time on putting together some release notes for Zeos 8. I attached a draft to this thread and will welcome all opinions and suggestions. I hope to take the time to finally get an oficial release of Zeos 8 during the upcoming weekend.

With best regards,

Jan

[aehimself]

I did not see any mentions of TZMemTable, which I rely heavily on in my database manager.

The rest is going to be rather "selfish", but I am familiar with things I added / fixed.

In Oracle there is a new OCIMultiThreaded parameter, which seem to be an important switch.

So - this basically means that Zeos (8.0) currently (without this patch) cannot use Oracle in in a multithreaded application in the usual way - one connection per thread? If so, I will have to add that to the release notes.

DBLib now also supports .ClientVersion, however I only tested it with AppVeyor builds. If it's noted, maybe we'll get more feedback on what we have to correct in it.

Imported -, Exported keys and Cross reference now supposed to work on MySQL and MariaDB.

As a known limitation it's important to mention that Oracle protocol still has issues with UTF16 and 64 bit. I still owe the sample dataset I know, the battle with the legal team is over and is on it's way.

There were also a number of smaller, quality-of-life changes like DefineAllSelectSchemasFromQuery, TZTokenizer comment skips, getting rid of Exception.Create and introducing EZUnsupportedException, introducing the ACTIVE_DATASET_SQL_CHANGE_EXCEPTION directive.

SQLConnection.GetColumnNames now considers the schema specified.

To be honest I don't know which one got ported back to 7.2 and which is release-note-worthy. Feel free to include or ignore all at your preference :)

[miab3]

There is an interesting toggle in Zeos.inc:

// If defined the ZMemTable component allows importing and exporting it's layout and data to a TStream descendant. Only tested with D10.3+ !!!
{.$DEFINE ZMEMTABLE_ENABLE_STREAM_EXPORT_IMPORT}

It works for me with all tested versions. So I unlock it by default.

Michal

[marsupilami]

Sooo: Hereis a new draft of the release notes. The changes are:

• I added a small Chapter about TZMemTable. I hope ist is enough
• added OCIMultiThreaded
• added dblib .ClientVersion
• added mported -, Exported keys and Cross reference on mysql/mariadb
• added a note about UTF16 on Oracle
• added a note about DefineAllSelectSchemasFromQuery
• added a note about the bugfix in TZTokenizer
• added a note about TZConnection.GetColumnNames
• added a chapter on new format settings for date and times (TZFormatSettings)

@aehimself:

• regarding DefineAllSelectSchemasFromQuery: Is there a chance to get a better description of what it can do tha I have come up with? What I wrote is kinda mysterious even to me - most probably I don't fully understand the use case.
• regarding ACTIVE_DATASET_SQL_CHANGE_EXCEPTION: Why didn't we activate this in general? Wasn't the poll clear about that? Was I against it?
• regarding Exception.Create: I really think this bugfixing and doesn't need a note. But I can add it with a txt like "In the past Zeos ocassionally created exceptions of the generic Exception type. This was changd to raising EZSQLException."?

@Michal:
I added a note about the define. Which versions of Delphi and FPC did you test? Do they work if mix versions? For example: Can Delphi 7 correctly read an Export from Delphi 11.3?

Opinions?

Jan

[miab3]

It works in all the ones I mentioned, that is since D2006 for sure.
I haven't tested between versions (I'm using it as a temporary, local application database).
It seems that there is no compatibility between versions yet (maybe in the future it would be worth developing a single portable format)
As for the D7, I wouldn't recommend Zeos8 for it, I get too many unexpected errors, it is true that testing on Windows11 (maybe too much hardcore).
Rather, D2006-D2007 as the lowest.

Michal

[aehimself]

regarding DefineAllSelectSchemasFromQuery: Is there a chance to get a better description of what it can do tha I have come up with? What I wrote is kinda mysterious even to me - most probably I don't fully understand the use case.

The current implementation of defineSelectSchemaFromQuery does a full stop after the first query was fully parsed. If you gave it 5 SQL queries it parsed the first one and that's all.
This was extended so parsing will continue until the actual end, giving back all aliases of all tables of all queries supplied. So let's say your user enters multiple queries in SynEdit for example, you can get everything back for further processing.

regarding ACTIVE_DATASET_SQL_CHANGE_EXCEPTION: Why didn't we activate this in general? Wasn't the poll clear about that? Was I against it?

There were some (including you I believe) who voted against changing the default behavior but the most important decision is definitely backwards compatibility. Changing something as impactful as this would cause lots and lots of applications changed and compiled again.

regarding Exception.Create: I really think this bugfixing and doesn't need a note. But I can add it with a txt like "In the past Zeos ocassionally created exceptions of the generic Exception type. This was changd to raising EZSQLException."?

Some was changed directly to EZSQLException, some was changed to EZUnsupportedException... always depending on the actual context. They are all EZSQLException descendants, though.

[marsupilami]

Soo - regarding DefineAllSelectSchemasFromQuery:
I literally took the wording of aehimself for the description of the feature. Thank you

regarding ACTIVE_DATASET_SQL_CHANGE_EXCEPTION:
I added a small chapter about that one too.

regarding Exception.Create:
I added a small chapter about this.

@aehimself: What do you think about stoffmans idea to add a version field to the implementation of SaveToStream and and LoadFromStream in TZMemTable? Maybe it also would make sense to add a kinda magic number or something like that that has to be present at the beginning of valid Zeos data?

Regarding the general availability of SaveToStream and and LoadFromStream: I am hesitant to generally enable this feature when the generated files are not compatible with each other across different versions of Delphi and maybe also FPC. I assume users would expect exactly this and we would have a lot of questions about that as soon as userst start to use newer versions of Delphi or exchange data between FPC and Delphi.
@aehimself: How do you handle these things in your use case?
For me it seems safe to "hide" this feature behind a define so users who want to use it must have read (and hopefully understood) the relevant chapter of the release notes?

@miab3:

As for the D7, I wouldn't recommend Zeos8 for it, I get too many unexpected errors,

What errors do you get on D7?

Best regards,

Jan

[miab3]

Hi Jan,

For example:

err1.png

err2.png

Idea @stoffman:
"Make the first byte of the file a version number of the format"
He's nice.
Might actually leave it disabled by default for now:
{.$DEFINE ZMEMTABLE_ENABLE_STREAM_EXPORT_IMPORT}

Michal

[marsupilami]

For example:
err1.png
err2.png

Could you try to generate Stack traces or even better a simple test case where this happens?

[miab3]

From what I see, errors in D7 appear especially at:

ZMemTable1.SaveToStream(fs);
ZMemTable1.LoadFromStream(fs);

In D2007 in the same situation no.

Michal

[aehimself]

@Jan:
I am hesitant about the versioning, especially adding the version number to the binary data. If the stream is not a valid Zeos data, the import will fail anyway.
We already talked about being TClientDataSet compatible and use an XML format instead but I failed to find a layout what I should follow.

@Michal,
So it's not Zeos in general having issues, only ZMemTable; and more specifically .SaveToStream and .LoadFromStream. Do you maybe have stack traces? Would help to pinpoint the issue.

What I suggest is to keep it disabled for the time being, and as time allows I'll take a look and attempt to rewrite the logic to fix the issues mentioned here. That also means previous data streams will be incompatible, so...

[marsupilami]

@Jan:
I am hesitant about the versioning, especially adding the version number to the binary data. If the stream is not a valid Zeos data, the import will fail anyway.

I do understand what you mean. But adding a version number and maybe a magic number certainly could help users in understanding what goes wrong. The error message "This isn't a valid TZMemtableStream" or "This version of TZMemtable Streams is not supported" is more specific and understandable than any other message about not being able to load the stream data. Also it could help in understanding if an error is a Zeos error or an error done by the user.

We already talked about being TClientDataSet compatible and use an XML format instead but I failed to find a layout what I should follow.

I do remember that. I tried to find something on the net but there isn't much. Maybe we should invent our own format. The WebServiceProxy server already encodes a Zeos IZResultSet as XML. Maybe we should try to reuse the code from there?

What I suggest is to keep it disabled for the time being, and as time allows I'll take a look and attempt to rewrite the logic to fix the issues mentioned here.

Maybe we will be lucky and it is just something in Delphi 7 that doesn't require a change in the stream data layout? We could mark the feature as experimental in the Release notes. Something like

Note:This feature is currently an experimental feature. The Layout of the generated streams may change between Zeos 8 minor versions until it is marked as stable. So streams may not be readable by future Zeos versions.

That also means previous data streams will be incompatible, so...

Isn't this the case where versioning could help? This would allow to read older streams on newer Zeos versions I know, I am being a pest again
Maybe it makes sense th have a 32Bits version number like FFFFFFFF and decrement it with each new version. Having this many bits set to 1 should be quite uncommon?

[miab3]

Maybe it makes sense th have a 32Bits version number like FFFFFFFF and decrement it with each new version. Having this many bits set to 1 should be quite uncommon?

With this, FF FF FF FF is a good idea because now the first 4 bytes are occupied by the number of fields, so it does not come close to FF FF FF FF.

Michal

[aehimself]

The error message "This isn't a valid TZMemtableStream" or "This version of TZMemtable Streams is not supported" is more specific and understandable than any other message about not being able to load the stream data.

In my opinion even if the format changes, the component should be able to load it - worst case scenario with some loss of data. This way the only real error message is that the stream is not TZMemTable data.

The WebServiceProxy server already encodes a Zeos IZResultSet as XML. Maybe we should try to reuse the code from there?

It makes sense, a lot of it actually. From redundancy perspective - now two code sections are doing the same thing, only in a different way. From compatibility perspective - even if never used, I love the idea of the proxy driver and TZMemTable to be able to read each others output. Lastly, it works correctly I assume :)
If you move the code to it's own class (or directly into methods of IZResultSet) implementing it in TZMemTable will be extremely easy.

If we go this way (which I fully support) we can even remove the compiler directive and always enable it.

[marsupilami]

It makes sense, a lot of it actually. From redundancy perspective - now two code sections are doing the same thing, only in a different way. From compatibility perspective - even if never used, I love the idea of the proxy driver and TZMemTable to be able to read each others output. Lastly, it works correctly I assume
If you move the code to it's own class (or directly into methods of IZResultSet) implementing it in TZMemTable will be extremely easy.

If we go this way (which I fully support) we can even remove the compiler directive and always enable it.

Well - saving the data is done. But now that I think more about it, there are problems: The code operates on the DBC level. So FieldDefs are not saved. They would have to be recreated when loading with the current code. This also will lead to problems when it comes to things like calculated fields or lookup fields.
So while I still think, we can use the code for saving the data, we might want to think about saving the TZMemTable Metadata by using Delphis / Lazarus methods for streaming components. They should be mostly compatible, given that DFM and LFM files look the same?

But then this is going off topic regarding the release notes

Attached is the latest version of the release notes. They are not finished because I realized that the documentation on TZConnection.RawTransliterateOptions is missing...