When the ZEOS 7 new version for Delphi 2010 ?

[Wilco]

Dear all,

I now use Delphi 2010 and i've tryied the zeos 7 alpha version. It does not yet work correct, so my question is when can we get the new version of zeoslib ?


Kind regards,
Wilco

[Wilco]

Bonjour à tous,

Je développe avec Delphi depuis les années 90 (j'ai dû commencé avec Delphi3). Avec Delphi 7, j'utilise les composants Zeos et DevExpress et lorsqu'en en décembre 2009, j'ai acheté la version Delphi 2010, je me suis aperçu qu'il n' y avait pas de version de Zeos compatible.

Concernant Devexpress, les composants ont été adaptés afin d'être compatibles avec Delphi 2010, donc pas de problème de ce coté la.

Mon problème se situe au niveau de Zeos. Le 31 décembre 2009, la version zeos 7.0.0 alpha était mis en téléchargement. Hélas, sur les différents forums (developpez.com, zeoslib portal), les utilisateurs qui l'ont essayée mentionnent trop de bugs et de problèmes avec Zeos version 7 alpha (la seule version annoncée compatible avec Delphi 2010).

Je décide donc d'attendre. J'espérais que dans le premier trimestre 2010, une version opérationnelle de Zeos sortirait ...

Fin septembre 2010, je me dits que je pourrais peut être essayé moi même cette version 7 alpha de Zeos. Après quelques obstacles à surmonter lors de l'installation, j'arrive à faire mes premiers tests et ils ne sont pas concluants.

Par exemple : Zquery.parambyName('variable').AsString := 'valeur'; génère une exception à l'exécution. En fait la valeur n'est pas transmise à la requète SQL.

Cette erreur que j'ai constatée par moi même avait été relayée sur le forum de zeos dès janvier 2010.

Je suis surpris qu'en plus de 9 mois, le team de zeos n'ait pas réussi à corriger ces bugs et à proposer au minimum une version 7 béta de zeos pour Delphi 2010.

Je suis un fan de Delphi et j'espère bien pouvoir utiliser très prochainement cette version 2010 que j'ai achetée il y a maintenant 10 mois.

Toute cette littérature pour ne poser qu'une seule question :

A quand une version opérationnelle de zeoslib pour Delphi 2010 ?

A contre coeur, je continue d'utiliser ma vielle version Delphi 7.

Wilco
Développeur Delphi + zeos + mysql

[devil_inside]

i use ZeosLib 7 alpha +D2010 +Firebird and all work fine..
what you mean with "does not yet work correct" example?

[tygrys]

i use ZeosLib 7 alpha +D2010 +Firebird and all work fine..
what you mean with "does not yet work correct" example?

On some database configuration it does not work well.

ie. I have Firebird 1.5 database with win1250 encoding internaly. I'm also using polish windows (that means win1250 ansi internal) and I set connection properties to codepage=win1250.

It works fine under delphi 2006, but under XE it destroys native characters in database table on insert/update when using direct SQL.

Statement: INSERT INTO TABLE VALUES('ąęść')
produces junk in the field. This is due to ZEOS converting the SQL to UTF8 no mater the actual connection encoding.

Statement INSERT INTO TABLE VALUES(:value)
and setting ZQuery.Params[0] := 'ąęść' works ok, and inserts/updates using that method are ok.

I think, we need some logic to use utf8 encoding when talking with database, not just {$IFDEF DELPHI_12UP}

Tygrys

[mdaems]

tygrys,
I fully agree, but we really need people who know what they are talking about when it comes to charsets, encodings, ... AND Delphi, Pascal, ...
I do not qualify for this at all. I'm anly a hobby pascal programmer and live in a country where unicode isn't an issue at all. (Dutch is a language which has just a 26 letter alphabet)
So if you have the time and knowledge to implement (or even just design) this, please tell us.

Mark

[tygrys]

Lucky you with 26 letters :-)

I may understand how encodings and charsets work, but I have no knowledge on Zeos internals. So far, version 6.6 was ok for me with international character support. When I upgraded to XE I found problems with recompiling 6.6, so I installed 7.0. It compiles fine, but destroys data in the database in my program. So I looked at source and found how unicode support was made in Zeos and I really don't understand that. I think I can help to explain how it should work, but I need a person who understand how it is done inside Zeos.

Maybe the problem is that Utf8String(SQL) is not a typecast, but a character conversion. It actually changes the content of the SQL string. It is ok if we talk to the database using UTF8, it is bad if we use ANSI or UCS16. And this depends on a database connection encoding, not the compiler version.

Some other problems reported here about encoding character data in SQL is also affected by this issue. (ie. bugtrack 0000228).

Tygrys

[mdaems]

Hi Tygrys,

I fully agree the encoding stuff looks clumsy.
Main reason is : at some moment some very smart(?) people decided a string isn't a normal string anymore. What worked before just didn't anymore and we had to do a 'quick' conversion, while keeping old behaviour intact.
But then, while testing some assumptions of connection charsets may have slipped in allowing for using Utf8String, ...

I need a person who understand how it is done inside Zeos.

Well, good luck finding somebody who can help with that...
As far as I understand it, it's just not 'done inside Zeos'.
My idea about what *should* be done is
- decide on an internal encoding to be used for sql strings and string data. Is this depending on the environment you're working in (windows/linux, locale, project setting, ...)? In that case the choice should be to use this encoding throughout the component and clientside dbc cach side.
- defining a plaindriver (or dbc, not sure) connection level connection encoding, set on connect from connection information or used to force connection encoding.
- create a function to convert a string (ansi or unicode, depending on compiler) from one encoding to the other
- Start changing everything from the core to the compoent layer to implement this. (Nice and funny job, I think)
- find a solution for encoding blobs passed inline into the query string when just passing by this wormnest of code
- If we've some remaining energy and motivation : catch users trying to mess this up by sending sql statements changing the connection encoding between sql statements. (By giving the resultsets a string encoding property?)

Does that look like a good start of a high level design?

Mark

[regs]

Hey can anyone help with setting up SQlite encoding. I suppose codepage doesn't work for it. They how could I switch between ANSI and UTF8?

[tygrys]

Hi Tygrys,

I fully agree the encoding stuff looks clumsy.
Main reason is : at some moment some very smart(?) people decided a string isn't a normal string anymore. What worked before just didn't anymore and we had to do a 'quick' conversion, while keeping old behaviour intact.
But then, while testing some assumptions of connection charsets may have slipped in allowing for using Utf8String, ...

....
Does that look like a good start of a high level design?

Mark

Well, I think it is much simpler then tahat :-)
Zeos 7.0 works genarally like 6.6 and IMHO misses one important thing.

Someone (in Zeos stuff) decided to convert all SQL incoming from application to UTF-8 before sending it to database driver. It is not a good idea, because connection codepage can be ANSI, not UTF8.

The Delphi string is not UTF8 string, but unicodestring. Theay are different. Unicodestring can contain ansi content, utf8 content or ucs2 content. And that encoding can be found in the string header.

But Zeos performs Utf8String(SQL) which changes this to utf8 regardles of anything. This is not a simple typecast, this is actualy a conversion of characters in the string. If the connection codepage is ANSI we should have Ansistring(SQL) there. If it is UTF-8 it is right to have UTF8 string here. If it is UCS2 this should be different also (or maybe left alone), i dont know if database drivers can handle ucs strings.

Problem affects Firebird and MS SQL, Postgress seems to run fine with this. I don't know how others.

If I change Utf8String(SQL) to AnsiString(SQL) programs run fine - but when the connection encoding will be UTF-8 they will not I suppose. Also, that way I cannot pass true Unicode content to the database. But - again - this depends on database table encodings.

Maybe we will need an extra property in ZConnection to define how strings should be handled before sending to the database.
something like: encAnsi, encUTF8, encUSC2. Therefore appropiate conversion will be applied to the SQL string before it will be sent to the database.

Or maybe it is a design error and Zeos should know how to handle content by itself? I noticed that all my StringFields have to be now WideStringFields, this is not a big problem, but maybe a source of encoding mess?

Someone who put UTF8String(SQL) into DBC should consider this issue.

Tygrys

[tygrys]

My idea about what *should* be done is
- decide on an internal encoding to be used for sql strings and string data. Is this depending on the environment you're working in (windows/linux, locale, project setting, ...)? In that case the choice should be to use this encoding throughout the component and clientside dbc cach side.
- defining a plaindriver (or dbc, not sure) connection level connection encoding, set on connect from connection information or used to force connection encoding.
- create a function to convert a string (ansi or unicode, depending on compiler) from one encoding to the other
- Start changing everything from the core to the compoent layer to implement this. (Nice and funny job, I think)
- find a solution for encoding blobs passed inline into the query string when just passing by this wormnest of code
- If we've some remaining energy and motivation : catch users trying to mess this up by sending sql statements changing the connection encoding between sql statements. (By giving the resultsets a string encoding property?)

Does that look like a good start of a high level design?

Mark

This looks great. :-)

I think we should avoid conversions at all, we have just to send correct information about them to the database. This way it worked in 6.6 and I think this was good enough to just mimic it in 7.0.

In 6.6 I have ansistring in my windows with accent characters encoded WIN1250. I have table in the database which uses WIN1250 as encoding/collation. If I set database driver to use Codepage=WIN1250 this just works.

When I use now Delphi XE I have a UnicodeString in my application, and it still has Ansi data in it and still uses WIN1250 encoding. And I like all to work as before. :-)

The addition to the 7.0 is the fact, that now I can have Unicode data in my string. They can be UTF or UCS or maybe different Ansi codepage.
If _AND ONLY IF_ my database has fields that can handle such data and database connection can send them properly encoded I will need a unicode conversion.

But nowadays in 7.0 the conversion is made {$IFDEF DELPHI12_UP}.
And the encoding is forced to be UTF-8
And all my string fields now are TWIdeStringField, which is not a good idea because they reffer to the ansi CHAR or VARCHAR fields in the database. But I can live with latter. :-)

And I think this all together is not good.

Tygrys

[mdaems]

But nowadays in 7.0 the conversion is made {$IFDEF DELPHI12_UP}.
And the encoding is forced to be UTF-8

As I said : I fully agree, but that's the way it has been developed by somebody who DID speand a lot of time to get things working, at leat under his development conditions.

As this is an open source project developed by volunteers only, the only way to get better results is by stepping up yourself to get this done the right way. I'm sure loads of peopl will be available to test this code. Maybe you'll find some people who even want to actively help you. I hope others will respond to this thread in a more cooperative way than I do.

As I'm not an experienced 'charsets user' and because I'm not using a recent Delphi version I can only support you running the test suite to check what's broken by your patches. (I can even manage to run this tests for D2009)
But when your changes are reasonable and do not break the test suite of no delphi12+ compilers, I'll fully cooperate. I'm even willing to consider arranging SVN commit rights when first contributions seem OK.

Mark

[tygrys]

But nowadays in 7.0 the conversion is made {$IFDEF DELPHI12_UP}.
And the encoding is forced to be UTF-8

As I said : I fully agree, but that's the way it has been developed by somebody who DID speand a lot of time to get things working, at leat under his development conditions.

So, I'd like to talk to that "somebody" and ask him why he did it that way.
If I'll understand the idea, I can suggest what to do to make it work in my case. Or, maybe I can explain where the trouble is and why it is not working - so the person who knows the idea will change things to work better.

I just don't know how it should be - I just can patch a place where things go wrong for me. But my patch will not be good for everybody - and I'm really sure it will destroy things in other enviroment.

I may suggest the option to "make it default way" or "make it my way", but this is a workaround, not a proper design.

I'm open to help Zeos team, but I don't feel competent to develop ZeosLib. I may build complete test case for someone to show the problem and show my patch. But changes in the Zeos code are to be made by the person who knows what he is doing. :-)

Tygrys

[gto]

Hello tygrys !

First, I would like to apologize for my lack of presence at Zeos forum and community, as I've moved (and still moving) from city, job and school in the past six months.

About the Unicode question, I totally agree with your(s) idea(s). I've did the most of conversion work, but never believed it was good enough (although it, somehow, works ). Even about Unicode and encodings, I know a lot now than when I've did that, so, yes, agree with your points again.

And finally, when we start to work ?

Contributing to Zeos code isn't that difficult nor is needed that you're the best developer in the world. You're probably a better developer than I am! If you aren't sure, start, propose, discuss, and someone will help you (us). That's the way things go in Free Software world !

[]'s!

[tygrys]

Hello tygrys !

And finally, when we start to work ?

Contributing to Zeos code isn't that difficult nor is needed that you're the best developer in the world. You're probably a better developer than I am! If you aren't sure, start, propose, discuss, and someone will help you (us). That's the way things go in Free Software world !

Greetings gto.

We start right now :-)

First we need some method of communication. It could be this thread, we may start another. I'll pm you my skype to make things faster. Or maybe you have other idea?

Second, the goals (as I see them):
1) Make the 7.0 compatible with itself in different Delphi versions, that means that change from Delphi 2007 (and before) to 2009 (and after) will not require to change the database application. Now, because of {$DEFINE DELPHI12_UP} they behave different.

2) Make the Unicode and widestringfield an option, not a must-have.
It should be unicode-enabled, not unicode-forced :-)
I don't know if all databases supported by Zeos accept UTF8. AFAIK Firebird 1.5 don't.

3) All that Mark have written in this thread about the design - that mean how data will pass from application to the database and what conversions Zeos need to do to make it unicode-enabled.

Third, the databases. I am using Firebird, Postgerss and MS SQL and can test in this enviroments. We will need people who can test other databases. I think we should start with one database and make things working there first. Bet your choice.

Ok, I'll join the team :-)
tygrys

[ab]

Hello all,

It's my first post here, with concrete proposal.

I've successfully written some database related units, working for Delphi 6 up to XE. The key is to use a "proprietary" type internally, using UTF-8, then provide dedicated conversion when you want to interact with the VCL (i.e. the string type, depending on the Delphi version it was compiled with).

Our SQLite3 Framework has 100% UNICODE compatibility, that is compilation under Delphi 2009/2010/XE. The code has been deeply rewritten and tested, in order to provide compatibility with the String=UnicodeString paradigm of these compilers.

Since our SQLite3 framework is natively UTF-8 (this is the better character encoding for fast text - JSON - streaming/parsing and it is natively supported by the SQLite3 engine), we had to fix the way our framework used strings, in order to handle all versions of Delphi (even pre-Unicode versions, especially the Delphi 7 version we like so much), and provide compatibility with the Free Pascal Compiler.

Some string types have been defined, and used in the code for best cross-compiler efficiency (avoiding most conversion between formats):

- RawUTF8 is used for every internal data usage, since both SQLite3 and JSON do expect UTF-8 encoding;

- WinAnsiString where WinAnsi-encoded AnsiString (code page 1252) are needed;

- generic string for i18n (e.g. in unit SQLite3i18n), i.e. text ready to be used within the VCL, as either AnsiString (for Delphi 2 to 2007) or UnicodeString (for Delphi 2009/2010/XE);

- RawUnicode in some technical places (e.g. direct Win32 *W() API call in Delphi 7) - note: this type is NOT compatible with Delphi 2009/2010/XE UnicodeString;

- RawByteString for byte storage (e.g. for FileFromString() function or BLOB temporary storage);

- some special conversion functions to be used for Delphi 2009/2010/XE UnicodeString (defined inside {$ifdef UNICODE}...{$endif} blocks);

- never use AnsiString directly, but one of the types above.

I would like to use Zeos as the database layer for my upcoming project, named mORMmot - http://synopse.info/forum/viewtopic.php?id=143

My proposal is that I could make a whole code refactoring using UTF-8 as internal format of the ZEOS layer, with some types similar to the one I quoted above.
Then the team could take it as a start.

In all cases, I'll start a fork of ZeosLib for our little mORMot.
What do you think about that?