[Q] Unicode / Delphi 2009 / ZEOS 7.x

TheEdge · Post by **TheEdge** » 27.12.2008, 09:57

G'Day,

I find myself in the position of desperately needing Delphi 2009 support (without UNICODE) and as the 7.x is still ALPHA and as far as I can see still fundamentally broken (Notwithstanding Mark's quick responses to my problem of CHAR() and VARCHAR() fields getting chopped).

I was wondering why 7.x is taking such a big step in one hit viz:

1. Delphi 2009 Support AND
2. UNICODE support offered by 2009 AND
3. Restructuring

I know Borland changed the meaning of the string reserved word which makes it difficult. However I was wondering why as a starting point for 7.x the 6.x code base was taken with the following change(s):

1. Add ZEOS controlled string types as follows:

All versions of Delphi < 2009

TZString_WideString = WideString;
TZString_Ansi = string;
TZString_Unicode = TZString_WideString;

And then for Delphi >= 2009

TZString_Unicode = string;
TZString_Ansi = AnsiString;
TZString_WideString = WideString;

2. Amend all string and widestring references:

string --> TZString_Ansi
widestring --> TZString_WideString

Why do this?:

1. Existing stable 6.6.x code base can be used in Delphi 2009 (with existing UNICODE support)
2. Any code that is to be transferred from 6.6.x to 7.x will work as the string type is abstracted and under ZEOS control

So what am I missing here? Because I am sorely tempted to make the 6.x branch compile under 2009 as I need this working now. :-)

TheEdge · Post by **TheEdge** » 28.12.2008, 03:41

So to prove my theory I did as I described above in terms of the string replacements and then added the following to ZCompatibility:

Code: Select all

{$IFDEF VER200}
   TZChar_Ansi = AnsiChar;
   TZString_Ansi = AnsiString;
   TZString_Unicode = string;
   TZString_WideString = WideString;
{$ELSE}
   TZChar_Ansi = Char;
   TZString_Ansi = string;
   TZString_Unicode = WideString;
   TZString_WideString = WideString;
{$ENDIF}

and got it all compiling and installed in Delphi 6 & 2009. Simple testing with my existing test application (see my other post) looked like it all worked against my database.

However as I don't have the time to maintain this or track issues I am going to use an ODBC connection in the mean time until ZEOS can support Delphi 2009.

I hope to come back to ZEOS and work on some of the speed issues reported as there appears to be some string handling functionality that could be improved performance wise. Whether this leads to an overall improvement profiling will reveal this ultimately.

Post by **mdaems** » 31.12.2008, 10:52

You're right,

That's why we don't do these 3 steps at once.
7.0 will probably only contain the basic D2009 support and some smaller changes. So there will be no full Unicode support. This would take too long to implement.

My current planning :
- merge testing branch to trunk in the first week of 2009.
- at the same time new snapshots will be published.
- first official alpha release at the end of january

BUT : this means we need feedback between these steps. I can't believe there will be no obvious problems with these individual steps. So no feedback means 'nobody' needs it...

Mark

Mark

TheEdge · Post by **TheEdge** » 01.01.2009, 01:34

Hi Mark,

With the merge of testing back to trunk are you planning to create TZ..... string types so that you have better control over what kind of string is really in use?

This for instance would have taken care of things that were highlighted in the 557 change viz:

333 - Result := SizeOf(Char)*ColumnInfo.Precision + 1;
333 + Result := SizeOf(Char)*ColumnInfo.Precision + SizeOf(Char);

Yes SizeOf() is the better way to achieve the size of 1 required, but what happens should this still need to be ANSI and the compiler "decides" that Char is really a UNICODE char and returns 2 bytes to you? Much better to have this under your direct control.

I realise this will be more effort in the short term but I believe it will pay dividends in the long term especially in terms of finding subtle memory overwrites etc.

Just my 2c worth.

Post by **mdaems** » 03.01.2009, 14:49

Hi,

I'm sorry, but I do NOT see the benefit of adding separate string types in solving this 557 case. I think you would have always needed SizeOf unless you also used AnsiString in D2009 in that place. But I may be wrong, of course. I'm not a Delphi specialist at all. If you are, you're very welcome to join the team (but if you are not, you're welcome equally much).

But thinking about this proposal I think it may be the way to go when enabling full unicode support.

Changing 6.6 code drastically is not an option. It's stable and should stay like that.

If you want to do the effort, we can add a 6.7 branch. Which would be 6.6 with your changes only.
Conditions :
- 6.7 should pass all test suite tests that 6.6 passes
- 6.7 must go through als phases an other version goes through dev/alpha/beta/stable. The cycle can be short however, if all goes well.
- I need a 'proof of concept' before I start this procedure. Seems you have done the changes already. Can I have a copy I can fire the test suite on?
- You or somebody else must take the responsibility to maintain that version with patches done to 6.6.

If this proves to work we'll have to do the execercise for the 7.0 version. As long as we're not in beta phase that's very well possible. It will make the road to 7.0 longer, but the need for a version 7 will be much less.

Mark

mse · Post by **mse** » 04.01.2009, 18:09

mdaems wrote: I'm sorry, but I do NOT see the benefit of adding separate string types in solving this 557 case.

We used the approach with an own string type definition with success in MSEgui. MSEgui uses the types msestring and msechar for almost all strings. At the start of the development msestring was mapped to widestring (msestring = widestring). Later I switched to UTF8String (msestring = UTF8String) because of the widestring problems of FPC. After two months I switched back to widestrings and wrote bugreports until FPC widestrings worked because I found that widestrings are more user friendly and easier to handle than UTF8Strings as base encoding of a GUI framework.
Currently msestring = widestring. If the new FPC UnicodeString (16 bit string, referencecounted on all platforms) is available in a FPC release, msestring will be mapped to UnicodeString.
The replacement of the stringtype is less complicated if there is a separate type definition.

Martin

TheEdge · Post by **TheEdge** » 07.01.2009, 07:04

mdaems wrote:Hi,

I'm sorry, but I do NOT see the benefit of adding separate string types in solving this 557 case. I think you would have always needed SizeOf unless you also used AnsiString in D2009 in that place. But I may be wrong, of course. I'm not a Delphi specialist at all. If you are, you're very welcome to join the team (but if you are not, you're welcome equally much).

But thinking about this proposal I think it may be the way to go when enabling full unicode support.

Changing 6.6 code drastically is not an option. It's stable and should stay like that.

If you want to do the effort, we can add a 6.7 branch. Which would be 6.6 with your changes only.
Conditions :
- 6.7 should pass all test suite tests that 6.6 passes
- 6.7 must go through als phases an other version goes through dev/alpha/beta/stable. The cycle can be short however, if all goes well.
- I need a 'proof of concept' before I start this procedure. Seems you have done the changes already. Can I have a copy I can fire the test suite on?
- You or somebody else must take the responsibility to maintain that version with patches done to 6.6.

If this proves to work we'll have to do the execercise for the 7.0 version. As long as we're not in beta phase that's very well possible. It will make the road to 7.0 longer, but the need for a version 7 will be much less.

Mark

I would love to join the DevTeam. However I am time poor at the moment so my contributions would be sporadic at best. I was wondering why the need for 6.7, but rather add the string types to 7.x as that is in a changing state anyway. Units can be done piecemeal as people get to them as changing the string types is not a onerous or big job, just tedious.

gto · Post by **gto** » 07.01.2009, 16:14

Thinking here, setting our own string type seems the way to go.

it took me long to see this, by the way

TheEdge · Post by **TheEdge** » 24.01.2009, 00:50

Any thoughts on my post above gto's? I would like to join the dev team but have heard nothing. :-)

Post by **mdaems** » 29.01.2009, 22:50

However I am time poor at the moment

TheEdge,

Sorry, but I was thinking the timing wasn't quite right at the moment.

The 6.7 idea I did was just for when it all must go very quick to get a D2009 compatible stable release. So I really would prefer it to get into the 7.0 version. (to avoid the mess with another branch to maintain is my selfish motivation)

I have been experimenting with your proposal myself for some time, but for one reason or another, this didn't work out very well.

So if you can send me a working SVN diff against Testing branch with your (eventually partial) changes, please do so. That way I can push it through the test suite to see how that reacts on different compilers/databases.

You're very welcome on the team, but I prefer to check what's coming in before accepting you (or anyone) fully with the necessary rank and rights.

Mark

TheEdge · Post by **TheEdge** » 30.01.2009, 06:48

Hi Mark,

I understand. Will send you my diff's. I would like to test against mySQL while I am doing this, so could you point me in the right direction so I can use the test suite to at least validate that much before pushing the change to you.

My selfish reason being that I want this to work on mySQL first.

Another question I had about 7.x is whether there is a structure change in the framework? I have seen posts and suggestions to that effect, was just wondering if there was a decision made or a working document or similair? My main reason for this is that the other thing I want to validate / improve is the speed of the framework as in some limited testing I got *better* performance out of BDE/Paradox than I did out of ZEOS/SQLite. Now I am not saying ZEOS is the bottleneck but I want to quantify it at least and improve it if I can.

Post by **mdaems** » 30.01.2009, 08:39

You're on the same wavelength as gto... I'm not really a performance specialist. But if you have good proposal, just fire it.

For the moment we did not do a real structure change. Apart from splitting a big Metadata Class and adding support for prepared statement execution. The last one's only effectively working for mysql now as that's the database I'm using too. (Testing branch, will be merged into trunk next week.)

The test suite... to do it in an automated way you need Apache Ant to run the build.cmd script in the build directory. Then you only need to change the build/build.properties and database/test.properties files. Executing the script cleans, builds the components, builds the testframework and the test projects and fin ally runs the tests themselves, logging everything in the logs directory.

To do it manually:
- Open the zeosdbodev package group
- install the components as usual
- install the test framework package, which depends on the components installed first, so needs a reinstall after most changes, I think
- compile the test programs
- adjust the database/test.properties for your server
- run the test exe's from the packages/xxx/build directory (use command line). It uses a relative path to the properties file so you can't just put them where you want.

There are some documentation files in the doc directory from the official release. (It's also in svn but there you have only the xml sources)

Mark

EDIT : some failing tests are 'normal' so you should run it on an unchanged zeoslib version first and compare the resulting logs afterwards.