Zeoslib Portal

miab3,

hope i got it now R2006.

Question, Michal. What do you think would be a nice RowSize if i address all Strings and Bytes as Pointer?

on 64Bit OS a Pointer has 8Bytes on 32Bit a pointer has 4 Bytes. The biggest values currently (on my working copy) is either stBytes (Pointer+Smallint(Length)) or Extendend(10Bytes).

an 8KB Row could pick up 1024/2048 columns with Data(x) on 64Bit/32Bit and so on.
That's the first step for full Mem-optimazion i want to have. Currently i've no idea how to get a varying size of the RowAccessor running without speed decrease. But moving the strings out of that array and only address reserved memory is equal fast but eats only the Memory for the strings we need (except the ByteArray 'em selves).

I'll test it on 64Bit now if it's equal stable..

So what do you think are 4KB enough or should i prefere 8KB/row? In the current stage i could also add defines bud they are ugly..

EgonHugeist,

After R2006 Lazarus for Linux compiles without errors.

On the transition to pointers:
- Whether it is really will be faster?
- In accordance with the standards of db servers need more, such as 4096 fields for MySQL:
http://database-management-systems.find ... d-vs-MySQL
- I am using 128kB buffer without losing speed.

Michal

miab3,

http://database-management-systems.find ... d-vs-MySQL

are there such limitations for MsSQL, ADO, Oracle? I don't know it yet.

I am using 128kB buffer without losing speed.

Well this i know, Michal. Because you propsed this patch before.
But that means: 1.000.000 Rows * 128KB = ~122GB Memory. Which means the OS is starting moving memory to the Harddisk (if so) and this will slow down the performance.

I have a lightning fast Laptop with 32Gb but such an select would kill all performance. Now we've protocols like SQLite or MySQL which are fetching all data once (libmysql does it, then you got your result).

My intention is/was to store only pointers to strings or byte arrys. The target pointer contains allocated Memory. No string no memory. Long/very large strings more memory. Equal to Lob-Streams. Actually we've a patch inside the sources which says: (MySQLUtils.pas) Such a limitation was made to avoid the RowbufferWidthExceeded exception. What would happen on selecting 20 of such field? They won't fit into your/our RowBuffer. But that can simply be done on using only Memory addresses like Streams are doing this.

Exacly this i've ready now (works on Win64 too). Remainig would be a varying size instead of a fixed (currently 64Kb on my side 128 on yours) of the RowBuffer to handle x Fields and the memory which sould be used.

Maybe i commit first if the last goal was also solved?

EgonHugeist,

Have you tried?
This is only the maximum size of the buffer that virtually no change in the size of consumed memory and speed.

Michal

miab3,

nope i didn't test it. But i can read the souces and each fetched row pointer of an virtual or cached resultset will be added to a TList. Then we can navigate between the columns and it doesn't matter if the provider supports forward-only fetching or whatever.

Did you execute TZQuery.Last or TZQuery.Recordcount if you've select a very huge count of rows + columns?

Well but i will test it also by my selves. IT would be interesting if we've an optimation or not.. On the other hand: never change a running system (((: and i'll trash the approach.

Hi !

I'm finally finishes the project on Linux (xubuntu), now I going to do same on Win7,
but got an error while installing on Lazarus:

C:\lazarus_svn\zeos7_test\src\core\ZEncoding.pas(384,35) Error: Identifier not found "MultiByteToWideChar"
C:\lazarus_svn\zeos7_test\src\core\ZEncoding.pas(387,30) Error: Identifier not found "MultiByteToWideChar"
C:\lazarus_svn\zeos7_test\src\core\ZEncoding.pas(400,35) Error: Identifier not found "MultiByteToWideChar"
C:\lazarus_svn\zeos7_test\src\core\ZEncoding.pas(400,54) Error: Identifier not found "MB_PRECOMPOSED"
C:\lazarus_svn\zeos7_test\src\core\ZEncoding.pas(403,30) Error: Identifier not found "MultiByteToWideChar"
C:\lazarus_svn\zeos7_test\src\core\ZEncoding.pas(403,49) Error: Identifier not found "MB_PRECOMPOSED"
C:\lazarus_svn\zeos7_test\src\core\ZEncoding.pas(438,35) Error: Identifier not found "WideCharToMultiByte"
C:\lazarus_svn\zeos7_test\src\core\ZEncoding.pas(441,30) Error: Identifier not found "WideCharToMultiByte"
C:\lazarus_svn\zeos7_test\src\core\ZEncoding.pas(456,35) Error: Identifier not found "WideCharToMultiByte"
C:\lazarus_svn\zeos7_test\src\core\ZEncoding.pas(457,31) Error: Identifier not found "WC_COMPOSITECHECK"
C:\lazarus_svn\zeos7_test\src\core\ZEncoding.pas(457,47) Error: Identifier not found "WC_DISCARDNS"
C:\lazarus_svn\zeos7_test\src\core\ZEncoding.pas(457,62) Error: Identifier not found "WC_SEPCHARS"
C:\lazarus_svn\zeos7_test\src\core\ZEncoding.pas(457,79) Error: Identifier not found "WC_DEFAULTCHAR"
C:\lazarus_svn\zeos7_test\src\core\ZEncoding.pas(461,30) Error: Identifier not found "WideCharToMultiByte"
C:\lazarus_svn\zeos7_test\src\core\ZEncoding.pas(462,31) Error: Identifier not found "WC_COMPOSITECHECK"
C:\lazarus_svn\zeos7_test\src\core\ZEncoding.pas(462,47) Error: Identifier not found "WC_DISCARDNS"
C:\lazarus_svn\zeos7_test\src\core\ZEncoding.pas(462,62) Error: Identifier not found "WC_SEPCHARS"
C:\lazarus_svn\zeos7_test\src\core\ZEncoding.pas(462,79) Error: Identifier not found "WC_DEFAULTCHAR"
C:\lazarus_svn\zeos7_test\src\core\ZEncoding.pas(640,21) Error: Identifier not found "GetACP"
C:\lazarus_svn\zeos7_test\src\core\ZEncoding.pas(659,33) Hint: Mixing signed expressions and longwords gives a 64bit result
C:\lazarus_svn\zeos7_test\src\core\ZEncoding.pas(902) Fatal: There were 19 errors compiling module, stopping

Lazarus 1.1 svn, Fpc 2.6.1 svn, Win7 Starter. (with FPCUP), Zeos7 svn testing from now.
Last week I can't remember the rev.no. (below 2000) was ok.

patyit,

hmm their is the unit windows missing. Did compile and test it with fpc2.6.0/LCL1.02 today. No issues to see. Current rev: 2007 can you please check it again? If the issue remains where can i download an FPC2.6.1?

EgonHugeist,

You are right, windows unit is //-ed by me someday when I have problems with compiling Lazarus on Linux ... and normally forget it ...

Sorry for that ! It is now all right.
(tested with rev. 2008 on Win7 Starter and XUbuntu 12.04 32bit)

By the way the easiest way to install latest Fpc and Lazarus is usung FPCUP utility from :
https://bitbucket.org/reiniero/fpcup/downloads

EgonHugeist,

Zeos7.0.1-beta branches_ testing_R2010:
http://svn.code.sf.net/p/zeoslib/code-0 ... s/testing/
compiles and runs on (I have tested):

- D2006,
- D2007 -> C++,
- DXE2 32/64 -> C++(32),
- Lazarus Win 1.1/fpc 2.7.1 32/64,
- Lazarus Lin 1.1/fpc 2.7.1 64 on Debian 64-bit (LMDE 201204).

(Firebird 2.5.3, MySQL 5.5.18, MariaDB 5.5.28, Oracle 11g EE, PostgreSQL 9.2.1, MSSQL 2008 R2 Express)

Michal

EgonHugeist wrote:
On non Windows plattforms do the functions i call not exsits. Thats why i've added the LConvEncoding unit from Lazarus. But actually we are not allowed to provide this unit(which would happen without Lazarus) so i've added the functionality with the WITH_LCONVENCODING define in ZeoaLazarus.inc. (we need this unit only on non MsWindows plattforms)

So uncomment:

{.$IFNDEF MSWINDOWS}
{.$DEFINE WITH_LCONVENCODING} //Only available for Lazarus to activate the LConv uncomment this define
{.$ENDIF}

I believe that you can use {$IFDEF LCL} to make difference between Lazarus (LCL dependency) and FPC without Lazarus (no LCL dependency).

Then we know that LConvEncoding unit is here (it is part of LCL), as the project has LCL dependecy, so we don't have to include it with Zeos.

So, instead of commenting defines, here I would recommend something like this:
And generally, as LCL controls use UTF8, and FPC by itself does not, you often need to distinguish FPC projects which have LCL dependency (that is, Lazarus projects) from those which don't have it. You can ask for LCL define.

patyit, miab3,

thanks for confirmation!

Zoran,

uh this sounds great! I'll test it right now. If you are right (i think so) then i know some more places where such an define can bea helpfull.

Question: Is this define provided by FPC or an build in define on compiling the Project?

EgonHugeist wrote: Question: Is this define provided by FPC or an build in define on compiling the Project?

It is mentioned in Lazarus wiki - http://wiki.lazarus.freepascal.org/Code ... .2F_macros

I don't know where it is defined, but I tested to see if it is defined in a Lazarus application.

The test, from Lazarus main menu chose:

1. chose Project->New Project->Application
add two lines to test fpc and lcl defines, you can put it at the beginning of lpr file: This is the output:

yes FPC
yes LCL

2. Now chose Project->New Project->Program
add the same two lines in program, now we have: Now, the output is this:

yes FPC
NO LCL

So, the LCL projects do have this LCL define, non-LCL projects do not have it. Therefore, I think that it is just what we need.

Zoran,

thanks for the details. Yes i found it too: http://wiki.freepascal.org/Code_Conversion_Guide

So i'll start some rework. BTW: http://svn.code.sf.net/p/zeoslib/code-0 ... esting-7.1 has been started. If you doubt... ((:

EgonHugeist,

Zeos7.1-beta branches_ testing_R2018:
http://svn.code.sf.net/p/zeoslib/code-0 ... sting-7.1/

In ZAbstractConnection.pas line 335

Unnecessary (disturbing):

FUTF8StringAsWideField := False;

Michal

EgonHugeist,

Michael,
Is it possible to speed up the update/modify SQLite with the new CAPI?
How?

Michal