Accelerating OpenERP accounting: Precalculated period sums

Borja Lpez Soiln http:// !"ami!es

#nde$
%urrent approach &sum o' entries(

%urrent approach e$plained! Per'ormance anal)sis! Alternati,e -: Accumulated ,alues using triggers

Proposal: *Precalculated period sums+

Proposed .) /erdinand 0assauer &%hricar( Proposed .) Borja L!S! &34eoPolus(

Alternati,e 1: Period totals using the OR2

%urrent approach ,s Precalculated period sums

%urrent approach: Sum o' entries

%urrentl) each time )ou read the credit/de.it/.alance o' one account OpenERP has to recalculate it 'rom the account entries &mo,e lines(! 5he magic is done .) the *_query_get()+ method o' account.move.line6 that selects the lines to consider6 and the *__compute()+ method o' account.account that does the sums!

#nside the current approach

_query_get() filters: .uilds the *78ERE+ part o' the S9L :uer) that selects all the account mo,e lines in,ol,ing a set o' accounts!

Allo s to do comple$ 'ilters6 .ut usuall) loo" li"e *include non;dra't entries 'rom these periods 'or these accounts+!

__compute() sums: uses the 'ilter to :uer) 'or the sums o' de.it/credit/.alance 'or the current account and its children!

<oes just one S9L :uer) 'or all the accounts! (nice!) 8as to aggregate the children ,alues on p)thon!

Sample :uer) done .) __compute

SELECT l.account_id as id, COALESCE(SUM(l.debit), 0) as debit, COALESCE(SUM(l.credit), 0) as credit, COALESCE(SUM(l.debit),0) COALESCE(SUM(l.credit), 0) as balance FROM account_move_line l
Account = children > lot o' ids?

WHERE l.account_id ! (2, 3, 4, 5, 6, ..., 1648, 1649, 1650, 1651) A!" l.state #$ %dra&t% A!" l.'eriod_id ! (SELEC( id )*OM account_'eriod +,E*E &iscal-ear_id ! (.)) A!" l.move_id ! (SELEC( id )*OM account_move +,E*E account_move.state / %'osted%) GROUP BY l.account_id

Sample :uer) plan

0UE*1 2LA! --------------------------------------------------------------------------------------------------------------------,as3A44re4ate -$ (cost/56.78..56.75 ro9s/. 9idt3/.7) (cost/;5.00..56.7< ro9s/. 9idt3/.7)

!ested Loo' Semi :oin

:oin )ilter= (l.'eriod_id / account_'eriod.id) -$ !ested Loo' -$ (cost/;5.00..56.5< ro9s/. 9idt3/<<) (cost/;5.00..;5.0. ro9s/. 9idt3/;)

@gh?6 se:uential scan on a ta.le ith &potentiall)( lots o' records!!! :&

,as3A44re4ate -$

Seq Scan on account_move

(cost/0.00..;5.00 ro9s/. 9idt3/;)

)ilter= ((state)==te>t / %'osted%==te>t) -$ nde> Scan usin4 account_move_line_move_id_inde> on account_move_line l nde> Cond= (l.move_id / account_move.id) )ilter= (((l.state)==te>t #$ %dra&t%==te>t) A!" (l.account_id / A!1 (%A<,8,;,5, ..., .@;?,.@50,.@5.B%==inte4erCD))) -$ nde> Scan usin4 account_'eriod_&iscal-ear_id_inde> on account_'eriod nde> Cond= (account_'eriod.&iscal-ear_id / .) (cost/0.00..0.<? ro9s/. 9idt3/;) (cost/0.00...<.;? ro9s/. 9idt3/<@)

Per'ormance Anal)sis %urrent approach .ig O -/1

*Selects all the account mo,e lines+ 5he :uer) comple$it) depends on l6 the num.er o' mo,e lines 'or that account and &recursi,e( children: O(query) > O(f(l)) *8as to aggregate the children ,alues+ 5he comple$it) depends on c6 the num.er o' children! O(aggregate) > O(g(c))

%urrent approach .ig O 1/1

O(__compute) = O(query) + O(aggregate) O(__compute) > O(f(l)) + O(g(c)) 7hat "ind o' 'unctions are f and gA LetBs do some empiric testing &'unnier than maths6 isnBt itA(!!!

LetBs test this chart!!! -/1

5he o''icial Spanish chart o' accounts6 hen empt): 8as a.out -CDD accounts! 8as E le,els! (to test this chart of accounts install the l10n_es module)

LetBs test this chart!!! 1/1

8o man) accounts .elo each le,elA
Account code 4um.er o' children &recursi,e( D C GH -I1 -CJK

Le,el E F GHDDDD &lea' account( Le,el G ; GHDD Le,el H ; GHD Le,el 1 ; GH Le,el - ; G Le,el D F D &root account(

5o get the .alance o' account *G+ e need to sum the .alance o' -I1 accounts?

O"6 loo"s li"e the num.er o' children c has a lot o' in'luence6 and the num.er o' mo,es l has little or zero in'luence6 4(c) $$ &(l) Lets split them!!!

4o it is clear that 4(c) is linear? &note: the nL o' children gro s e$ponentiall)( O(4(c)) = O(c)

So6 the in'luence as little6 .ut linear too? O(&(l)) = O(l)

Big O ; %onclusion
O(__compute) = O(l) + O(c) c has an une$pectedl) .ig in'luence on the results >M Bad per'ormance on comple$ charts o' accounts? c does not gro ith time6 .ut l does!!! >M OpenERP accounting .ecomes slo er and slo er ith time? &though itBs not that .ad as e$pected(

Proposal: Precalculated sums

OpenERP recalculates the de.it/credit/.alance 'rom mo,e lines each time! 2ost accounting programs store the totals per period &or the cumulati,e ,alues( 'or each account! 7h)A

Reading the de.it/credit/.alance .ecomes much 'aster! !!!and reading is much more data intensi,e than riting:

Accounting reports read lots o' times lots o' accounts! Accountants onl) update a 'e accounts at a time!

#tBs reall) 'asterA

Precalculated sums per period means:

O('):uer) &get the de.it/credit/.alance o' each period 'or that account( instead o' O(l):uer)6 ith ' .eing the num.er o' periods6 ' ## l. @sing opening entries6 or cumulati,e totals6 ' .ecomes constant >M O(.) #' aggregated sums & ith children ,alues( are also precalculated6 e donBt ha,e to do one O(c)aggregation per read!

#tBs O&-( 'or reading?? ( ut creating!editing entries is a it slo"er)

Alternati,e -: Accumulated ,alues using triggers &#(

Proposed .) /erdinand 0assauer! 8o does it or"A

4e o.ject to store the accumulated de.it/credit/.alance per account and period &letBs call it account.period.sum(!
Opening 2o,e line ,alues in period Nalue in ta.le GDD GDD -st =1DD6 =ED CED 1nd =1E CJE Hrd ;GDD 1JE Gth =1E6 =1DD EDD

5riggers on Postgres &#$!pg%&$( update the account_period_sum ta.le each time an account mo,e line is created/updated/deleted!

Alternati,e -: Accumulated ,alues using triggers &##(

8o does it or"A&cont!(

5he data is calculated accumulating the ,alues 'rom pre,ious periods! &/erdinand protot)pe re:uires an special naming o'
periods 'or this(!

%reates S9L ,ie s .ased on the account account_period_sum ta.le! /or reports that sho data aggregated .) period:

4e reports can .e created that either directl) use the S9L ,ie s6 or use the account!period!sum o.ject!

5he account.account.__compute() method could .e e$tended to optimize :ueries &modi'ied to ma"e use o' the account_period_sum hen possi.le( in the 'uture!

Alternati,e -: Accumulated ,alues using triggers &###(

0ood points
5riggers guarantee that the data is al a)s in s)nc! 5riggers are 'ast!
Protot)pe a,aila.le and or"ing? ; 'used this method
already in very ig installations ( some 100 accountants some millions moves "ithout any pro lems) (*erdinand)

Bad points
<ata.ase dependent triggers! 5riggers are harder to maintain than P)thon code! 2a"es some assumptions on period names!

&e,en i' some.od) rites directl) to the data.ase?(

&as OpenERP currentl) does not 'lag opening periods apart 'rom closing ones(

Alternati,e 1: Period totals using the OR2 &#(

Proposed .) Borja L!S! &34eoPolus(! 8o does it or"A

4e o.ject to store the de.it/credit/.alance sums per account and period (and state)+
Opening 2o,e line ,alues in period Nalue in ta.le GDD GDD -st =1DD6 =ED 1ED 1nd =1E 1E Hrd ;GDD ;GDD Gth =1E6 =1DD 11E

E$tends the account.move.line open o.ject to update the account.sum o.jects each time a line is created/updated/deleted!

Alternati,e 1: Period totals using the OR2 &##(

8o does it or"A&cont!(

E$tends account.account.__compute() method to optimize :ueries:

#' the :uer) 'ilters onl) .) period/'iscal )ear/state6 the data is retrie,ed 'rom the account.sum o.ject! #' the :uer) 'ilters .) dates6 and one ore more 'iscal periods are 'ull) included on that range6 the data is retrie,ed 'rom 'or the account.sum o.jects &'or the range co,ered .) the periods( plus the account.move.lines &the range not co,ered .) periods(! /iltering .) e,er) other 'ield &'or e$ample partner_id( causes a 'all.ac" into the normal __compute method!

Alternati,e 1: Period totals using the OR2 &###(

0ood points <ata.ase independent! Optimizes all the accounting! /le$i.le! 4o #$!pg%&$ triggers re:uired6 just P)thon >M Easier to maintain! Bad points <oes not guarantee that the sums are in s)nc ith the mo,e lines!

&.ut no.od) should directl) alter the data.ase in 'irst place!!!(

P)thon is slo er than using triggers!

4o protot)pe )et? :(
&But ta"e a loo" at 5r)ton stoc" :uantit) computation(

%urrent approach NS Period sums

%urrent approach Pros

Precalculated sums Pros

4o redundant data! Simpler :ueries! Slo !

/ast6 al a)s! <rill;do n na,igation! 4eed to "eep sums in s)nc ith mo,e lines! 2ore comple$ &__compute( or speci'ic :ueries to ma"e use o' the precalculated sums!

%ons

%ons

Reports and dash.oard charts/ta.les are per'ormance hungr)!

Becomes e,en slo er ith time!

#recalculated sums , -rill do"n navigation (.hricar prototype) 1!/

#recalculated sums , -rill do"n navigation (.hricar prototype) 0!/

#recalculated sums , -rill do"n navigation (.hricar prototype) /!/

And one last remar"!!!

!!!all this is applica.le to the stoc" :uantities computation too?