Académique Documents
Professionnel Documents
Culture Documents
1
MarkRittman,Director,RittmanMeadConsulting forCollaborate09,Florida,USA,May2009 TheOracleBIApplications7.9.5consistsofapredefineddimensionaldatawarehouse,ETLroutines,an OracleBIEnterpriseEditionrepositoryandexampledashboardsandreports.TheETLroutinesarebuilt usingInformaticaPowerCenterandarescheduledandorchestratedusingtheOracleDataWarehouse AdministrationConsole. ThedatawarehouseprovidedwiththeOracleBIApplicationsisdesignedtobedeployedoneither OracleDatabaseEnterpriseEdition,MicrosoftSQLServerorIBMDB/2.Assuch,whilstitusescommon datawarehousingfeaturessuchasbitmapindexes,itdoesnotmakeuseofanyOraclespecificfeatures suchassegmentcompression,partitioningormaterializedviews.Itispossiblehowevertomakeuseof thesefeatures,andthispapersetsoutamethodologyfortheirusewithOracleBIApplications7.9.5, alongwiththeOracleDataWarehouseAdministrationConsole10.1.3.4orhigher. AnOracleBusinessAnalyticsWarehouseOverview TheOracleBusinessAnalyticsWarehouseconsistsofanumberofstagingandpresentationtablesthat togethersupporttheloadingandqueryingofenterprisedataviaaconformeddimensionalmodel. Tablesarecreatedasregularheaptables,withaminimalamountofNOTNULLcheckconstraintsbutno primarykeys,foreignkeys,partitionsorotheradditionalitemsofmetadata.Tablesareloadedvia InformaticaPowerCenter8.1.1,usingthePowerCenterIntegrationServiceandrowbyrowdataloading. Aggregatetablesarecreatedandpopulatedtosupportkeyfacttables,usingseparateETLprocesses afterthemainfacttableloadsthattruncate,andthenrebuildtheaggregates. ItishoweverpossibletocustomizetheOracleBusinessAnalyticsWarehousetotakeadvantageof featuressuchassegmentcompression,partitioning,materializedviewsandotherOracledata warehousefeatures.Itisalsopossibletoaddadditionalmetadatasuchasprimarykeyandforeignkey constraints,dimensionsandotherfeaturestosupportmoreefficientqueryingofdetailleveland summarizeddata.ToillustratehowtheseOraclefeaturescanbeusedtooptimizetheloadingand queryingoftheOracleBusinessAnalyticsWarehouse,thispaperwilltakeonofthefacttableswithinthe datawarehouseandapplythesetechniquestoit. PerformanceOptimizationScenario TheOracleBusinessAnalyticsWarehousecontainsatablecalledW_SALES_INVOICE_LINE_Fthat containsfactdataonthesalesinvoicesgeneratedbythebusiness.Itissupportedbyanaggregatetable, W_SALES_INVOICE_LINE_Athattakesdatafromtheoriginaltableandsummarizesittoimprove queryperformance.Inthesampledatasetusedinthispaper,thesetwotableshadthefollowingrow countandsize.
select segment_name , bytes/1024/1024 "Size in MB" from user_segments where segment_name in ('W_SALES_INVOICE_LINE_F, 'W_SALES_INVOICE_LINE_A'); SEGMENT_NAME Size in MB ------------------------------ ---------W_SALES_INVOICE_LINE_A 9 W_SALES_INVOICE_LINE_F 189.125
ThesetablesareloadedbytwoDACtasks,andaDACTaskGroup: 1. TASK_GROUP_Load_SalesFactcallsthefollowingSILandPLPtasks,andwhentheyhave completedrecreatesanyindexesrequiredforsupportingqueries. 2. SIL_SalesInvoiceLinesFact,whichinitiallydropstheindexesonthefacttable,thencalls eithertheSIL_SalesInvoiceLinesFactandSIL_SalesInvoiceLinesFact_Full Informaticaworkflowsforincrementalandfullloadsrespectively,andthenrecreatesjustthose indexesrequiredfortherestoftheETLprocess. 3. PLP_SalesInvoiceLineAggregate,whichagaindropsindexesthistimeontheaggregate table,thencallseitherthePLP_SalesInvoiceLinesAggregate_Loadand PLP_SalesInvoiceLinesAggregate_Load_FullInformaticaWorkflowsforincremental andfullloadsoftheaggregatetable,thenrecreatestheindexesrequiredfortherestoftheETL process. InitialBenchmarks Tocreatebaselinefigurestocompareyouroptimizationsto,starttheOracleDataWarehouse AdministrationConsole10.1.3.4orhigher,andcreateanewsubjectareawithintheDACRepositorythat usesthetwotasksandonetaskgrouplistedabove.
fulltablescanscanbeperformedfasteraslessblocksarerequiredtoretrieveallthetables data.
TotestthebenefitsofcompressingtheW_SALES_INVOICE_LINE_Ffacttable,firsttruncateitandthen alterthetabletoaddcompression.
SQL> truncate table w_sales_invoice_line_a; Table truncated. SQL> alter table w_sales_invoice_line_f compress; Table altered.
YoucanthenrestoretheETLsourcetablesbacktotheiroriginalstateandrunthefull,andthen incrementalloadsintothefactandtheaggregatetablesinordertotestthatcompressionisworkingas expected.Notethattableswillonlybecompressedwhendataisinserted,asInformaticaPowerCenter bydefaultusesbulkloadfunctionalityperformtableinserts.Updates,ormixedinsert/updateloadswill notbenefitfromcompressionasInformaticawillreverttoconventionalpathinserts,andofcourse updatesremovecompressionfromOracletablesunlessyouareusingtheAdvancedCompression Optiontothedatabase. PartitioningtheFactTable PartitioningisanoptiontotheEnterpriseEditionoftheOracleDatabasethatallowsyoutosplitone largephysicaltableintoseveralsmallerphysicalones,withusersstillseeingitasonebigtablebutgiving youfarmoreflexibilityinhowyoucanstoreandmanagedatawithinit.Partitioningistypicallyused withlargefacttablesandallowstheDBAtoassigneachpartitiontoseparatetablespaces,whichcan thenbestoredondifferentphysicaldiskunitsandbackedupindependently.Aswithtablecompression though,unfortunatelytheDataWarehouseAdministrationconsoledoesnothaveanyconceptof partitioningandyouwillthereforehavetocarryoutsomeadditionalstepstousethisfeature. TablessuchastheW_SALES_INVOICE_LINE_FtablearenormallycreatedbytheDACadministrator wheninitiallyinstallingtheOracleBusinessAnalyticsWarehouse,byselectingTools>ETLManagement >ConfigurefromtheDACmenu.HoweverthereisnoprovisiontocreatetablesusingthePARTITION BY(orCOMPRESS)clauses,andsowecaneithercreatethetableoutsideoftheDAC,aswedidinthe previousstepfortheCOMPRESSclause,orwecanusetheActionsfeatureinDAC10.1.3.4.1tocreate ourtableforus,usingtherequisiteclause,beforewetryanddoafullloadintothetable. Actionsareanewfeatureofthe10.1.3.4.1versionoftheDACandallowustocreatetable,indexand taskactions. TableactionsallowustooverridetheTruncateandAnalyzestepscarriedoutontablesduring anETLprocess Indexactionsallowustooverridethecreationanddroppingactionsassociatedwithindexes TaskactionsallowustoexecuteSQLandPL/SQLstepsbeforeorafterataskexecutes.
UsingtheactionValuedialog,createtwosteps,onetodropthetableandtheothertocreateit.
MakesurethedropstepislistedabovethecreatestepissettoContinueonFail,andenterthe followingSQLstatementtodropthetable:
drop table w_sales_invoice_line_f
DonotplaceanysemicolonsattheendoftheSQLscriptasthiswillcauseittofailwhenrun. Eventhoughthisnewtaskactionwilldropandthenrecreate,usingpartitioning,the W_SALES_INVOICE_LINE_Ftable,theDACstillholdsdetailsofitinitsrepositorytogetherwithdetailsof theindexesthatareassociatedwithit.Asthingsstand,theDACwoulddroptheseindexesaspartofthe SILtaskandrecreatethemusingthetaskgrouptask,howeveritdoesnotnormallyunderstandthe conceptoflocalindexesandwilltryandcreatethemwithoutanylocalorglobalclause,whichhasthe effectofcreatingthemasglobalindexes.ToinstructtheDACtocreateourindexesaslocalindexes,you nowneedtocreateanIndexActiontooverridethenormalindexcreationprocessfortheseindexes. Thefirststepinthisprocessistodefinetheindexaction,thenyouwillassociateitwiththerelevant indexes.Tocreatetheindexaction,selectTools>SeedData>Actions>IndexAction,nametheindex action,pressSaveandthenentertheValueeditor.
TheSQLstatementtobeusedisshownbelow,withtheDACfunctionshighlightedinbold:
create bitmap index getIndexName() on getTableName() (getIndexColumns()) local nologging parallel
UsingtheAddActionsdialog,selectCreateIndexastheactiontype,BothastheLoadType,andthen selecttheindexactionthatyoucreatedinthepreviousstepfortheActionType.
UsingMaterializedViewsforTableAggregation TheOracleBIApplicationsuseaggregatetablestoimprovetheperformanceofqueriesthatrequest aggregateddata.TheseaggregatetablesarethenmappedintotheOracleBusinessIntelligence EnterpriseEditionmetadatalayer,sothattheBIServercaninternallyrewritequeriestousethese aggregates.PostLoadProcessing(PLP)Informaticaworkflowsloaddataintotheseaggregatetables, eitherasacompleterefreshorincrementally,aspartofthesubjectareaload. TheEnterpriseEditionoftheOracleDatabasehassimilarfunctionalityintheformofMaterializedViews. Theseallowdatabaseadministratorstodefineaggregatesinthedatabase,whicharethenusedatquery runtimetosatisfyqueriesthatrequiredaggregateddata.MaterializedViewscanbefastrefreshableand canbedesignedtoeithersatisfyasingleaggregation,arangeofaggregationsorcanevenbecreated usinganOLAPAnalyticWorkspacetomeettheaggregationneedsofanentirestarschema. Likepartitionedtablesandlocalindexes,theDACdoesnotcontainoutoftheboxfunctionalityto createandmaintainmaterializedviews.ToaddthisfunctionalitytoyourETLprocess,youtherefore needtoaddtwonewactionstotheDACrepository: 1. AnactiontocreatetheW_SALES_INVOICE_LINE_Aobjectasamaterializedviewratherthana regulartable,andtocreatetherequiredMaterializedViewLogstosupportfastrefreshofthis object. 2. Anactiontoperformtherefreshofthematerializedview,whichyouwilluseinplaceofthe regularPLP_SalesInvoiceLinesAggregate_Load and PLP_SalesInvoiceLinesAggregate_Load_Full Informaticaworkflows. Toseehowtheexistingaggregatedtableispopulated,andtoextractthebaseSQLstatementthatyou willneedtocreatethematerializedview,opentheInformaticaDesignerapplicationandlocatethe PLP_SalesInvoiceLinesAggregate_Load_Full mapping.Whenyouviewthemappinglogic,you willseethattheW_SALES_INVOICE_LINE_FtableisjoinedtotheW_DAY_DtablethroughaSource
Qualifiermapping,whichisthensupplementedwithasequencenumberthatisusedtopopulatethe ROW_WIDcolumn.
Whilstwecannotreproducethesequencefunctionalitywithamaterializedview,youwillbeableto takethedatafromthesetwotablesanduseittoinitiallypopulate,andthenrefresh,thematerialized view. Likethepreviousexamplewhereyoucreateapartitionedtable,creationofthematerializedviewwillbe performedbyanewtaskactionthatyouwillassociatewiththe PLP_SalesInvoiceLinesAggregate_Load DACtaskwhenruninFullmode.Tocreatetheaction, selectTools>SeedData>Actions>TaskAction,andcreateandsaveanewtaskaction.AttheValue dialog,createindividualstepstodropandrecreatetherequiredmaterializedviewlogs,thendropand recreatethematerializedview,usingthefollowingSQLstatements: 1.DropMaterializedViewLogonW_SALES_INVOICE_LINE_F
drop materialized view log on W_SALES_INVOICE_LINE_F
2.DropmaterializedviewlogonW_SALES_INVOICE_LINE_F
drop materialized view log on W_DAY_D
3.CreateMaterializedViewLogonW_SALES_INVOICE_LINE_F
create materialized view log on W_SALES_INVOICE_LINE_F with sequence, rowed ( sales_ordln_id, sales_pckln_id ... discount_line_flg) including new values
4.CreateMaterializedViewLogonW_DAY_D
create materialized view log on W_DAY_D with sequence, rowed ( row_wid, calendar_date ... x_custom) including new values
5.DropMaterializedView
drop materialized view W_SALES_INVOICE_LINE_A
6.CreateMaterializedView
create materialized view W_SALES_INVOICE_LINE_A pctfree 0 build immediate refresh fast as select 1 as row_wid, w_sales_invoice_line_f.chnl_type_wid from w_sales_invoice_line_f, w_day_d where . group by )
Delete,andrecreatetheexecutionplanforyoursubjectarea,topickupthechangestothePLP mapping.Onceyouhavedonethis,youarenowreadytorerunyourDACexecutionplan,toassess whatimprovementtoprocessingtimethesechangeshavemade. QuantifyingtheImprovements WhenrunagainsttheauthorsinstallationofOracleBusinessIntelligenceApplications7.9.5,usinga subsetoftheOracleEBusinessSuiteVisiondataset,thefollowingtimingswererecordedusingthese scenarios: 1. BaselinerunofthestandardoutoftheboxETLroutines 2. AdditionoftheCOMPRESSclausetotheW_SALES_INVOICE_LINE_Ftable 3. AddingofpartitioningtotheW_SALES_INVOICE_LINE_Ftable,andkeepingcompression.
4. Alloftheabove,andreplacementoftheW_SALES_INVOICE_LINE_Atablewithafastrefresh materializedview. Theresultsofthesescenariosareshowninthetablebelow. Scenario LoadType Rows Elapsed Loaded Time 478571 904secs 553 499secs 478571 887secs 553 482secs Time Improvemen t Vs.Baseline n/a n/a 2% 3% 7% Fact Table Size 189Mb 189Mb 43MB 43MB 44MB Size improvemen t Vs.Baseline n/a n/a 77% 77% 77%
1Baseline 1Baseline 2With Compression 2With Compression 3With Partitioning & Compression 3With Partitioning & Compression 4With Partitioning, Compression and Materialized View 4With Partitioning, Compression and Materialized View
478571 834secs
Incremental
553
470secs
6%
44MB
77%
Full
478571 437secs
51%
44MB
77%
Incremental
553
334secs
33%
44MB
77%
Forexample,theDACdoesnotcreateprimarykeyorforeignkeyconstraintsonthetablesthatit creates,whichtogetherwiththedefaultsettingfortheSTAR_TRANSFORMATION_ENABLEDparameter fornewlycreateddatabases,inmostcasesmeansthatstartransformationsarenotusedwhenthe databasehandlesqueriesagainstfacttablesthatinvolvefilteringagainsttwoormoredimensions.You couldthereforeaddadditiontaskactionstoSILandPLPtaskstocreateanddroptheseconstraints, possiblyusingtheRELY NOVALIDATEclausestominimizeunnecessaryredo,setthe STAR_TRANSFORMATION_ENABLEDparameterappropriatelyandtakeadvantageofthiskeyOracledata warehousingfeature. AnotheroptimizationpossibilityistousetheOLAPOptiontoOracleDatabase11gtoreplacethe materializedviewoutlinedinthispaperwithaCubeOrganizedMaterializedView,whichcouldprovide aggregationsforanentirestarschemaatmultiplelevelsofaggregation.YouwouldneedtouseOracle AnalyticWorkspaceManager(afreedownloadfromhttp://otn.oracle.com)tocreatethecube organizedmaterializedview,butonceitiscreateditcouldberefreshedinthesamemannerasthe materializedviewthatthispaperdescribes. Conclusions TheETLandqueryoptimizationtechniquesprovidedoutoftheboxwiththeOracleBIApplications providesareappropriateforgenericdatabases,butcanbeimprovedaponifyoumakeuseofthe specificdatawarehouseoptimizationsavailableonyouractualtargetdatabase.TheEnterpriseEdition oftheOracleDatabaseprovidesmanysuchfeaturesincludingsegmentcompression,materializedviews andpartitioning,andthispapersetsouthowtheycanbeusedinconjunctionwiththenewactions featureavailablewiththeOracleDataWarehouseAdministrationConsole10.1.3.4andhigher. AbouttheAuthor MarkRittmanisanOracleACEDirectorandiscofounderofRittmanMeadConsulting,aspecialist OraclepartnerdeliveringOracledatawarehousing,businessintelligenceandperformancemanagement solutions.MarkiscochairoftheODTUGBI&DWSIG,iseditorofOracleScene,themagazineoftheUK OracleUserGroup,writesregularlyforOracleMagazine,OracleTechnologyNetworkandtheODTUG TechnicalJournal,andrunsablogathttp://www.rittmanmead.com/blog.Markcanbereachedat mark.rittman@rittmanmead.comifyouwouldliketodiscussthecontentsofthiswhitepaper.