Vous êtes sur la page 1sur 44

Management & Technology Consulting

Management & Technology Consulting

11/4/2011

Management & Technology Consulting

11/4/2011

Objectives: Like for Like Technical Upgrade of Oracle Applications


11.5.10.2 12.1.3

Retrofit existing reports & customizations for compatibility and usability on R12 Provide training and coordinate testing across 4 major regions
North America, Asia, India and Europe

Management & Technology Consulting

11/4/2011

Scope:
26 Ledgers GL, FA, AP, AR, TAX, IE, CE, PO, INV, OM 40,000 User Base 955 Custom Objects Required Analysis 3TB Database Global training and testing 7 Months

Management & Technology Consulting

11/4/2011

Steering Committee: 15 Project Manager: 2 2 Leads (Training & Testing) 9 Functional Analysts Technical Lead: 2

2-3 DBAs

12 Developers

Global Business SMES: 50

Management & Technology Consulting

11/4/2011

Decide upgrade approach and scope Determine timeline & resources


Maximize Testing Robust Cutover Planning Testers should have a strong knowledge of business processes

Issue Strategy Communication

Management & Technology Consulting

11/4/2011

Training Approach Phase 1 Phase 2 Phase 3


Onsite Classroom Training

Provided by Oracle University


Functional Analysts and Business Core Team

Train the Trainer Provided by Jibe Consulting Delta training of mandatory R12 functionality

End User Training Provided by Amazon Business Core Team Brown Bag Sessions, Onsite Training, Web Conferences

Management & Technology Consulting

11/4/2011

Training delivered prior to test cycles to ensure maximum benefit End user training delivered prior to UAT/Go-Live On-Site training was the most successful

Management & Technology Consulting

11/4/2011

Turnover within the organization led to training gaps


Follow up and obtain training plans from all areas

Small Changes = Big Questions


Provide some level of training for all new/changed functionality

Face Time
Business users who were offered onsite training/ web sessions were the most successful in navigating through the new functions Video Recordings

Timing
Deliver training to coordinate with testing and Go-Live

Management & Technology Consulting

11/4/2011

3 Test Cycles
TC1
Standard Oracle Functionality Core Business Team Introduced interfaces and customizations Business SMEs All business processes, custom objects and reports End Users

TC2

UAT

Management & Technology Consulting

11/4/2011

10

Challenges
Educating testers on how and what to test Differences between unit testing and end to end testing Volume of testing
How to measure progress

Solution
Conducted testing planning sessions with entire team Held planning work sessions with individual teams Assigned a finance owner to each end to end scenario

Management & Technology Consulting

11/4/2011

11

Global Reference AR-US-001 AR-US-002 AR-US-003 AR-US-004 AR-US-005 AR-US-006 AR-US-007 AR-US-008 AR-US-009 AR-US-010 AR-US-011 AR-US-012

Scenario Name Autoinvoice Import Program Payables Open Interface Import

Category Vendor Returns Vendor Returns

Week 1

Week 2 1 1 1 1 1

Week 3

Target Test Completion Date Responsible 7/12/2011 Kala Jones 7/12/2011 7/12/2011 7/12/2011 7/12/2011 7/12/2011 7/12/2011 7/12/2011 7/12/2011 7/12/2011 7/12/2011 Kala Jones Kala Jones Kala Jones Kala Jones Kala Jones Kala Jones Kala Jones Kala Jones Kala Jones Kala Jones Kala Jones

Prepayment Matching Program (Prepayments Vendor Returns Matching Program) AMZ Load Vendor Return Data From FATE AMZ Customers Interface load from Vendor Returns Customer Interface AMZ Load AR Vendor Returns Invoices AutoInvoice Master Program View/Change Payment Terms AMZ Vendor Return Create Reclass Journal Vendor Returns Vendor Returns Vendor Returns Vendor Returns Vendor Returns Vendor Returns Vendor Returns

1 1 1 1 1 1

AMZ Vendor Returns Current State Summary Vendor Returns Report AMZ Vendor Returns Current State Detail Report AMZ AR Aging Report - Customer/Product Summary AMZ AR Invoice Aging Detail Report Vendor Returns

7/12/2011 1

AR-US-013 AR-US-014

AR Unit Test 1 AR Unit Test 1

7/8/2011 7/8/2011

Andrew Nienhaus Andrew Nienhaus

Management & Technology Consulting

11/4/2011

12

Blocks of time scheduled by category Color coding helped testers identify when they were needed

Management & Technology Consulting

11/4/2011

13

Total Pass Total Scripts Module Accounts Payable Accounts Receivable Cash Management Fixed Assets General Ledger Inventory Order Management Purchasing eBusiness Tax TOTAL # 2556 651 90 240 517 86 125 380 26 4671 2424 604 86 194 361 86 125 375 8 4263 % 95% 93% 96% 81% 70% 100% 100% 99% 31% 91% #

Total Fail % 132 48 1 0 86 0 0 5 18 290 5% 7% 1% 0% 17% 0% 0% 1% 69% 6%

Total Incomplete # 0 0 3 46 68 0 0 0 0 117 % 0% 0% 3% 19% 13% 0% 0% 0% 0% 3% #

Attempted % 2556 651 87 194 449 86 125 380 26 4553 100% 100% 97% 81% 87% 100% 100% 100% 100% 97%

Metrics were summarized for reporting Drilldown by process and country

Management & Technology Consulting

11/4/2011

14

3 Week Test Cycle Objectives


Test standard Oracle functionality Business teams identified test scenarios & created test scripts Identify End to End Scenarios Planning of future test cycles

Management & Technology Consulting

11/4/2011

15

Test Cycle 1 - Challenges


Challenges Managing Expectations Users not aware of what was a customization Solution Functional Analysts present in the testing rooms Knowledge Transfer

Unexpected Issues with Standard Functionality Custom Folders Custom Application ASLs no longer supported Issues Logging Process Multiple entries for the same issue Entries which did not clearly state the problem

Deleted all custom folders Created all new responsibilities Converted all ASL items to requisition templates Process Owner Assisted with reporting metrics and logging issues

Management & Technology Consulting

11/4/2011

16

5 Weeks
3 Weeks Unit/Transaction Testing 2 Weeks Integrated/End to End Testing

2343 Unit Test Scripts Executed 59 End to End Scenarios

Management & Technology Consulting

11/4/2011

17

Objectives
Identify and resolve system issues Identify training gaps Complete generation/rework of test scripts

Exit Criteria
All Blocker issues resolved All Critical issues identified and resolution ETA determined

Management & Technology Consulting

11/4/2011

18

Challenges Global Coordination Testers outside of the US struggled with lack of support Expectations were not clear Testing Interfaces Initial testing was not successful Frustration

Solution Bi-Weekly testing status calls Weekly calls with Functional Analysts Daily communication of blocking issues and progress All interfaces thoroughly tested by FA prior to user testing Interfaces scheduled to ensure data preparation was complete Interfaces tested globally

Management & Technology Consulting

11/4/2011

19

3 week test cycle


Unit testing and Integrated testing

End Users responsible for testing


Global participation

4636 Test Scripts


Majority were in AP and AR Performance testing

Management & Technology Consulting

11/4/2011

20

Objectives
Resolve any remaining issues Ensure business and system readiness for Go-Live Complete performance testing

Exit Criteria
All Blocker Issues Resolved All Critical Issues Identified
Needed to be resolved by month-end

Management & Technology Consulting

11/4/2011

21

Challenges

Solution

Functionality Issues New testers found new issues Customizations and reports tested for the first time

Performance Issues Reports did not complete due to large Performance Issues Identified key reports and did volume of data Reports did not complete performance tuning
due to large volume of data

Additional resources added Encouraged as much testing as possible

Worked with Oracle for solutions

Performance Testing Few resources and limited time to complete

Attempted load testing during UAT Ran several time consuming processes and monitored execution

Management & Technology Consulting

11/4/2011

22

Performance Issues
Test, test, test and test some more

Tax and AP Data Issues


Multiple data fixes required Test both new and upgraded data

PO/AP Reconciliation
Significant changes that were not tested by all geographies

Create Accounting
Completely new concept, ensure users understand the changes and implications to the month end process

Management & Technology Consulting

11/4/2011

23

Agenda
Customization Planning Approach Development Challenges Performance Issues Concurrent Manager R12 Patch Monitoring and Tuning

Management & Technology Consulting

11/4/2011

24

Retrofit 900+ customizations in time for TC2 testing. Development time


P1 customizations ~2 months. P2 customizations ~3.5 months

Minimize business downtime due to R12 patch upgrade

Management & Technology Consulting

11/4/2011

25

Inventory

Inventory all the customizations


What can retire What will upgrade cleanly What needs to be re-implement in R12. E.g. check printing

Identify

Resource planning for the development team


Finalize the list of customizations that will need to be retrofitted. Data model changes tool used Document id 1290886.1. Cross reference with the customization list Assign level of effort to broken customizations Level of effort used: Low(8), medium(16) and high (40 hrs.) Load resources appropriately Assign resources to business modules. E.g. designated developer for bank integration

Provide technical training to the developers

Begin Development phase

Management & Technology Consulting

11/4/2011

26

Management & Technology Consulting

11/4/2011

27

Data model comparison report


Help understand the data model changes Download document id 1290886.1

Management & Technology Consulting

11/4/2011

28

Check Print development

Start early. Long development cycle Long test cycle with the banks Print issues
Report output shifted across printers even though they were the same make and model Issue was due to the print drivers not being up to date for the printer. Some of the printers were using altogether different drivers as a result of changes over time. Used pdftops to stabilize the shifting of the output

Custom Application (Amazon.com) associated to Custom Responsibilities

E.g. AMZ Purchasing Manager associated to Amazon.com instead of seeded Purchasing application Caused breakage of self service forms in Payments, iProcurement and iExpense LOV errors in certain standard reports E.g. Create Accounting Solution: Re-do responsibilities and remove amazon.com association. ~1200 responsibilities impacted Consideration for R12 upgrade
Management & Technology Consulting 11/4/2011 29

BI Publisher reports are memory hungry compared to traditional Reports 6i Several month end reports failed to complete. E.g. Payables Posted Invoice/Payments Register
45,000 pages of pdf output for one month Started as 16 GB of report output. Solution: Oracle support provided a stripped down version of the XML Output Pre-allocated 2GB memory on the concurrent program definition. Set Options = -ms2048m -mx2048m 3 properties need to be set to True for the report template under XML Administrator Responsibility

Management & Technology Consulting

11/4/2011

30

Auto-Select and Build Payments programs erroring out after hitting batches of 20K+ invoices
Solution: Profile AP: Gather PPR Table Stats set to Yes at site level Performance patch released by Oracle support

Create Accounting process running longer 20+ hrs. impacting month close activities
Solution: Bug raised with Oracle support and patch received Created a daily run schedule for Create Accounting job for each geography Assigned multiple processors to Create Accounting process to decrease run time. Caution: optimize the number of processor based on server and concurrent manager load. Screenshot on next page
Management & Technology Consulting 11/4/2011 31

Performance Issues (cont.)

Recommend doing a load test during UAT testing to uncover performance issues

Management & Technology Consulting

11/4/2011

32

Amazon uses 30+ custom managers to process jobs under Concurrent Manager

Used to segregate quick and long running jobs. E.g. Print POs and Alerts(quick jobs) vs. Invoice import (long running) Separate managers for high volume jobs. E.g. Print AR Invoices. Daily load of > 10K jobs
Managers were constantly backed up. Several hundred jobs sitting in pending state everyday 2 hrs. average wait time before jobs gets picked for processing Global users being impacted by spill over from other geographies Impacted several time sensitive activities E.g. month end reports, PO printing SLAs Created negative user experience. Users constantly requesting IT to bump up the priority of jobs

Pain Points

Management & Technology Consulting

11/4/2011

33

Observations:

All workers not being utilized. E.g. between 10-18 workers would be used instead of the full 20. Would use the full allotted only at times. Reduced the number of concurrent managers from 30 to 18 Updated cache size of the managers to twice the number of workers Decreased sleep seconds of quick running jobs to 20 Increased sleep seconds to 120 for long running jobs

Solution:

Summary

Key is to minimize the number of managers Optimize:


Number of workers/processes Sleep seconds Cache size. Thumb rule is twice the number of workers

Management & Technology Consulting

11/4/2011

34

5 Days Downtime Upgrade 3TB of database

Management & Technology Consulting

11/4/2011

35

Major Cutover Activities

Management & Technology Consulting

11/4/2011

36

Useful tool to communicate with the IT and Business management Helps understand the long activities and focus on reducing the time of those activities Helps the team not to get confused with >5,000 lines of detailed cutover plan. Was used in conjunction with the detailed cutover plan Contingency built into major activities. Relieves pressure if any activity gets delayed

Management & Technology Consulting

11/4/2011

37

Biggest contributors to the 5 days of downtime


R12.1.1 patch = 55 hrs.
Oracle data fixes for tax, AR invoices and AP =15 hrs.
10 data fixes totaling up to 15 hrs. of downtime Long running due to the number of transactions being updated. E.g. 86 million rows in ap_invoices_all Missing Establishment_id in ZX tables Ensure Oracle provides a script that can be run with parallel workers

Post-Upgrade reports = 8 hrs.


Issue was the runtime of the reports. Month end reports ran for >5 hrs

Management & Technology Consulting

11/4/2011

38

Technical upgrade is the process of upgrading 11i environment to R12.1.3

Involves upgrading database to 11gR2 (done as a precursor project) Laying technical stack for R12 Running R12.1.1 patch Running R12.1.3 patch Post upgrade patches including one-offs identified by project team

Recommendation: Plan on running at least 5 tech upgrades

Practice and practice more to make it a repeatable process Identify long running jobs and tune them

Management & Technology Consulting

11/4/2011

39

Business shut down activities


Process all transactions Submit all accounting transactions Verify all interfaces are clean Must be coordinated across all business teams

Smoke Testing
Key Interfaces and processes Provide assurance to business and management that system was ready

Ramp Up Slowly

Management & Technology Consulting

11/4/2011

40

Erratic runtime of AP/AR jobs. Oracle database optimizer would change the sql execution plan and cause jobs to run longer
This was due to:
Oracle dynamic sampling Lack of database statistics Use of temporary tables as part of upgrade

Solution:

Pin SQL queries to a specific plan by using SQL profiles Monitor long running queries in OEM Implement a better explain plan if recommended by Oracle OEM SQL profile the plan Export and use for next run
Management & Technology Consulting 11/4/2011 41

Procedure:

Technical Upgrade Process flow and tuning exercise

Management & Technology Consulting

11/4/2011

42

Management & Technology Consulting

11/4/2011

43

Oracle database name can only be in A-Z or 0-9 character format


Rapidwiz failed to run during R12 prod upgrade. Did not like _ underscore in FIN_P db name Solution: Changed the name to FINP. Caused the tnsnames file to be changed for external systems.

Diligently follow R12 upgrade guide (Part No. E16342-01)


Excellent suggestions in the technical and functional areas Good tips in pre-upgrade functional steps that help reduce downtime.
E.g. Submit Accounting missed in 11i and caused one of job to run for 7 hrs. instead of 3 hrs. during the tech upgrade

Management & Technology Consulting

11/4/2011

44

Vous aimerez peut-être aussi