Académique Documents
Professionnel Documents
Culture Documents
11/4/2011
11/4/2011
Retrofit existing reports & customizations for compatibility and usability on R12 Provide training and coordinate testing across 4 major regions
North America, Asia, India and Europe
11/4/2011
Scope:
26 Ledgers GL, FA, AP, AR, TAX, IE, CE, PO, INV, OM 40,000 User Base 955 Custom Objects Required Analysis 3TB Database Global training and testing 7 Months
11/4/2011
Steering Committee: 15 Project Manager: 2 2 Leads (Training & Testing) 9 Functional Analysts Technical Lead: 2
2-3 DBAs
12 Developers
11/4/2011
11/4/2011
Train the Trainer Provided by Jibe Consulting Delta training of mandatory R12 functionality
End User Training Provided by Amazon Business Core Team Brown Bag Sessions, Onsite Training, Web Conferences
11/4/2011
Training delivered prior to test cycles to ensure maximum benefit End user training delivered prior to UAT/Go-Live On-Site training was the most successful
11/4/2011
Face Time
Business users who were offered onsite training/ web sessions were the most successful in navigating through the new functions Video Recordings
Timing
Deliver training to coordinate with testing and Go-Live
11/4/2011
3 Test Cycles
TC1
Standard Oracle Functionality Core Business Team Introduced interfaces and customizations Business SMEs All business processes, custom objects and reports End Users
TC2
UAT
11/4/2011
10
Challenges
Educating testers on how and what to test Differences between unit testing and end to end testing Volume of testing
How to measure progress
Solution
Conducted testing planning sessions with entire team Held planning work sessions with individual teams Assigned a finance owner to each end to end scenario
11/4/2011
11
Global Reference AR-US-001 AR-US-002 AR-US-003 AR-US-004 AR-US-005 AR-US-006 AR-US-007 AR-US-008 AR-US-009 AR-US-010 AR-US-011 AR-US-012
Week 1
Week 2 1 1 1 1 1
Week 3
Target Test Completion Date Responsible 7/12/2011 Kala Jones 7/12/2011 7/12/2011 7/12/2011 7/12/2011 7/12/2011 7/12/2011 7/12/2011 7/12/2011 7/12/2011 7/12/2011 Kala Jones Kala Jones Kala Jones Kala Jones Kala Jones Kala Jones Kala Jones Kala Jones Kala Jones Kala Jones Kala Jones
Prepayment Matching Program (Prepayments Vendor Returns Matching Program) AMZ Load Vendor Return Data From FATE AMZ Customers Interface load from Vendor Returns Customer Interface AMZ Load AR Vendor Returns Invoices AutoInvoice Master Program View/Change Payment Terms AMZ Vendor Return Create Reclass Journal Vendor Returns Vendor Returns Vendor Returns Vendor Returns Vendor Returns Vendor Returns Vendor Returns
1 1 1 1 1 1
AMZ Vendor Returns Current State Summary Vendor Returns Report AMZ Vendor Returns Current State Detail Report AMZ AR Aging Report - Customer/Product Summary AMZ AR Invoice Aging Detail Report Vendor Returns
7/12/2011 1
AR-US-013 AR-US-014
7/8/2011 7/8/2011
11/4/2011
12
Blocks of time scheduled by category Color coding helped testers identify when they were needed
11/4/2011
13
Total Pass Total Scripts Module Accounts Payable Accounts Receivable Cash Management Fixed Assets General Ledger Inventory Order Management Purchasing eBusiness Tax TOTAL # 2556 651 90 240 517 86 125 380 26 4671 2424 604 86 194 361 86 125 375 8 4263 % 95% 93% 96% 81% 70% 100% 100% 99% 31% 91% #
Attempted % 2556 651 87 194 449 86 125 380 26 4553 100% 100% 97% 81% 87% 100% 100% 100% 100% 97%
11/4/2011
14
11/4/2011
15
Unexpected Issues with Standard Functionality Custom Folders Custom Application ASLs no longer supported Issues Logging Process Multiple entries for the same issue Entries which did not clearly state the problem
Deleted all custom folders Created all new responsibilities Converted all ASL items to requisition templates Process Owner Assisted with reporting metrics and logging issues
11/4/2011
16
5 Weeks
3 Weeks Unit/Transaction Testing 2 Weeks Integrated/End to End Testing
11/4/2011
17
Objectives
Identify and resolve system issues Identify training gaps Complete generation/rework of test scripts
Exit Criteria
All Blocker issues resolved All Critical issues identified and resolution ETA determined
11/4/2011
18
Challenges Global Coordination Testers outside of the US struggled with lack of support Expectations were not clear Testing Interfaces Initial testing was not successful Frustration
Solution Bi-Weekly testing status calls Weekly calls with Functional Analysts Daily communication of blocking issues and progress All interfaces thoroughly tested by FA prior to user testing Interfaces scheduled to ensure data preparation was complete Interfaces tested globally
11/4/2011
19
11/4/2011
20
Objectives
Resolve any remaining issues Ensure business and system readiness for Go-Live Complete performance testing
Exit Criteria
All Blocker Issues Resolved All Critical Issues Identified
Needed to be resolved by month-end
11/4/2011
21
Challenges
Solution
Functionality Issues New testers found new issues Customizations and reports tested for the first time
Performance Issues Reports did not complete due to large Performance Issues Identified key reports and did volume of data Reports did not complete performance tuning
due to large volume of data
Attempted load testing during UAT Ran several time consuming processes and monitored execution
11/4/2011
22
Performance Issues
Test, test, test and test some more
PO/AP Reconciliation
Significant changes that were not tested by all geographies
Create Accounting
Completely new concept, ensure users understand the changes and implications to the month end process
11/4/2011
23
Agenda
Customization Planning Approach Development Challenges Performance Issues Concurrent Manager R12 Patch Monitoring and Tuning
11/4/2011
24
11/4/2011
25
Inventory
Identify
11/4/2011
26
11/4/2011
27
11/4/2011
28
Start early. Long development cycle Long test cycle with the banks Print issues
Report output shifted across printers even though they were the same make and model Issue was due to the print drivers not being up to date for the printer. Some of the printers were using altogether different drivers as a result of changes over time. Used pdftops to stabilize the shifting of the output
E.g. AMZ Purchasing Manager associated to Amazon.com instead of seeded Purchasing application Caused breakage of self service forms in Payments, iProcurement and iExpense LOV errors in certain standard reports E.g. Create Accounting Solution: Re-do responsibilities and remove amazon.com association. ~1200 responsibilities impacted Consideration for R12 upgrade
Management & Technology Consulting 11/4/2011 29
BI Publisher reports are memory hungry compared to traditional Reports 6i Several month end reports failed to complete. E.g. Payables Posted Invoice/Payments Register
45,000 pages of pdf output for one month Started as 16 GB of report output. Solution: Oracle support provided a stripped down version of the XML Output Pre-allocated 2GB memory on the concurrent program definition. Set Options = -ms2048m -mx2048m 3 properties need to be set to True for the report template under XML Administrator Responsibility
11/4/2011
30
Auto-Select and Build Payments programs erroring out after hitting batches of 20K+ invoices
Solution: Profile AP: Gather PPR Table Stats set to Yes at site level Performance patch released by Oracle support
Create Accounting process running longer 20+ hrs. impacting month close activities
Solution: Bug raised with Oracle support and patch received Created a daily run schedule for Create Accounting job for each geography Assigned multiple processors to Create Accounting process to decrease run time. Caution: optimize the number of processor based on server and concurrent manager load. Screenshot on next page
Management & Technology Consulting 11/4/2011 31
Recommend doing a load test during UAT testing to uncover performance issues
11/4/2011
32
Amazon uses 30+ custom managers to process jobs under Concurrent Manager
Used to segregate quick and long running jobs. E.g. Print POs and Alerts(quick jobs) vs. Invoice import (long running) Separate managers for high volume jobs. E.g. Print AR Invoices. Daily load of > 10K jobs
Managers were constantly backed up. Several hundred jobs sitting in pending state everyday 2 hrs. average wait time before jobs gets picked for processing Global users being impacted by spill over from other geographies Impacted several time sensitive activities E.g. month end reports, PO printing SLAs Created negative user experience. Users constantly requesting IT to bump up the priority of jobs
Pain Points
11/4/2011
33
Observations:
All workers not being utilized. E.g. between 10-18 workers would be used instead of the full 20. Would use the full allotted only at times. Reduced the number of concurrent managers from 30 to 18 Updated cache size of the managers to twice the number of workers Decreased sleep seconds of quick running jobs to 20 Increased sleep seconds to 120 for long running jobs
Solution:
Summary
11/4/2011
34
11/4/2011
35
11/4/2011
36
Useful tool to communicate with the IT and Business management Helps understand the long activities and focus on reducing the time of those activities Helps the team not to get confused with >5,000 lines of detailed cutover plan. Was used in conjunction with the detailed cutover plan Contingency built into major activities. Relieves pressure if any activity gets delayed
11/4/2011
37
11/4/2011
38
Involves upgrading database to 11gR2 (done as a precursor project) Laying technical stack for R12 Running R12.1.1 patch Running R12.1.3 patch Post upgrade patches including one-offs identified by project team
Practice and practice more to make it a repeatable process Identify long running jobs and tune them
11/4/2011
39
Smoke Testing
Key Interfaces and processes Provide assurance to business and management that system was ready
Ramp Up Slowly
11/4/2011
40
Erratic runtime of AP/AR jobs. Oracle database optimizer would change the sql execution plan and cause jobs to run longer
This was due to:
Oracle dynamic sampling Lack of database statistics Use of temporary tables as part of upgrade
Solution:
Pin SQL queries to a specific plan by using SQL profiles Monitor long running queries in OEM Implement a better explain plan if recommended by Oracle OEM SQL profile the plan Export and use for next run
Management & Technology Consulting 11/4/2011 41
Procedure:
11/4/2011
42
11/4/2011
43
11/4/2011
44