Science & Technology

Information Technology Post Implementation Review Guide. Immigration & Naturalization Service. Information Technology Investment Management Series

Description
Information Technology Investment Management Series Information Technology Post Implementation Review Guide Office of Strategic Information and Technology Development Immigration & Naturalization Service
Published
of 26
All materials on our website are shared by users. If you have any questions about copyright issues, please report us to resolve them. We are always happy to assist you.
Share
Transcript
Information Technology Investment Management Series Information Technology Post Implementation Review Guide Office of Strategic Information and Technology Development Immigration & Naturalization Service Version 1.0 December 2001 Contents Chapter 1 Introduction Chapter 2 Documentation Chapter 3 Post Implementation Review Report Elements Appendices Background and Purpose Objectives Participants Documentation Collection Final Report Project Plan File Functional Requirements Document Cost Benefit Analysis Security Reviews and Documentation Maintenance Activity Summary Overview Evaluation Summary Analysis and Implementation Outputs Security Computer Operations Maintenance Activities Appendix A: Acronyms and Abbreviations Appendix B: Post Implementation Review Report Outline Appendix C: Questionnaire A-1 B-1 C-1 i Chapter 1 Introduction Background and Purpose Objectives Participants Documentation Collection A post-implementation review (PIR) is the analysis and documentation used to evaluate the effectiveness of a new system or significant system revision after it has been operational for an initial period of time. This is a standard review designed to evaluate any Immigration and Naturalization Service (INS) system and provide insight in areas such as lessons learned, opportunities for improvement, and issues that require special attention. This review will assess how successful a system is in areas such as: Functionality Performance Cost versus benefit Life cycle development activities The review team will comprise individuals with functional knowledge and subject matter experts with technical or functional experience in the system being reviewed. The Office of Strategic Information and Technology Development (SITD) or the chair of the system users group will coordinate the activities of the team. At a minimum, representatives from the following groups will be interviewed to collect information: End users and their managers System administrators Computer operations personnel Office of Information Resources Management (OIRM) project technical representatives INS system proponents The interviews will be conducted in accordance with the questionnaire contained in Appendix C. In order to check the performance of a system against projections or expectations, the following documents should be obtained if available: Functional requirements document (including concept of operations and requirements traceability matrix) Cost Benefit Analysis (CBA) Security reviews and documentation Maintenance activity summary 1 Chapter 2 Documentation Final Report Project Plan File Functional Requirements Document Cost Benefit Analysis Security Reviews and Documentation Maintenance Activity Summary Information collected from interviews and documentation will be analyzed and a comprehensive report will be prepared. The final report will contain any combination of subject areas identified in Appendix B. PIR results can be used to support subsequent investment decisions, strengthen development procedures, and further improve the system. Project plan documentation should be read prior to the review and used as a baseline against which actual results can be compared. The functional requirements document should be read, along with the project plan, prior to the review and used as a baseline of performance targets against which actual results can be compared. The CBA should be reviewed to verify that the system benefits still outweigh the costs. The comparison of projected costs, actual costs, and derived benefits is the foundation of the cost justification portion of the review. Outcome-based performance measurement information should also be identified and collected. Documented security procedures as well as the results from security reviews should be obtained to verify the existence of a solid security plan. Maintenance logs should be examined, and frequent or unusual entries should be explored to ensure that the system is operating properly. 2 Chapter 3 Post Implementation Review Report Elements Overview Project Identification System Proponent History of the System Functional System Description and Data Usage The PIR is used to evaluate the effectiveness of the system development after the system has been in production for a period of time (normally 6 months). The objective is to determine if the system does what it is designed to do. The review assesses how successful the system is in terms of functionality, performance, and cost versus benefits, as well as the effectiveness of the life cycle development activities that produced the system. The PIR is scheduled to follow the release of a system or system revision by an appropriate amount of time to allow determination of the effectiveness of the system. A representative from the functional development group or other member of the major user organization participates in the review. The project sponsor ensures that all documentation and all personnel needed to participate in the review are accessible. The reviewer and an assigned team collect the information needed for the PIR by interviewing end users and their managers, system administrators, and computer operations personnel. The report is then prepared and provided to stakeholders. The PIR is a free-form report, and not all sections are relevant or necessary to the final product. A description of the PIR report is outlined in Appendix B, PIR Report Outline. The following describes the sections of the report. Provide the identifying information associated with the project, including the applicable project control code, system acronym, and system title. Provide the name of the system proponent. Briefly describe the system s history and predecessor, if any. State the mission needs and information requirements, including how the system is expected to help users. Briefly describe what the system does functionally and how the data are used by the system. 3 Chapter 3 Post Implementation Review Report Elements Evaluation Summary General Satisfaction with the System Current Cost-Benefit Justification Needed Changes or Enhancements The purpose of this section is to provide a summary of the overall adequacy and acceptance of the system. Describe users experience with the implemented system. Comments should address the following: Level of user satisfaction Strengths of the system, including specific areas of success Any problems Frequently used features Infrequently used features Features not used at all Suggested improvements Assess whether the system is paying for itself. Base the assessment on the anticipated benefits and costs projected in the concept development phase and revised during subsequent phases of the Systems Development Life Cycle (SDLC). This section is intended to review the costs, benefits, and outcome-based performance measures identified for the system. Comments should address the following: The extent of the benefits and performance improvement and if they are reported to be less or greater than those projected in the development analysis and functional requirements report If any difference is permanent or will change over time If the system is or will be cost-justifiable Gauge the magnitude of effort needed to change or improve the system. Describe the nature and priority of the suggested changes; more detail will be provided in other sections. Comments should address the following: The suggested changes The scope of the changes The resource requirements to effect the changes 4 Chapter 3 Post Implementation Review Report Elements Analysis and Implementation Purpose and Objectives Scope Benefits Development Cost The purpose of this section is to gauge the completeness of the functional requirements and implementation according to the study. Evaluate the adequacy of the original definition of purpose and objectives presented in the functional requirements document and if the objectives were achieved during implementation. Evaluate if any objectives have changed, or should have changed. Comments should address the following: Extent to which goals were met The level of the objective definition Extent to which objectives were met Possible changes to the objectives Analyze whether proper limits were established in the feasibility study and if they were maintained during implementation. Comments should address the following: Variations from the scope definition as agreed to in the concept development The extent to which the scope was followed Any possible future changes to the scope Analyze whether the benefits anticipated in the concept development and requirements definition analyses were realized. Detail all benefits, quantifiable or non-quantifiable, and any quantifiable resources associated with each. Comments should address the following: The adequacy of the benefit definition The level of benefits realized The anticipated benefits that can be realized The reason for the variance between planned and realized benefits Determine the adequacy of the development cost estimated and any deviation between the estimated and actual development costs. Comments should address the following: The adequacy of the original and subsequent cost estimates The actual costs, by type The reasons for any difference between estimated and actual costs 5 Chapter 3 Post Implementation Review Report Elements Operating Costs Training Outputs Usefulness Timeliness Analyze the adequacy of the operating cost estimates and deviations between estimate and actual costs. Summarize the resources required to operate the system. Comments should address the following: The adequacy of the operating estimates The actual operating costs The difference Evaluate whether all levels of user training were adequate and timely. Comments should address the following: The timeliness of the training provided The adequacy of the training The appropriateness of the training Identification of additional training needs by job category The ability of personnel to use the training provided The purpose of this section is to evaluate the adequacy and usefulness of the outputs from the system. Care must be taken to ensure that all reports are evaluated. Measure the extent to which the users need the output of the system. Comments should address the following: Utility Absolutely essential Important and highly desirable Interesting (proves what is already known) Incomplete (does not provide all the necessary information) Unnecessary Identification of information/reports needed but not currently generated by the system or unable to be obtained Demonstration of the ability to do without the reports Alternatives for obtaining the information where improvements can be achieved Determine whether output production performance meets user needs. Address the frequency with which output arrives on time, early, and late; and the amount of follow up needed to obtain the output. 6 Chapter 3 Post Implementation Review Report Elements Data Quality Security Data Protection Disaster Recovery Controls Audit Trails Assess the need to provide for effective use of shared data to enhance performance and system interoperability. Comments should address data accuracy and data reliability. The purpose of this section is to determine whether the system provides adequate security of data and programs. In addition to access security, procedures for backup, recovery, and restart should be reviewed. Determine whether the security, backup, recovery, and restart capabilities adequately safeguard data, including master, transaction, and source. Online systems naturally require special techniques (such as transaction logging). Comments should address the following: The adequacy of the security, backup, recovery, and restart procedures Suggested changes and the effort required to make the changes Determine whether appropriate files, programs, and procedures are established to enable recovery from a disaster resulting in the loss of data. Comments should address the following: The adequacy and currency of off-site storage procedures The extent that procedures cover the following: Master and transaction data Source and object programs Documentation (such as systems, operations, user manuals) The results of any adequacy-of-recovery test Evaluate the adequacy of the controls on the database, source documents, transactions, and outputs of the system. Comments should address the following: The level of controls present in the entire system and on each component (such as transaction batch and file) The adequacy of the controls, including the strengths and possible areas for improvement The amount of resources required to obtain improvements Review the ability to trace transactions through the system and the tie-in of the system to itself. Comments should address the following: The thoroughness of the audit trails 7 Chapter 3 Post Implementation Review Report Elements The level of improvements necessary, if any The requirements of audit trails as outlined in trusted computer security criteria, such as the Defense Department DOD STD class C2 (controlled access protection) requirements, if any Allowed Access Computer Operations Control of Work Flow Scheduling Evaluate the adherence to restriction of access to data. State desired privacy criteria for the system and then evaluate how the criteria have been followed up to this point. Comments should address the following: Established privacy criteria Recommended privacy criteria Adherence to and violations of privacy The cost of providing this level of privacy The potential effect on individuals if the privacy criteria are not followed The purpose of this section is to ascertain the current level of operational activities. Although the user point of view is primary to the post implementation review report, the computer operations view is also important to investigate. Evaluate the user interface with the data processing organization. Investigate the submittal of source material, the receipt of outputs, and any problems getting work in, through, and out of computer operations. Comments should address the following: Any problems in accomplishing the work Frequency and extent of the problems Suggested changes Effort required to make the changes Determine the ability of computer operations to schedule according to user needs and to complete scheduled tasks. Comments should address the following: Any problems in accomplishing the work Frequency and extent of the problems Suggested changes Effort required to make changes 8 Chapter 3 Post Implementation Review Report Elements User Interface Computer Programming Peak Loads Analyze the usability of the system. The transaction throughput and error rate are included in this analysis. Comments should address the following: Volume of data processed (number of transactions) Number of errors made Frequency of problems with the interface Suggested changes Effort required to make the changes Analyze computer processing issues and problems. Some areas to review are as follows: The correct or incorrect use of forms and off-line files The adequacy of instructions (such as forms lineup and proper responses on the console) The extent of re-runs, if any Assess the ability of the system to handle peak loads and to resolve backlogs when they occur. Any offloading that could be helpful should be investigated. Comments should address the following: The level of user satisfaction The adequacy of the response time (for online systems) The effect of delays on online and/or batch systems Suggested changes The effort required to make the changes Maintenance Activities The purpose of this section is to evaluate maintenance activity involving the system. Activity Summary Maintenance Review Provide a summary of maintenance activity to date. Provide type, number of actions, and scope of changes required. Estimate a projected maintenance workload based on the findings of the review. Discuss the adequacy of maintenance efforts or if a major enhancement/revision is required. Review completed and pending changes to the system. Provide conclusions regarding the benefits to be achieved by completing recommended changes. Provide conclusions about the amount of maintenance required based on activity that has occurred to date. 9 Chapter 3 Post Implementation Review Report Elements System Maintenance Discuss the system maintenance based on the design, types of changes required, documentation, and knowledge about the system (both user and technical personnel). 10 Appendix A Acronyms and Abbreviations CBA INS OIRM PIR SDLC SITD Cost Benefit Analysis Immigration and Naturalization Service Office of Information Resources Management Post-Implementation Review Systems Development Life Cycle Office of Strategic Information and Technology Development A-1 Appendix B Post Implementation Review Report Outline Post Implementation Review Report Cover Page Table of Contents 1.0 Introduction 1.1 Project Identification 1.2 Requesting Organization 1.3 History of the System 1.4 Functional System Description and Data Usage 2.0 Evaluation Summary 2.1 General Satisfaction with the System 2.2 Current Cost-Benefit Justification 2.3 Needed Changes or Enhancements 3.0 Analysis and Implementation 3.1 Purpose and Objectives 3.2 Scope 3.3 Benefits 3.4 Development Cost 3.5 Operating Cost 3.6 Training 4.0 Outputs 4.1 Usefulness 4.2 Timeliness 4.3 Data Quality 5.0 Security 5.1 Data Protection 5.2 Disaster Recovery 5.3 Controls 5.4 Audit Trails 5.5 Allowed Access 6.0 Computer Operations 6.1 Control of Work Flow 6.2 Scheduling 6.3 User Interface 6.4 Computer Processing 6.5 Peak Loads 7.0 Maintenance Activities 7.1 Activity Summary 7.2 Maintenance Review 7.3 System Maintenance B-1 Appendix C Questionnaire Representative Questions For System Proponents and Other Technical Representatives (Attach additional sheets if necessary) System Name: Acronym: A1. Does this system have a predecessor? Yes No A2. If yes, what is it? A3. Briefly describe the history of the system. A4. What are the mission needs that the system meets? A5. What are the information requirements of the system? A6. Briefly describe the system s function. C-1 Appendix C Questionnaire A7. Rate the benefits provided by the system on a scale of 1 to Not Beneficial Indifferent Extremely Beneficial A8. How do the actual benefits of the system compare to those projected in the cost-benefit analysis and functional requirements document? Exceeded stated benefits Met but did not exceed stated benefits Did not meet stated benefits A9. Are there any aspects of the system that you would like to see changed/enhanced? Data Entry Database Communications Report Capabilities Security/Access Control Other None Please describe ideas for change: A13. In comparison to what was established in the feasibility study, the actual scope of the system is: More broad More focused Same scope If the scope differs, please explain how: C-2 Appendix C Questionnaire A14. Do you see any need to make adjustments to the scope? Yes (Please describe) No A15. How adequate was the benefit definition? More than adequate Adequate Not adequate A16. Please describe all the benefits anticipated in the concept development and requirements definition (both quantifiable and not) and any quantifiable resources associated with each. A17. What percentage of anticipated benefits have been realized so far? 0 20% 20 40% 40 60% 60 80% % A18. What percentage of anticipated benefits do you believe will be realized in the future? 0 20% 20 40% 40 60% 60 80% % A19. If there is a difference between planned and realized benefits, what is the reason? A20. What are projected versus actual costs by type (development, operating, etc.)? C-3 Appendix C Questionnaire A21. How adequate were original and subsequent cost estimates? Highly overestimated Overestimated Exact Underestimated Highly underestimated A22. If there are differences between estimated and actual costs, cite possible reasons. A23. Does the system provide
We Need Your Support
Thank you for visiting our website and your interest in our free products and services. We are nonprofit website to share and download documents. To the running of this website, we need your help to support us.

Thanks to everyone for your continued support.

No, Thanks
SAVE OUR EARTH

We need your sign to support Project to invent "SMART AND CONTROLLABLE REFLECTIVE BALLOONS" to cover the Sun and Save Our Earth.

More details...

Sign Now!

We are very appreciated for your Prompt Action!

x