CornerStone Findings Presentation

10/27/97

· CornerStone Overview

· CornerStone Findings

· Next Steps

CornerStone Overview

Software Process Improvement Initiative (SPII) Goals

Improve the work environment for LaRC's software community, leading to higher morale and increased productivity
Develop sustainable mechanisms for continuous improvement in the productivity and quality of software developed across LaRC
Increase customer satisfaction with LaRC software products

CornerStone Goals

The CornerStone Phase is the foundation of LaRC's overall Software Process Improvement Initiative
The goals of the CornerStone Phase are:

Develop a plan to improve LaRC's software development practices

Identify current state of software development at LaRC
Identify current best practices used in software development at LaRC
Develop a High Performance Model for LaRC's software development activities (incorporates the appropriate elements of the Capability Maturity Model, ISO 9000-3, Strategic and Quality Framework, and Baldrige Award Criteria)

Obtain management's support, complete with resources, to implement LaRC's Software Process Improvement Plan

CornerStone Activities

Lay the Foundation

Establish Infrastructure

Set up CornerStone Team
CornerStone Planning and Approval

Baseline

Workshop Training
Customer Workshops
Supplier Workshops
Follow-Up Interviews (as needed)
Best Practices Documentation
Analyze Workshop Results
Prepare and Present Findings

Plan

Define LaRC Software Engineering Process Group (SEPG)
Define LaRC Management Steering Group
Develop SPI Plan
Review SPI Plan with Sponsors
Present SPI Plan

Implement Improvements . . .

CornerStone Sponsors

Center Sponsors

Kristin Hessenius/Associate Director, LaRC

Fay Collier/ISO 9000 Project Manager, LaRC

Pat Dunnington/CIO, LaRC

Rob Kudlinski/ISSD

Division Sponsors

Doug Arbuckle/FDCD

Bill Wessel/OSEMA

Carl Gray/FSED

Milt Holt/FETD

Lenny McMaster/AESD

Bill Smith/SSCD

David Stephens/FMAD

CornerStone Team Members

Pat Schuler, IOG/ISSD (Team Leader)

Norma Campbell, RTG/FDCD

Victoria Chung, IOG/ISSD

Michael Holloway, RTG/FETD

Chuck Niles, IOG/FSED

Pam Rinsland, IOG/AESD

Jim Townsend, RTG/FMAD

Jim Watson, OSEMA/OMA

Sue Voigt, SASPG/SSCD

Consultant: Cindy Torpey, ChangeBridge Inc.

CornerStone Scope

Center-wide involvement
Organizations involved in software management, development, assurance, training, and maintenance
Customers of software projects
101 civil servants and contractors interviewed

CornerStone Principles

Start with a process framework
Baseline (not audit) current state

Conduct interviews and discussions
Observe strict confidentiality

Involve Center management
Work as a Center-wide team
Focus on LaRC needs

Capability Maturity Model

Key Process Areas (KPAs) Used by CornerStone Activity

Repeatable / Level 2
(All KPAs covered in interviews)

Requirements Management
Software Project Planning
Software Project Tracking & Oversight
Software Quality Assurance
Software Subcontractor Management
Software Configuration Management

Defined Level / Level 3
(Selected KPAs covered in interviews)

Training Program
Intergroup Coordination
Peer Reviews
Software Product Engineering

CornerStone Approach . . .

Baseline current state (snapshot in time)

. . . some improvements are already initiated

Baseline included all areas which impact the software development groups

. . . some are under our control, others will require a coordinated effort

Not all points are at the same detail

. . . some are very specific, other are more general

Processes were assessed, not specific groups

. . . especially when we discuss SQA and Training Program

Identify areas for improvement

. . . not specify how to

CornerStone Findings

Findings Order for Each Key Process Area (KPA)

Purpose and Description
Best Practices
Improvement Opportunities
Consequences

Index to Findings

Repeatable (Level 2) KPAs

Requirements Management

Software Project Planning
Software Project Tracking & Oversight
Software Quality Assurance
Software Subcontractor Management
Software Configuration Management

Defined (Level 3) KPAs

Training Program
Intergroup Coordination
Peer Reviews
Software Product Engineering

Requirements Management

Purpose

Establish a common understanding of the customer's requirements that will be addressed by the software project

Description

Establish and maintain an agreement with the customer on the requirements for the software project
The "customer" is a system engineering group, TAG, a project office, another internal organization, or an external customer
Agreement covers both the technical and nontechnical (e.g., delivery dates) requirements
Agreement forms the basis for estimating, planning, performing, and tracking the software project's activities throughout the software life cycle

Best Practices

Requirements are prioritized and managed according to priority
Developer works with customer to interactively draw out requirements
Operational Concepts Documents help in understanding the requirements (some projects put on the web for increased visibility)
Test cases traceable back to requirements
Explicit contractor task to capture and manage requirements
Rapid prototyping to validate requirements with customer

Improvement Opportunities

Importance of requirements is not realized
Requirements are neither defined early enough, nor refined far enough
Inadequate resources dedicated to requirements management
Software requirements do not track to system requirements
Don’t know how to adequately document requirements
Customers are not trained in requirements generation
Changes in requirements handled by overtime masks over-commitment
Requirements creep is not managed nor controlled (cost is not understood)
No established policy to guide projects in requirements management
Role and responsibility of systems analysis is not clearly understood
Requirements are not consistently documented and reviewed with the customers
Segmentation of responsibilities between researchers and software developers leads to uncertainty in product functionality

Consequences

Project schedule and cost are significantly increased when requirements are inadequately defined and documented
Rework is common due to changing requirements
Customer satisfaction is decreased when their requirements are not met

Software Project Planning

Purpose

Establish reasonable plans for performing software engineering and managing the software project

Description

Develop estimates for the work to be performed, establish the necessary commitments
Define and document the plan to perform the work

Best Practices

Project cost, effort and duration estimates are developed and tracked
Software Work Breakdown Structure is generated at the same level as hardware WBS
Scheduling tools (MS Project, REVIC) are used
Statements of Work specify creation of project plans
Software staged release at planned milestones
Software manager designated to oversee a software development project
Software personnel participate in project plans
SQA involved in project planning

Improvement Opportunities

Schedules are driven by externally set milestones, instead of actual work required
Software Management Plans are not consistently documented, if at all
No planning metrics are captured across the organization on which to base realistic future estimates
Commitments are not honestly negotiated
Inadequate project planning is compensated for by overtime
No LaRC project policy for managing software projects - the approach to software project management is project dependent
The value of software project plans is not fully understood by developers, managers, or researchers
Personnel are not aware of available procedures and guidelines for software estimating
No guidelines are available for documenting software development plans

Consequences

Software projects do not meet schedules or budgets (could face cancellation)
Over-commitment causes burnout of civil servants and contractors which leads to replacement costs and schedule delays

Software Project Tracking and Oversight

Purpose

Provide adequate visibility into actual progress so that management can take effective actions when the software project's performance deviates significantly from the software plans

Description

Track and review the software accomplishments and results against documented estimates, commitments, and plans
Adjust plans based on the actual accomplishments and results

Best Practices

Informal interchange meetings are held frequently for tracking
Automated tools (MS Project, PC Artemis) are used to track project progress
Milestones used as checklist
Formal reviews held at selected milestones
Dedicated business manager to track software project status

Improvement Opportunities

Corrective actions are not taken in a timely manner
Software costs are not accurately reflected in project charge structures
No measurements or lessons learned are captured and used for future projects
Software size is not estimated or tracked
Managers are not trained in software project management and tracking
Technical issues are not managed and communicated
Software Development Plans are not used to manage the project
Software project reviews are ad hoc and ineffective; results are not elevated to management (guidelines needed)

· Consequences

Products not completed on expected dates
The real software project status is not known until it is too late for corrective action

Software Quality Assurance

Purpose

Provide management with appropriate visibility into the process being used by the software project and of the products being built

Description

Review and audit the software products and activities to verify that they comply with the applicable procedures and standards
Provide the software project and other appropriate managers with the results of these reviews and audits

Best Practices

SQA contractors advise and give consultation to projects on software engineering when requested (on configuration management and software management plan)
Infractions are reported to participating project teams
Airworthy flight software has met FAA standards

Improvement Opportunities

Need sufficient staff for uniform coverage of SQA on Langley software projects
Increase awareness of added value of SQA practices
Develop usable guidelines for SQA activities (current LHB not widely accepted)
Review SQA activities on a regular basis
Track cost and associated return on investments on SQA tasks

Software Subcontractor Management

Purpose
Identify qualified software subcontractors and manage them effectively

· Description

Select a software subcontractor
Establish commitments with the subcontractor
Track and review the subcontractor's performance and results

· Best Practices

Periodic reviews are conducted with contractors to review progress and communicate status
COTRs and Technical Monitors are designated to establish and manage the software contract
Source Evaluation Boards recommend contractors based on their ability to perform the work
Success has been demonstrated using Performance Based Contracting
Formal quarterly evaluations are required
Software services are acquired using standard NASA contracting regulation>

· Improvement Opportunities

No accountability for poor contractor performance
COTRs and Technical Monitors are not trained in software engineering or managing software development efforts
Performance Based Contracting is misunderstood and ineffectively applied
Source Evaluation Board frequently recommends lowest bidder rather than best qualified
Required reviews are not consistently conducted
Contractor turnover is high and requires frequent re-orientation resulting in increased costs
No written LaRC policy or guidelines for managing software contracts
Contractors over-commit and rely on "free" overtime
Skill level and training on the contract does not match what is required

· Consequences

Frequent dissatisfaction with contractor's products
PBC task generation is more time-consuming

Software Configuration Management

Purpose

Establish and maintain the integrity of the products of the software project throughout the project's software life cycle

Description

Identify the configuration of the software (i.e., selected software work products and their descriptions) at given points in time
Systematically control changes to the configuration
Maintaining the integrity and traceability of the configuration throughout the software life cycle
Manage both the software products delivered to the customer (e.g., the software requirements document and the code) and the items identified with or required to create these software products (e.g., compiler)

Best Practices

Configuration Control Boards are established and used to control changes, assess impacts and communicate status
Automated tools (ClearCase, PVCS, RCS, LibLink, CVS, LabView) are used to control software work products (requirements, change rationale, code and supporting documentation)
Intent of SCM can be met without the use of automated tools for projects
Web-based change request system established (ClearDDTS)
SCM process documented and followed on some projects

Improvement Opportunities

Rigor of formal SCM is not always appropriate for project's size, phase and purpose
Change request status is not adequately communicated to all project personnel
Lack of awareness of existing guidelines and templates results in significant duplication of effort
Insufficient funding for SCM staff, tools and training
SCM typically applied too late in project phase
SCM plans are not documented and/or followed and changes are not controlled
Lack of understanding of the benefits of SCM
Contracts do not always require the appropriate level of SCM and there is inadequate SCM on deliverables

· Consequences

Lack of SCM results in compromised mission certainty and validity of data and products

Training Program

Purpose

Develop the skills and knowledge of individuals so they can perform their roles effectively and efficiently

Description

Identify the training needed by the organization, projects, and individuals
Develop or procure training to address the identified needs
Evaluate current and future skill needs and determines how these skills will be obtained
Use informal vehicles when appropriate (e.g., on-the-job training and informal mentoring)

Best Practices

Training Office exists to meet the needs of LaRC including the software development community
Some projects and organizations successfully leverage the capabilities of the Training Office to meet their needs
Training Office annually surveys LaRC to establish training needs

Improvement Opportunities

Increase is needed in training budget in order to cross train and retrain LaRC personnel
Inconsistent management support for training
Insufficient time is allocated for attending training
Training is not considered a priority
Need full implementation of Employee Development Plan (EDP) which ties to the SQF and the agency strategic plan
No effective mechanism for consistently training contractors and civil servants working on the same project
No project training plan
Travel funds shortage limits training opportunities and technical exchange

Intergroup Coordination

Purpose

Establish means for the software engineering group to participate actively with the other engineering groups so the project is better able to satisfy the customer's needs

Description

Software engineering group participates with other project engineering groups to address system-level requirements, objectives, and issues
Representatives of the project's engineering groups participate in establishing the system-level requirements, objectives, and plans by working with the customer and end users, as appropriate
Requirements, objectives, and plans become the basis for all engineering activities

Best Practices

Workshops and shared facilities bring different groups together for improved information exchange
Interface Control Documents, frequent meetings and timely meeting notes ensure effective exchange of information
Web-based information and e-mail exchange facilitates technical coordination
Physical co-location of discipline specialists promotes intergroup coordination during a project
Teamwork training is available
Communication between software programs used by different groups achieved by in-house developed interfaces (scripts, GUIs, wrappers, standard features, filters)

Improvement Opportunities

Need to have representatives of all technical disciplines (including software) involved from the start of a project (requirements, cost estimates, operational concepts, requirements allocation)
Due to the lack of systems engineering performed at LaRC, software and hardware staff are forced to fill that role in an ad hoc manner
Poor communication among groups doing similar work within LaRC results in duplication of work
No effective mechanism to ensure all disciplines are represented on project teams
Software specialists not aware of overall project concept
Perception that software can fix anything, including poor hardware selection, leads to wasted funds and staff hours
Lack of documented commitments between engineering groups
Changes are not effectively communicated across the engineering groups
Contractors resist communication due to proprietary fear

Software Product Engineering

Purpose

Consistently perform a well-defined engineering process that integrates all the software engineering activities to produce correct, consistent software products effectively and efficiently

Description

Perform the engineering tasks to build and maintain the software using the project's defined software process and appropriate methods and tools

Best Practices

Operational Concept, Users Manual and Test Plans/Cases developed prior to code
Apply techniques such as Object Oriented Design (OOD), reverse engineering, structured programming, and modularity to reduce code complexity, increase reusability and improve maintainability
Automated software development tools (Purify, Quantify, PureCoverage, Rational Rose, ClearCase, ClearDDTS, Object Manual, LabView GUI) are used to design, test, configuration management, and document software
Provide realistic operational testing scenarios for software
Office automation tools used in innovative ways for software development, testing and documentation (databases, spreadsheets)
A project has demonstrated LaRC’s ability to meet FAA standards for airworthy flight software
On line Web sites exist that describe software engineering guidebooks, formal methods and metrics collection database
Third party tests against requirements agreed to by users
Continuous rapid prototyping used to demonstrate feasibility and early progress
Requirements gathering techniques resulted in improved software product
Centralized facility provides access, guidance and expertise for domain-specific tools

Improvement Opportunities

Insufficient time allocated for software engineering tasks (requirements definition, design, testing, integration, configuration management, and documentation)
Software engineering activities often hidden because their value is not recognized, perceived as overhead by projects, and researchers/projects do not want to pay for reuse (no corporate view for long-term investment)
No rewards for good software engineering
Single point failures on many projects created by one person and insufficient documentation
Testing address physics only, not software quality
No LaRC dissemination guidelines for software exist
Lack of in-house software product engineering expertise
Need point of contact where software engineering expertise and guidance can be obtained
Ineffective tool utilization due to poor communication and awareness
Web sites not advertised
Software engineering techniques not appropriately tailored for small projects
No incentive to take research code to production quality or make it reusable by other projects
Lack of expertise in project planning and scheduling leads to insufficient time for fundamental software engineering tasks

Software Product Maintenance

Best Practices

Use portable computer to set up, perform experimental test, and analyze results

Improvement Opportunities

Insufficient funding for sustaining and maintenance of software
Ineffective implementation of configuration management prohibits quick minor changes to software for experimental use
Insufficient verification of software prior to delivery leads to high maintenance cost
Due to insufficient manpower and maintenance procedures, staff is not adequately notified on software changes
Constant changes in COTS upgrade decrease productivity due to perpetual learning curve

Peer Reviews

Purpose

Remove defects from software products early in the development process where it is more cost effective to remove them
Develop better understanding of software products and defects that might be prevented

Description

Peers methodically examine software work products to identify defects and areas where changes are needed
Identify and schedule specific products that will undergo peer review as part of the defined software process

Best Practices

Peer reviews identify problems early in the lifecycle resulting in saved time and decreased costs
Post-mortem review is effective mechanism to capture lessons learned
Informal review by an in-group peer works well on small projects
Peer reviews captures problems not otherwise identified
Completion of peer reviews is a phase exit requirement and indicates readiness to proceed
Peer reviews are effective mechanism to train staff and indoctrinate new hires
Peer review process is documented (NASA Formal Inspection Handbook, project documentation)
User requirements are basis for code reviews
E-mail and Key Activities are used to document peer review results

Improvement Opportunities

Peer reviews are not commonly used (even unknown by some projects) in spite of positive return on investment experienced at LaRC
Current review process frequently omits software issues
Current formal design review system (PDR, CDR) often misses large problems
Post-mortem reviews hardly ever held
Limited time, training, and staff are provided for performing peer reviews
No data collected on peer review results and lessons learned (dormant data base)

Related Concerns

Improper ISO 9000 implementation could impose restrictive procedures on research and significantly impact resources to perform future research
Numerous management issues were identified, such as:

The lack of management awareness of the pervasiveness and magnitude of work required to develop software at LaRC
Need investment, encouragement, and reward system for improvements, best practices, positive technical transfer
Lack of adequate resources for performing software development

Numerous communication issues were identified:

Stove piped developers need to share expertise and lessons learned
Need effective interaction with Performance Based Contractors and customers

Relation of LaRC's Strategic Quality Framework to real work is poorly defined

Next Steps

Critical Point in LaRC's SPI Program

CornerStone phase is a critical milestone in the software process improvement program
CornerStone phase baselines our current state and develops a plan for future process improvement
Proactive management is required to go forward with successful implementation of the Improvement Plan

Upcoming Activities

Receive feedback on Findings Briefing
Establish Management Steering Group (MSG)
Establish Software Engineering Process Group (SEPG)
Develop draft Software Process Improvement Plan
Review SPI Plan with sponsors & MSG
Finalize SPI Plan
Conduct SPI Plan Briefing
Implement Improvement Plan

Management Steering Group Description

Comprised of the LaRC CIO, ISO Project Manager, and Division Chiefs who have a stake in the successful achievement of the Software Process Improvement (SPI) goals
Works together with the SEPG to identify and provide the long-term commitments and resources required to coordinate the development and maintenance of software engineering processes for use by current and future software projects
Monitors and evaluates SPI efforts from the perspective of the total organization to ensure that the overall improvement efforts are in concert with LaRC’s mission and goals
Meets regularly with the Software Engineering Process Group (SEPG) to discuss the progress of the SPI program, problems, issues and concerns

Suggested Management Steering Group Membership

Doug Arbuckle/Associate Director, LaRC
Fay Collier/ISO 9000 Project Manager, LaRC
Pat Dunnington/CIO, LaRC
Luat Nguyen/FDCD
Burt Garrido/OSEMA
Carl Gray/FSED
Milt Holt/FETD
Rob Kudlinski/ISSD
Lenny McMaster/AESD
Bill Smith/SSCD
David Stephens/FMAD

Responsibilities of the Management Steering Group (MSG)

Ensure alignment of Software Process Improvement (SPI) initiatives with LaRC mission and goals
Approve strategic plan for SPI
Provide sponsorship, pro-active commitment, and visible management support
Allocate resources
Monitor the progress of the SPI initiative
Provide guidance and direction to the Software Engineering Process Group (SEPG)
Conduct regular meetings with the SPI Group
Promote cooperation and cross-functional communications
Obtain and sustain the support for the SPI initiative

Software Engineering Process Group (SEPG) Description

Focal point for software process improvement
Coordinate and plan the SPI program activities in concert with the guidance and direction of the MSG
Report the progress of SPI related activities to the MSG
Provide technical support and consultation
Staffed by experienced software development practitioners who facilitate the definition, maintenance, and improvement of the software engineering processes used within the organization
Work collaboratively with the projects to resolve process related problems and assist in the development, training and implementation of processes and procedures

Suggested Software Engineering Process Group (SEPG) Membership

Members of the CornerStone team
Volunteers from interview workshops
Appointees from participating divisions

Responsibilities of the Software Engineering Process Group (SEPG)

Define and manage the plan for development and implementation of software process improvements across the LaRC
Provide a pool of software engineering expertise and corporate knowledge (Formal part of LaRC organization with assigned resources and management commitment)
Provide consultation and guidance on appropriate level of software engineering implementation and future directions
Provide and facilitate education on software engineering to management and staff via workshops, seminars, symposiums, and setting up news/user groups and maintain Web site
Provide a repository for reuse code, documents, tool knowledge, procedures, processes, LaRC best practices, templates, lessons learned, metrics, and examples
Facilitate sharing of tools and maintenance of COTS across projects
Build and reinforce management support for the SPI program

Next Steps

Establish membership of MSG & SEPG (by Nov. 12)
Post Best Practices on Web
Develop SPI Plan
Review and Obtain Approval of Plan (by Dec. 19)
Complete Final Report
Implement Improvement Plan

Closing Thought

"Creative thinking may simply be the realization that there is no particular value in doing things the way we've always done them."

- R. Flesch, Educator