Introduction

Objective

Discuss the challenges of preparing a preliminary Test Concept Document at the solution independent level using the supplied checklist.

Method of Assessment

  • This assessment of suitability of the FIRST POCD for development of an appropriate PTCD is based on the following resources:

    • PTCD TCD Checklist supplied during the course^[1]^

      • While the PTCD / TCD Checklist assumes the existence of a PTCD or TCD the questions are useful for assessing the FIRST POCD

    • Developers Guide for PTCD and TCD, Version 1.1 19 January, 2006^[2]^

      • This checklist is similar but not identical to the checklist provided as a course handout by Viv Crouch.

    • DCDM^[3]^. The supplied PTCD checklist has been used as the primary checklist reference for this assignment.

    • EEET-5127 Assignment 1 Group Presentation^[4]^

  • This assessment of suitability of the FIRST POCD for development of an appropriate PTCD is based on the following criteria:

    • The FIRST Conops will be assessed as if it was the FIRST POCD. As this document is of US origin it has not been developed as per the ADF DCDM Handbook.

    • Numerous assumptions have been made by the author to complete this assignment. Previous assignments on the FIRST OCD provide a reference point as many elements missing from the FIRST OCD have been reengineered.

    • First pass and second pass approval focus on different aspects of capability approval:

      • First pass approval focuses on "reduction of development risk"^[5]^ and

      • Second pass approval focuses on the T&E activities that "assure the acquisition"^[6]^

  1. Assessment of Suitability

    1. Checklist Item 1 - Does the PTCD provided a brief description of the system capability and refer the reader to the POCD for further detail?

The FIRST Conops does not explicitly make reference to a PTCD. The FIRST documentation supplied for the assignment is not technically an OCD due to its US origins. The FIRST Conops is authored by a consultant^[7]^ representing the stakeholders and lacks both technical and user depth of information. All diagrams are simplified and only represent a small part of the necessary system. No organisational structures are presented that would help to define all stakeholders. Other FIRST documentation^[8]^ provides additional detail more relevant to Test and Evaluation activities.

The FIRST proposal describes the vision capability but not the system capability in significant detail to be considered as an appropriate POCD. The lack of operational scenarios describing what if scenarios for worst case events are conspicuous in their absence. The worst case scenarios are critical in determining T&E activities required to develop a PTCD / TCD.

References to the FIRST proposal are skewed to science publications such as New Scientist which perhaps indicates that at this time the FIRST proposal is more significant to the science and technology community rather than the visionaries or governments who will lead the FIRST vision. This may not be a bad thing as the FIRST proposal is highly dependent on the availability of a range of new technologies. The vision component of the FIRST proposal does not begin to be realised until the post transformation phases.

Checklist Item 2 - Are appropriate references, definition, acronyms and abbreviations identified?

A Glossary in Appendix 3 and an acronym list is provided in Appendix 4 of the FIRST Conops.

Checklist Item 3 - Are stakeholders, T&E authority and test agencies identified?

Stakeholders are loosely identified but without the relevant PFPS and PTCD it is necessary to make some major assumptions on the complete stakeholder list. The following stakeholders are assumed to be owners of the FIRST system:

  1. US Department of Defence (DOD) through the Air Force Space Command

  2. North American Space Agency (NASA)

  3. Federal Aviation Administration (FAA) and

The FIRST Conops does not mention countries other than the US or other global stakeholders such as spectrum owners, flight path controllers, etc. For a proposal as wide ranging and forward looking as FIRST it would be reasonable to expect the participation of many more stakeholders. An additional stakeholder could include the US Marines have a future capability requirement to move troops to any part of the world in 2 hours.^[9]^ This proposal suggests using hypersonic transport for this purpose. This capability has many similarities to the FIRST proposal.

Checklist Item 4 - Are Critical Operational Issues (COIs) and Critical Technical Parameters (CTPs) provided? If not developed is the activity identified to develop them?

COIs and CTPs are not clearly articulated in the FIRST Conops. A comprehensive effort to reverse engineer the COIs and CTPs was undertaken as part of an earlier group assignment.^[10]^

The FIRST proposal allows for a transformation period where technological enablers are developed for the FIRST solution. The Transformation phase is appropriate for a proposal as far ranging and visionary as FIRST particularly when so many new technologies and complex systems must be developed. The lack of any concrete coordinated funding for the transformation phase would rate as a significant risk. Evaluation of the related acquisition strategy would be necessary to ascertain to what extent the transformation phase was achievable.

The lack of technical substance and what if scenarios in the FIRST proposal makes determination of the relevant CTPs quite challenging. Whilst it is possible to reverse engineer the CTPs, as was conducted during the earlier group assignment, without a more complete OCD there is significant risk of not identifying all the relevant COIs and CTPs which would lead to an incomplete PTCD / TCD.

Checklist Item 5 - Are references linking the POCD and PFPS (If it exists) to respective COI and CTP provided?

The FIRST Conops supplied does not reference any other documents but the FIRST program has information at its website that presents a more comprehensive view of how the FIRST Conops / OCD fits within the wider program. into .^[11]^ A review of the information available depicts how the FIRST OCD fits within the complete vision as described in the Integrated Concept of Operations.^[12]^

Checklist Item 6 - Do COIs provide the appropriate level of description?

The FIRST Conops makes reference to a number of unknown technologies that will be available during the initial transformation phase that will make the FIRST proposal viable. While the FIRST Conops is visionary it is clearly to be technology led.

For determination of all the COIs with high confidence the FIRST OCD must contain all significant scenarios relating to failure modes of the system during all three phases. Failure modes often stress test systems, processes and capabilities more than typical operational scenarios.

Checklist Item 7 - If developed, are CTPs critical to project success (go/no go), are directly measurable and can only be resolved through T&E?

The system capability and technology roadmaps defined in the ARTWG^[13]^ and ASTWG^[14]^ provide insight into the CTPs critical to the FIRST proposal.

Checklist Item 8 - Does the PTCD describe the acquisition strategy and identify the key milestones in this strategy leading to 2nd Pass and more broadly post 2nd Pass?

The FIRST Conops does not mention an Acquisition Strategy. Scant reference is made to sharing costs during the acquisition of communications satellites only. US Congress has however legislated certain elements required for the transformation phase of the FIRST proposal including a single moon shot in 2018 - 2020.^[15]^

As the FIRST Conops is a US developed publication it does not follow the ADF DCDM process. But information contained in the ARTWG^[16]^ and ASTWG^[17]^ provide insight what systems will be subject to T&E in an effort to "assure the acquisition" as required for Second Pass approval.

Checklist Item 9 - Does the PTCD describe how T&E will be employed leading up to 2nd Pass to mitigate risk and inform source selection?

The FIRST Conops indicates the need for substantial T&E during the transformation phase.

Checklist Item 10 - Is the PTCD linked to key risk areas identified in the TRA?

The FIRST Conops does not provide this information nor does it provide references for further information.

Checklist Item 11 - Is the PTCD T&E actions appropriately synchronised with the S&T plan and the simulation plan in generating answers to key questions and informing key decisions?

The FIRST Conops makes frequent reference to simulation as a means to define, advance and validate subsystem elements of the FIRST proposal such as communications and use of spectrum.^[18]^ Although the FIRST OCD does not provide any indication of how T&E will be aligned with Modeling and Simulation other FIRST documentation, such as the Space Combat Strategic Plan, indicates considerable analysis has been undertaken.^[19]^

Checklist Item 12 - Is sufficient time allowed for T&E?

The FIRST Conops allows for a transformation period (2000-2020) for the development of appropriate technologies required for the subsequent FIRST phases. To answer the question of Is sufficient time allowed for T&E? If we compare the Apollo moon landing to the FIRST proposal we notice a fundamental management difference. The Apollo moon landing was schedule driven while the FIRST proposal is budget driven. Already we see that insufficient funds are being allocated for the known T&E activities such as development of the Shuttle replacement, the canceling of the Shuttle mid life updates and for only a single moon landing during the transformation phase.^[20]^Defence industry experience would indicate that under funded overambitious programs routinely experience schedule slippage and cost overruns.

Checklist Item 13 - Have ADF units which will be required been identified?

The FIRST proposal broadly identifies the military, commercial and civilian units who will have an interaction with the FIRST solution. However a Conops would normally provide more detail and identify specifically the defence units who perform the scenarios in the Conops.

Checklist Item 14 - Have significant resources (eg ships) and facilities (eg a specific test range) been identified?

As the FIRST proposal is dependent on many new as yet undeveloped or unproven technologies it would be difficult to identify which resources would be required for T&E. The FIRST Conops does identify major vehicle elements such as the Crew Exploration Vehicle (CEV) or shuttle replacement but there is still a focus on comparison with a ground based space transportation system as exists today. Post transformation the FIRST solution is intended to be primarily space based and not ground based. The FIRST Conops lacks sufficient technical detail for the proposed vision and perhaps this is due more to the skill set of the author consultant and unfairly presents the FIRST proposal as amateurish when in fact it is highly visionary and will impact most sections of human society.

Checklist Item 15 - Are funds to support the T&E concepts identified?

A number of public news articles discuss FIRST related funding issues such as "insufficient funding for moon shot" due to NASA not receiving an increase to its budget of 0.6% of the US Budget but having to retire the Shuttle early to redirect funding^[21]^

FIRST Baseline Briefings^[22]^indicate that efforts are underway to map costs to the FIRST Integrated Functional Model. A program as complex as the FIRST proposal requires an iterative approach to determine T&E costs as the transformation phase is required to develop new technologies. T&E effort can only be determined once new technologies mature to a sufficiently practical use state.

Checklist Item 16 - Is the information in the PTCD consistent with all other T&E and V&V information across the CDD and supporting project documents?

The FIRST Needs Assessment^[23]^provides a comprehensive description of high level needs and requirements related to the FIRST proposal. A review of T&E documentation could be baselined against the FIRST Needs Assessment for consistency. The ARTWG^[24]^ and ASTWG^[25]^ should also be consulted when reading the FIRST Conops / OCD along with the public available records of meetings that describe how the program is progressing. These records of meeting are available from the FIRST website.^[26]^

In discussions with BAE and DMO staff it appears that typically a PTCD will not be released when the POCD and PFPS are released. I suspect this trend may be due to the lack of familiarity of both the DMO and Australian industry with the process described in the DCDM in addition to the P/TCD being more difficult to develop as solution independent.

Conclusion

The FIRST Conops / OCD is not comprehensive or complete in its presentation of scenarios relating to the FIRST proposal of the three main phases. At the very least the Transformation phase could be better described. The graphics either show discrete human activities or "earth based" rich pictures. What is lacking is diagrams, flow charts and process swim lane diagrams describing the complex and circular interrelationships between operational elements of the FIRST proposal. The FIRST Conops / OCD appears tailor made for public consumption by removing most military references and other public safety / security sections.

This limitation, however, does not limit the creation of a Capability Development Document (CDD) pack comprising an OCD, FPS and TCD. Previous MSI assignments have indicated that it is necessary to have the POCD and PFPS well developed before the PTCD can be substantially progressed. This assumption has also been mentioned by other defence industry employees who have developed CDD document packs. During Assignments to date it has also been difficult to develop the PTCD as a truly solution independent document as major assumptions on platform selection is necessary to scope out the T&E activity costings. From a documentation viewpoint the complex interrelationships between the CDDs would be more readily understood and developed in a relational database structure rather than standalone documents.

At this early stage of the FIRST proposal the issues related to properly funding the transformation phase and coordinating much of the widely distributed research activities leading to transformation appear to be the greatest challenges prior to 2010 in reducing the development risk.

Appendix 1 - Supplied Checklist

PTCD / TCD Checklist Item No. Description PTCD 1 Does the PTCD provided a brief description of the system capability and refer the reader to the POCD for further detail? 2 Are appropriate references, definition, acronyms and abbreviations identified? 3 Are stakeholders, T&E authority and test agencies identified? 4 Are Critical Operational Issues (COIs) and Critical Technical Parameters (CTPs) provided? If not developed is the activity identified to develop them? 5 Are references linking the POCD and PFPS (If it exists) to respective COI and CTP provided? 6 Do COIs provide the appropriate level of description? 7 If developed, are CTPs critical to project success (go/no go), are directly measurable and can only be resolved through T&E? 8 Does the PTCD describe the acquisition strategy and identify the key milestones in this strategy leading to 2nd Pass and more broadly post 2nd Pass? 9 Does the PTCD describe how T&E will be employed leading up to 2nd Pass to mitigate risk and inform source selection? 10 Is the PTCD linked to key risk areas identified in the TRA? 11 Is the PTCD T&E actions appropriately synchronised with the S&T plan and the simulation plan in generating answers to key questions and informing key decisions? 12 Is sufficient time allowed for T&E? 13 Have ADF units which will be required been identified? 14 Have significant resources (eg ships) and facilities (eg a specific test range) been identified? 15 Are funds to support the T&E concepts identified? 16 Is the information in the PTCD consistent with all other T&E and V&V information across the CDD and supporting project documents?

Appendix 2 - DCDM Checklist
===========================

PTCD / TCD Checklist Item No. Description PTCD 1 Does the scope of the PTCD provide a brief description of the proposed capability system being acquired? 2 Does the scope of the PTCD include an outline of the proposed acquisition strategy? 3 Does the PTCD provide clear statements as to what the PTCD objectives are? 4 Does the PTCD reference the more significant project documents that contribute to the development of the PTCD? 5 Does the PTCD document the capability system performance requirements as Critical Issues (COIs and CTPs) that will need to be resolved through a program of Test and Evaluation? 6 Do the COIs and CTPs adequately describe the Critical Issues to be resolved through the results of Test and Evaluation? 7 Does the PTCD provide a description of the Test and Evaluation strategy to be implemented across each phase of the capability life-cycle? 8 Does the PTCD describe in detail the Test and Evaluation strategy and key activities to be conducted between First and Second pass approval? 9 Does the Test and Evaluation strategy identify how the results of T&E will inform key capability approval, contractual acceptance and service release milestones? 10 Does the PTCD make reference to the Technical Risk Assessment (TRA) and provide a strategy to resolve the major risks through a program of Test and Evaluation? 11 Is the PTCD Test and Evaluation strategy synchronised and consistent with the Science and Technology Plan and Simulation Plan? 12 Does the PTCD provide a summary of the major Test and Evaluation activities programmed across each phase of the capability life-cycle? 13 Has sufficient time been allowed in the project schedule to implement the Test and Evaluation strategy and inform key decision milestones? 14 Does the PTCD identify the key Test and Evaluation Authorities, Agencies and Stakeholders? 15 Does the PTCD identify the resources that will be required to achieve the Test and Evaluation objectives? 16 Have the costs to implement the Test and Evaluation strategy been identified?

References

  1. ↑ PTCD / TCD Checklist, UniSA ref Viv Crouch handout dated 28Nov2006 (believed to have been sourced from DTrials)

  2. ↑ Developers Guide for PTCD and TCD, Version 1.1 19 January, 2006

  3. ↑ DCDM 2006

  4. ↑ EEET-5127 Assignment 1

  5. ↑ Developers Guide for Preliminary Test Concept Document (PTCD) and Test Concept Document (TCD),, Version 1.1 19 January, 2006 Foreword pg 1

  6. ↑ Developers Guide for Preliminary Test Concept Document (PTCD) and Test Concept Document (TCD), Version 1.1 19 January, 2006, Foreword pg 1

  7. ↑ Booz | Allen | Hamilton

  8. ↑ http://firstprogram.ksc.nasa.gov

  9. ↑ New Scientist

  10. ↑ Assignment 1, EEET-5127, Space Group

  11. ↑ [http://firstprogram.ksc.nasa.gov/INDEX.CFM]{.underline}

  12. ↑ Integrated Concept of Operations, FIRST as sourced from [http://firstprogram.ksc.nasa.gov/Media/ACF15A4.ppt#434,1,Slide]{.underline} 1

  13. ↑ Advanced Range Technologies Working Group, NASA and US Air Force, September 2003

  14. ↑ Advanced Space Technologies Working Group, November 2003

  15. ↑ New Scientist, 15 January 2004

  16. ↑ Advanced Range Technologies Working Group, NASA and US Air Force, September 2003

  17. ↑ Advanced Space Technologies Working Group, November 2003

  18. ↑ Concept of Operations, for Space Launch and Test Ranges, FIRST

  19. ↑ [http://firstprogram.ksc.nasa.gov/Media/TRMC_Space_Final_Section-Strat_Plan.doc]{.underline}

  20. ↑ New Scientist, 14 January 2006, pg8

  21. ↑ New Scientist, 5 December 2006, [http://space.newscientist.com/article/dn10724-timetable-for-moon-colony-announced.html]{.underline}

  22. ↑ [http://firstprogram.ksc.nasa.gov/Media/Baseline_Briefing.pdf]{.underline}

  23. ↑ Needs Assessment, FIRST sourced from [http://firstprogram.ksc.nasa.gov/Media/ACF2559.doc]{.underline}

  24. ↑ Advanced Range Technologies Working Group, NASA and US Air Force, September 2003

  25. ↑ Advanced Space Technologies Working Group, November 2003

  26. ↑ [http://firstprogram.ksc.nasa.gov]{.underline}

    Continue reading articles in my Systems Engineering series