Proposed Content for an Algorithm Change Protocol (ACP)An ACP is a description of the set of specific methods that a manufacturer has in place to achieve and appropriately control the risks of the anticipated types of modifications delineated in the SPS. The ACP provides a step-by-step delineation of the data and procedures to be followed so that the modifications achieve their goals and the device remains safe and effective after the modification. The description below is intended to highlight some components of an ACP, but is not intended to be an exhaustive list of ACP components:Food and Drug AdministrationFDA_85a3d7be-fbea-11df-a90e-29337a64ea2a_08a72f26-5a1f-11e9-9ee9-02f17d73fa26To define the content of an Algorithm Change Protocol (ACP) for AI/ML-based software as a medical device_08a7305c-5a1f-11e9-9ee9-02f17d73fa26Data ManagementIssue plans addressing how data will be collected, added to existing data sets, and used._08a73124-5a1f-11e9-9ee9-02f17d73fa261Data management plan addressing how data will be collected, added to existing data sets, and used: This data management plan may include a quality assurance (QA) plan for determining which new data are appropriate for inclusion as part of an expanded training data set; an approach to the reference standard determination; a data augmentation strategy that allows for additional training and independent test data to be added; and an auditing and sequestration strategy to monitor, document test dataset independence, and control access to both the training and test datasets as additional data are being included and any revised algorithm is being retrained and tested.Quality AssuranceInclude a quality assurance (QA) plan for determining which new data are appropriate for inclusion as part of an expanded training data set_08a731d8-5a1f-11e9-9ee9-02f17d73fa261.1Reference StandardsInclude an approach to the reference standard determination_08a7328c-5a1f-11e9-9ee9-02f17d73fa261.2Data AugmentationInclude a data augmentation strategy that allows for additional training and independent test data to be added_08a73340-5a1f-11e9-9ee9-02f17d73fa261.3Auditing & SequestrationInclude an auditing and sequestration strategy to monitor, document test dataset independence, and control access to both the training and test datasets as additional data are being included and any revised algorithm is being retrained and tested._08a733ea-5a1f-11e9-9ee9-02f17d73fa261.4Re-Training & OptimizationDefine protocols for re-training / optimizing medical device algorithms._08a734a8-5a1f-11e9-9ee9-02f17d73fa262Protocols for re-training / optimizing the medical device algorithm: These protocols may include a retraining strategy that describes the objective of the retraining; the algorithm components that may be modified as a result of the learning process; and any criteria that must be met during the re-training process to trigger a more comprehensive performance evaluation using the test dataset.ObjectivesDescribe the objective of the retraining_08a7355c-5a1f-11e9-9ee9-02f17d73fa262.1Algorithm ComponentsDescribe the algorithm components that may be modified as a result of the learning process_08a73610-5a1f-11e9-9ee9-02f17d73fa262.2CriteriaDescribe any criteria that must be met during the re-training process to trigger a more comprehensive performance evaluation using the test dataset._08a736ce-5a1f-11e9-9ee9-02f17d73fa262.3Performance EvaluationEvaluate the performance of the algorithms._08a7378c-5a1f-11e9-9ee9-02f17d73fa263Performance evaluation protocols: These protocols may include a description of the intervals of when a new algorithm may be trained and evaluated to consider updating the medical device algorithm; the delineation of appropriate metrics and analysis procedures; statistical analysis plans; appropriate measures to minimize information leakage about the test data set if part of it is re-used in multiple evaluations; performance targets that the revised algorithm must achieve; and protocols for testing, which may be applicable for that device and type of change, for example, for testing “with clinicians in the loop,” as appropriate. IntervalsDescribe the intervals of when a new algorithm may be trained and evaluated to consider updating the medical device algorithm_08a7384a-5a1f-11e9-9ee9-02f17d73fa263.1Metrics & AnalysisDelineate appropriate metrics and analysis procedures_08a73926-5a1f-11e9-9ee9-02f17d73fa263.2Statistical AnalysesDocument statistical analysis plans_08a739f8-5a1f-11e9-9ee9-02f17d73fa263.3Information LeakageInstitute appropriate measures to minimize information leakage about the test data set if part of it is re-used in multiple evaluations_08a73ab6-5a1f-11e9-9ee9-02f17d73fa263.4Performance TargetsSpecify performance targets that the revised algorithm must achieve_08a73b88-5a1f-11e9-9ee9-02f17d73fa263.5TestingDefine protocols for testing, which may be applicable for that device and type of change, for example, for testing “with clinicians in the loop,” as appropriate. _08a73c5a-5a1f-11e9-9ee9-02f17d73fa263.6UpdatesDescribe how updated medical device algorithms will be tested, distributed, and communicated when released._08a73d22-5a1f-11e9-9ee9-02f17d73fa264Update procedures that describe how updated medical device algorithms will be tested, distributed, and communicated when released: These update procedures may include a description of the update plan including expected frequency of updates and whether the updates will be global (all devices use the same version of the algorithm) or local (multiple versions of the algorithm targeted for specific subpopulations are distributed); version tracking and control; obsolescence planning; requirements for host software/hardware requirements; any plans for ‘beta’ release of the updated medical device algorithm concurrent with the previous version; and communication procedures that describe how users will be notified of updates and any information that will be conveyed to users about the update.PlanInclude a description of the update plan_08a73e4e-5a1f-11e9-9ee9-02f17d73fa264.1VersioningAddress version tracking and control_08a73f20-5a1f-11e9-9ee9-02f17d73fa264.2ObsolescencePlan for obsolescence_08a73fe8-5a1f-11e9-9ee9-02f17d73fa264.3Software & HardwareSpecify requirements for host software/hardware requirements_08a740f6-5a1f-11e9-9ee9-02f17d73fa264.4Beta ReleasesDocument any plans for ‘beta’ release of the updated medical device algorithm concurrent with the previous version_08a741d2-5a1f-11e9-9ee9-02f17d73fa264.5CommunicationsDocument communication procedures that describe how users will be notified of updates and any information that will be conveyed to users about the update._08a7429a-5a1f-11e9-9ee9-02f17d73fa264.62019-04-022019-04-08https://www.regulations.gov/document?D=FDA-2019-N-1185-0001OwenAmburOwen.Ambur@verizon.net