AllMeds 2022 Real World Test Plan

AllMeds Official Real World Test Plan can be accessed via the following link: 

The key components of this document are detailed below.

Executive Summary

This is the real-world test plan for CY 2022 for AllMeds’ AllMeds Real World Test Plan 2022.pdfSpecialty EHR certified EHR solution. We have one version certified, and we will be testing on the most current version, 12 which is deployed to our provider partner community. As ONC has stated in its rule, “The objective of real-world testing is to verify the extent to which certified health IT deployed in operational production settings is demonstrating continued compliance to certification criteria and functioning with the intended use cases as part of the overall maintenance of a health IT’s certification.” We have worked toward this objective in designing our test plan and its subsequent real-world testing measurements and metrics. 

This document builds toward the final testing measurements and metrics we will use to evaluate our product’s interoperability within production settings. Within each use case, we document our planned testing methodology, associated ONC criteria, justification for measurement, expected outcomes from the testing, care settings applied for the respective measure, and if applicable the number of clients to use our real-world testing approach, including how our test cases were created, our selected methodology, the number of client/practice sites to use, and our general approach and justification for decisions. 

We have included our timeline and milestones for completing the real-world testing in CY 2022, and information about compliance with the Standards-Version Advancement Process updates. 

General Information

Timeline and Milestones for Real-World Testing CY 2022 

  • 1Q-2022 - Communication: Begin communication with clients to ask for their support and participation in real-world testing. The goal is to have a sufficient number of clients committed for real-world testing by the end of 1Q-2022. 
  • 2Q-3Q 2022 - Testing: During the 2nd and 3rd quarters of CY 2022, real-world testing with clients will be scheduled and performed. It is expected that a preparatory call will be done with clients to prepare them for testing activities. Results will be documented in the test results section of the test methods and ultimately used to build the test report. If any non-compliances are observed, we will notify the ONC-ACB of the findings and make the necessary changes required. 
  • 4Q-2022 – 2023 Real World Test Plan: During the 4th quarter of CY2022, the CY 2023 real-world test plan will be completed according to ONC and ONC-ACB requirements and expectations. The test plan will be prepared for submission before the end of the year.
  • February 2023 – 2022 real World Test Plan Results: AllMeds will document CY 2022 test results into our Real World Test Report and submit them to our ONC-ACB.

Standards Version Advancement Process (SVAP)

For CY 2022, AllMeds is not planning to make any version updates on approved standards through the SVAP process. This applies to all test scenarios described within.

  • Standard (and version): None
  • Updated certification criteria and associated product: N/A 
  • Health IT Module CHPL ID: N/A 
  • Method used for standard update:  N/A
  • Date of ONC-ACB notification:  N/A 
  • Date of customer notification (SVAP only): N/A
  • Conformance measure N/A USCDI-updated certification criteria (and USCDI version): N/A

Justification for the Real World Testing Approach

AllMeds’ real-world testing is focused on meeting the ever-changing demands of our provider partners and the industry as a whole. This focus will be centered around ensuring scenarios that independent ambulatory physician practices expect to encounter in a changing landscape are captured. Our approach will be to generate a test plan that satisfies key workflows (i.e. receiving an electronic summary of care and incorporating the data into the EHR and submitting an electronic CCDA referral summary). We will use objective measurements including data log reviews, API log inspection, surveys, and manual verification using third-party tools to ensure data is correctly retrieved or updated according to industry standards.  We will set thresholds on acceptable success rates to ensure our partners have confidence in the data exchanges.

Real-World Testing Measurements 

The measurements for AllMeds’ real-world testing plan are described below. 

Each measure contains the following elements:  

  • Associated ONC criteria
  • Testing Methodology used 
  • Description of the measurement/metric 
  • Justification for the measurement/metric 
  • Expected outcomes in testing for the measurement/metric 
  • Number of client sites to use in testing (if applicable) 
  • Care settings that are targeted with the measurement/metric

In each measure evaluated, we elaborate specifically on our justification for choosing this measure and the expected outcomes. All measurements were chosen to evaluate compliance with the certification criteria and interoperability of exchanging electronic health information (EHI) within the certified EHR.

Testing Methodologies 

For each measurement, a testing methodology is used. For our test plan, we use the following methodologies. 

  • Audit Trail/ Reporting: This methodology uses the audit logging or various reporting capabilities of the application to examine tasks performed in the system. This methodology often provides historical measurement reports which can be accessed at different times of the year to evaluate interoperability. It can serve as a benchmark for evaluating real-world testing over multiple time intervals.
  • Third-Party Software Confirmation or Attestation:  This methodology leverages industry-standard or industry-required technology and services to evaluate data sharing.  Other third-party software may be used as well to simulate or confirm activities when another option is not available. It may also be necessary to receive attestation reports from third-party applications to verify the receipt and accuracy of data when access to the third-party system is unavailable or prohibited. 
  • Manual Chart Review:  This methodology leverages human intervention to visually review and confirm changes to data as expected.  When data is shared to the application that may cause a change to a patient’s medical chart, it may be necessary for a human to review the expected change and sign-off that the update occurred as expected.
  • User Survey: This methodology evaluates interoperability and compliance of EHR Module capabilities through feedback from users. This methodology can provide insight into how clinicians employ and use a feature that reveals the actual value and impact of interoperability of the EHR Module. 

Number of Clients Sites
 For each measure, we denote the minimum number of practice partners we plan to use for the evaluation. The numbers vary depending on the methodology as well as overall use of the associated EHR Module criteria by our users. For criteria that are not widely used by our customer base, we may test the respective measure in our own production-sandbox environment given the lack of customer experience with the criteria functionality. 

Practice Settings Targeted 

AllMeds’ Specialty EHR application focuses on serving the needs of independent ambulatory medical practices across the United States. Our Real-World Test Plan is designed with these practice settings in mind. In each measure, we do also address the care settings targeted and note any necessary adjustment or specific factor to consider with this specific measure.

Real-World Test Case: Comprehensive Care Coordination

Measurement Overview

One of the most critical steps for users of our application is to facilitate the seamless flow of data that is captured during an encounter and then share that data when necessary, with those interested in the patient’s care.  It is our opinion that the exchange of data is both multi-faceted and interconnected and creates the best outcome for those interested in comprehensive care coordination. A routine medical visit should result in data sharing events across the medical industry to key stakeholders with the limited burden on the provider/practice entering the data.

This comprehensive test case will cover the workflow in which a patient is seen at a medical practice for a routine visit then referred to another provider for additional care. We will start with the assumption that the provider treating the patient receives a clinical summary document. This document will be reviewed and incorporated into the medical record.   Following the visit, data will flow to third-party data registries based on data captured. The data will also flow to the patient via the patient portal. Key phrases are covered as follows:


Phase 1: Records received and reviewed for a patient prior to a medical visit.

Scenario 1.A - Receive a clinical summary for an upcoming visit from an alternate provider via DIRECT Email (170.315(h)(1) Direct Project)

  • Description: Receive a clinical summary for an upcoming visit from an alternate provider via DIRECT Email
  • Associated Certification Criteria: 170.315(h)(1) Direct Project
  • Justification:  The use case will describe the case in which a provider transitions a patient to the care of a provider/practice using AllMeds Specialty EHR. The provider transitioning the patient will be able to send a DIRECT e-mail containing a CCDA document to the provider using AllMeds Specialty EHR. We will use audit logs and manual verification to confirm the successful receipt of the DIRECT Email. Providers will be able to confirm the validity of the CCDA via the CCDA Validation tool built within AllMeds Specialty EHR.
  • Testing Method: Manual Verification and Audit Trail Review
  • Expected Outcomes:
    • Audit trail verification of a new DIRECT E-Mail received for the desired user. The audit trail should accurately reflect the count of records to increase by 1 following a successful transmission.
    • Manual verification of a DIRECT E-Mail received in AllMeds Specialty EHR Communications Inbox should reveal 1 new record in the inbox.
    • Successful validation of the CCDA via CCDA Validation in AllMeds Specialty HER with an error rate of less than 10%.
  • Care Settings and Number of Clients Site to Test: We designed this measure to test general ambulatory sites that we support and target. We will test a minimum of three (3) client practice(s). This number covers a sufficient percentage of existing practices to provide a viable sample of users of the certified EHR.

Scenario 1.B - Review clinical care summary and incorporate medication/allergy changes from the CCDA (170.315(b)(2) Clinical Information reconciliation and incorporation)

  • Description: Review clinical care summary and incorporate medication/allergy changes from the CCDA
  • Associated Certification Criteria: 170.315(b)(2) Clinical Information reconciliation and incorporation
  • Justification:  The use case will describe the case in which a provider receives a transition of care summary (CCDA) from a referring provider then subsequently associates the CCDA with a patient record in AllMeds Specialty EHR. When the provider loads the chart that the CCDA was linked to, the user should be prompted to reconcile medications and allergies from the CCDA with the medications and allergies on files in the patient chart. Users will be able to manually see the comparison of medications and allergies along with suggested changes. Once reconciled, we can use audit trails and manual verification to ensure the patient chart is correctly updated with the reconciled information.
  • Testing Method: Manual Verification and Audit Trail Review
  • Expected Outcomes:
    • Audit trail verification of a new DIRECT E-Mail received for the desired user. The count of audit records should increase by 1 following the successful receipt.
    • Manual verification of the CCDA Reconciliation Process should reveal medications and allergies supplied in the CCDA and medication and allergies stored on the patient record within Dr. First. For each new record merged into the chart, the count of rows in the audit table should increase by 1.
    • Following the successful reconciliation of data, the audit trail and manual chart review should show the addition of added medications or allergies or the status change of medication or allergy.

  • Care Settings and Number of Clients Site to Test: We designed this measure to test general ambulatory sites that we support and target. We will test a minimum of three (3) client practice(s). This number covers a sufficient percentage of existing practices to provide a viable sample of users of the certified EHR.

Phase 2:  Complete a routine medical visit with a patient

Scenario 2.C - Refer a patient to a different provider for additional care (§170.315(b)(1) Transitions of care)

  • Description: Refer a patient to a different provider for additional care 
  • Associated Certification Criteria: §170.315(b)(1) Transitions of care 
  • Justification:  The use case will describe the scenario in which a provider decides to refer the patient that they are seeing to another provider for additional care. The provider will place a referral order, then send a transition of care clinical summary via DIRECT Email. It will provide a numeric value to indicate both how often this interoperability feature is being used as well as its compliance with the requirement. An increment to this measure indicates that the EHR can create a C-CDA patient summary record, including the ability to record all clinical data elements, and by sending the C-CDA patient summary record, the EHR demonstrates successful interoperability of an exchanged patient record with a 3rd party. This measurement shows support for Direct Edge protocol in connecting to a HISP for successful transmission which reveals compliance to the associated criteria listed above
  • Testing Method: Audit Trail Review, Third-Party Application Attestation
  • Expected Outcomes:
    • Audit trail should reveal a new transition of care summary being sent via DIRECT Email. For each new record sent, the audit trail record count should increase by 1.
    • Third-Party Attestation from the provider that the DIRECT Email was sent to was sent to should be obtained with 0 errors.
  • Care Settings and Number of Clients Site to Test: We designed this measure to test general ambulatory sites that we support and target. We will test a minimum of three (3) client practice(s). This number covers a sufficient percentage of existing practices to provide a viable sample of users of the certified EHR.

Phase 3: Patient provided access to their clinical information post-visit

Scenario 3.A - Patient obtains access to their clinical summary following a medical visit (170.315(e)(1) View, download, and transmit to 3rd party)

  • Description: Patient obtains access to their clinical summary following a medical visit Associated Certification Criteria: 170.315€(1) View, download, and transmit to 3rd party 
  • Justification:  The use case will describe the scenario in which a provider practice completes a routine medical visit and supplies patients with access to their clinical summary via the patient portal. The measure will evaluate the ability to successfully create a new patient portal account for a new patient and the ability to track if the patient viewed, downloaded, or transmitted their clinical care summary (CCDA).
  • Testing Method: Audit Trail Review
  • Expected Outcomes:
    • Audit trail should reveal a new summary of care document being generated for a patient upon completion of the visit. For each new CCDA generated, the audit trail should increase by 1.
    • Audit trail should reveal a successful connection by the patient to the Patient Portal.
    • Audit trail should review a successful view or download of the CCDA by the patient via the patient portal should increase by a count of 1.
  • Care Settings and Number of Clients Site to Test: We designed this measure to test general ambulatory sites that we support and target. We will test a minimum of three (3) client practice(s). This number covers a sufficient percentage of existing practices to provide a viable sample of users of the certified EHR.

Phase 4: CCDA’s recorded during the visit are shared automatically with a third-party data registry

Scenario 4.A - Upon completion of a visit, the clinical summary will be automatically downloaded and transmitted to a third-party registry that can receive CCDA files (170.315(b)(6) Data export)

  • Description: Upon completion of a visit, the clinical summary will be automatically downloaded and transmitted to a third-party registry that can receive CCDA files 
  • Associated Certification Criteria: 170.315(b)(6) Data export
  • Justification:  The use case will describe the scenario in which a provider practice completes a routine medical visit. The practice will have pre-configured the automated data export process to download newly generated CCDA’s on a nightly basis. The application should automatically download a copy of the CCDA for the patient that was recently seen to a designated folder on the practice’s hard drive.
  • Testing Method: Audit Trail Review, Third-Party Application Attestation
  • Expected Outcomes:
    • Audit trail should reveal a new CCDA being queued up for download. For each new CCDA generated, 1 new record should be added to the audit trail.
    • Audit trail should reveal the CCDA was successfully downloaded at the designated timeframe. Once the record is downloaded, the audit trail should update to reflect the fact that 1 new record was obtained.
    • Third-Party Attestation from the provider that the CCDA is stored as expected in the folder they had configured. A screenshot of the folder should reveal 1 new record downloaded for each visit.
  • Care Settings and Number of Clients Site to Test: We designed this measure to test general ambulatory sites that we support and target. We will test a minimum of three (3) client practice(s). This number covers a sufficient percentage of existing practices to provide a viable sample of users of the certified EHR.

Real-World Test Case: Third-Party Application Data Access

Measurement Description

As the medical industry evolves and medical data becomes more commercialized, third-party application developers or other EMR/PM companies will likely build tools that can be used by patients to securely access and manage key aspects of their health records.

This test case will cover the workflow in which a third-party developer chooses to build an application that would leverage AllMeds’ public APIs to securely retrieve data for a patient to display in their application. AllMeds’ public API documentation can be found on our website at https://isalushealthcare.com/wp-content/uploads/allmedscertification/allmeds-v12/Fhir_API_Doc_v12.pdf 

Scenario 5.A: Application access - patient selection (170.315(g)(7))

  • Description: Application access - patient selection 
  • Associated Certification Criteria: 170.315(g)(7)
  • Justification:  The use case will describe the scenario in which a third-party application developer builds an application to retrieve patient data from AllMeds Specialty EHR. The first step in utilizing the public API’s is to allow a patient to retrieve a patient-specific API token. Patients can access these patient-specific tokens securely via our patient portal. The third-party application will be required to receive and store the token provided by the patient along with the patient's first name, patient's last name, and patient's date of birth. Then, the application must call the public FIHR endpoint Patient to retrieve a Patient ID. This Patient ID token is used in all subsequent application access calls to retrieve patient-specific data. If any of the required elements do not match the data stored in AllMeds Specialty EHR, an appropriate error message will be returned.
  • Testing Method: Third-Party Application (Webservice Test Suite)
  • Expected Outcomes:
    • Third-Party Application (Web service Test Suite) is able to make a call to our production endpoint (https://dhitweb.allmeds.com:4435/fhir) utilizing the “Patient” FIHR resource method. When passing in a matching patient string a patient resource object should be returned. 0 errors should be returned when a valid call is executed.
  • Client Sites Tested:  At the time of this writing, no third-party application developers have chosen to utilize the public API endpoints provided. Therefore, real-world testing must be conducted by utilizing a web service testing suite to simulate making live API calls. We will utilize a third party web service test suite to follow the test plan and ensure appropriate responses and error messages are returned according to the API documentation provided on our website: https://isalushealthcare.com/wp-content/uploads/allmedscertification/allmeds-v12/Fhir_API_Doc_v12.pdf 

Scenario 5.B: Application access - data category request 170.315(g)(8)

  • Description: Application access - data category request
  • Associated Certification Criteria: 170.315(g)(8)
  • Justification:  The use case will describe the scenario in which a third-party application developer builds an application to retrieve patient data from AllMeds Specialty EHR. To complete this test case, the steps in Scenario 1.A must first be followed to obtain a Patient ID that will be used in this test case.  With a valid Patient ID in hand, an application should be able to make a call to the desired FIHR resource endpoint to retrieve one to many portions of a patient’s medical record as defined in the CCDA. If a valid request is made, the API should return the requested sections of the CCDA in JSON format (as described in the API documentation). If any of the required elements do not match the data stored in AllMeds Specialty EHR, an appropriate error message will be returned.
  • Testing Method: Third-Party Application (Webservice Test Suite)
  • Expected Outcomes:
    • Third-Party Application (Web service Test Suite) is able to make a call to our production endpoint (https://dhitweb.allmeds.com:4435/fhir) utilizing the patient ID and desired FIHR resource method. When passing in a valid request the API should return the requested FIHR resource object and all accompanying data. The response from the API call should match the API documentation.  0 errors should be returned when a valid call is executed.
    • Manual verification of the data elements in the patient chart should match the data returned in the API response. 0 errors should be found when comparing the data between the source system and the response.
  • Client Sites Tested:  At the time of this writing, no third-party application developers have chosen to utilize the public API endpoints provided. Therefore, real-world testing must be conducted by utilizing a web service testing suite to simulate making live API calls. We will utilize a third party web service test suite to follow the test plan and ensure appropriate responses and error messages are returned according to the API documentation provided on our website: https://isalushealthcare.com/wp-content/uploads/allmedscertification/allmeds-v12/Fhir_API_Doc_v12.pdf 

Scenario 5.C: Application access – all data request 170.315(g)(9)

  • Description: Application access – all data request
  • Associated Certification Criteria: 170.315(g)(9)
  • Justification:  The use case will describe the scenario in which a third-party application developer builds an application to retrieve patient data from AllMeds Specialty EHR. To complete this test case, the steps in Scenario 1.A must first be followed to obtain a valid patient ID that will be used in this test case.  With a successful patient ID in hand, an application should be able to make a call to the public FIHR Resource “DocumentReference”. This endpoint requires the patient ID, start date, end date, or a timeframe for the CCDA that the application would like to retrieve. If a valid call is made, the API should return the requested CCDA in XML format (as described in the API documentation). If any of the required elements do not match the data stored in AllMeds Specialty EHR, an appropriate error message will be returned.
  • Testing Method: Third-Party Application (Webservice Test Suite)
  • Expected Outcomes:
    • Third-Party Application (Web service Test Suite) is able to make a call to our production endpoint (https://dhitweb.allmeds.com:4435/fhir) utilizing the Document Reference FIHR Resource. When passing in a valid patient ID, the API should return the complete CCDA document for the patient. The response from the API call should match the API documentation. 0 errors should be returned when a valid call is executed.
    • Manual verification of the data elements in the patient chart should match the data returned in the API response. 0 errors should be found when comparing the data between the source system and the response.
  • Client Sites Tested:  At the time of this writing, no third-party application developers have chosen to utilize the public API endpoints provided. Therefore, real-world testing must be conducted by utilizing a web service testing suite to simulate making live API calls. We will utilize a third party web service test suite to follow the test plan and ensure appropriate responses and error messages are returned according to the API documentation provided on our website: https://isalushealthcare.com/wp-content/uploads/allmedscertification/allmeds-v12/Fhir_API_Doc_v12.pdf