Questions? Give as a call at (800) 995-6997
Home
Zoobook Features
Clients
Disclosures
Contact Us
REAL WORLD TEST PLANS 2024
  • The measurements used for our Real World Testing Plan are listed below

    • Associated Certification Criteria
    • Measurement/Metric
    • Description
    • Justification
    • Expected Outcome
    • Test Method*
  • *Test Method
    Each Measurement/Metric uses a testing methodology. Here are the testing methodologies that will be used for this Test Plan:

    • Internal Testing – Members of the development team or organization will be users insofar as they have the ability and motivation to voice and communicate problems.
    • Survey – This method evaluates the compliance of the Health IT Module to the following certification criteria(s) by gathering end user feedbacks.
ASSOCIATED CERTIFICATION CRITERIA MEASUREMENT/METRIC DESCRIPTION TEST METHOD JUSTIFICATION EXPECTED OUTCOME
§ 170.315 (b)(1): Transitions of Care Transition of Care Functionality Provider identified users are able to create transitions of care, validate and display C-CDA, send and receive summaries from the Health IT module. Survey, Internal Testing This measure will survey end users to determine real world interoperability and usability of the Health IT module.
Internal testing will also be done to ensure expected outcomes are met. 		Positive end user feedback will provide a benchmark to determine real-world interoperability and usability of the Health IT Module:
The Health IT Module must be capable of receiving and sending message via SMTP and transmission should be encrypted by TLS
Must have the capability to receive one or more attachments such as C-CDA documents and XDM packages. Our EHR must be able to process and recognize XDM mime types
Must be capable of sending/receiving XDR message with Full (XDS) metadata (IHE XDR Profile for Limited Metadata )
Health IT Module must be able to validate the format correctness of C-CDA documents
Health IT Module must be able to render C-CDA Documents and XDM packages with support for styling via css stylesheets for human readability
Health IT Module must have the capability to create and send valid C-CDA documents via SMTP Edge Protocol under a TLS connection
Must ensures that the required Data Elements for different settings are included in generated C-CDA’s
§ 170.315 (b)(6): Data Export C-CDA Data Export Functionality Provider identified users are able to successfully export user-identified client C-CDA files for user-defined date/time ranges from the Health IT module. Survey, Internal Testing This measure will survey end users to determine real world interoperability and usability of the Health IT module.
Internal testing will also be done to ensure expected outcomes are met. 		Positive end user feedback will provide a benchmark to determine real-world interoperability and usability of the Health IT Module:
Provider identified users can create export summaries in real-time; based on a relative date and time or a specific date and time.
Provider identified users can set configuration options for data elements, date and time ranges, and locations when generating a C-CDA
Provider identified users can generate a set of C-CDAs for between one client and all clients, without generating each C-CDA individually
C-CDA generation is limited to those users who have the proper role or system credentials, as determined by the provider
Provider identified users not having proper system credentials cannot generate C-CDAs. Applies to:

  • Document creation configuration
  • Timeframe configuration
  • Location Configuration

Only authorized users can set the limits of users who can do the above.
C-CDAs may be exported to local or network storage, in addition to integrated transmission methods
§ 170.315 (c)(1): Clinical Quality Measures – Record and Export Clinical quality measures (CQM) functionality Provider identified users are able to record and export all of the data that would be necessary to calculate each CQM Survey, Internal Testing This measure will survey end users to determine real world interoperability and usability of the Health IT module. Internal testing will also be done to ensure expected outcomes are met. Positive end user feedback will provide a benchmark to determine real-world interoperability and usability of the Health IT Module:
The Health IT Module must be able to record all of the data that would be necessary to calculate each CQM.
Must be able to export a data file at any time the user chooses and without subsequent developer assistance to operate:

  • Formatted in accordance with the standard specified in §170.205(h)(2);
  • Ranging from one to multiple patients; and
  • That includes all of the data captured for each and every CQM to which technology was certifiaed under paragraph (c)(1)(i) of this section.
§ 170.315 (c)(2): Clinical quality measures import and calculate Clinical quality measures (CQM) functionality Provider identified users are able to import a data file and use it to calculate all of the data that would be necessary to each CQM Survey, Internal Testing This measure will survey end users to determine real world interoperability and usability of the Health IT module. Internal testing will also be done to ensure expected outcomes are met. Positive end user feedback will provide a benchmark to determine real-world interoperability and usability of the Health IT Module:
The user must be able to import a data file in accordance with the standard specified in §170.205(h)(2) for one or multiple patients and use such data to perform the capability specified in paragraph (c)(2)(ii) of this section. A user must be able to execute this capability at any time the user chooses and without subsequent developer assistance to operate
The Health IT Module can calculate each and every clinical quality measure for which it is presented for certification.
§ 170.315(c)(3) Clinical quality measures report Clinical quality measures (CQM) functionality Provider identified users are able to generate CQM data files for transmission based on appropriate QRDA guidelines. Survey, Internal Testing This measure will survey end users to determine real world interoperability and usability of the Health IT module. Internal testing will also be done to ensure expected outcomes are met. Positive end user feedback will provide a benchmark to determine real-world interoperability and usability of the Health IT Module:
The user must be able to electronically create a data file for transmission of clinical quality measurement data:

(i) In accordance with the applicable implementation specifications specified by the CMS implementation guides for Quality Reporting Document Architecture (QRDA), category I, for inpatient measures in § 170.205(h)(3) and CMS implementation guide for QRDA, category III for ambulatory measures in § 170.205 (k)(3); or

(ii) In accordance with the standards specified in § 170.205(h)(2) and § 170.205(k)(1) and (2) for the period until December 31, 2022.
§ 170.315 (g)(7) Application access – patient selection Application access functionality An API must be able to receive a request with sufficient data to uniquely identify a patient and return an ID or other token that can be used by an application to subsequently execute requests for that patient’s data Survey, Internal Testing This measure will survey end users to determine real world interoperability and usability of the Health IT module. Internal testing will also be done to ensure expected outcomes are met. Positive end user feedback will provide a benchmark to determine real-world interoperability and usability of the Health IT Module:
The Health IT Module will uniquely identify a patient and return an ID or other token that can be used by the application to subsequently execute requests for that patient’s data.
The Health IT Module must be able to respond to requests for patient data (using an ID or other token) for each of the individual categories listed in the Common Clinical Data Set (CCDS) and return the full set of data for that category, according to the required data standards in a computable format.
The Health IT Module must be able to respond to requests for patient data associated with a specific date as well as requests for patient data with a specific date range.
§ 170.315 (g)(9) Application access – all data request Application access functionality Upon receipt of an ID/Token, the API will return the requested data in a summary format following CCD documentation template guidelines, or provide the CCDS, and address assessment and plan of treatment, goals, health concerns, and unique device identifiers for any implantable devices. Survey, Internal Testing This measure will survey end users to determine real world interoperability and usability of the Health IT module. Internal testing will also be done to ensure expected outcomes are met. Positive end user feedback will provide a benchmark to determine real-world interoperability and usability of the Health IT Module:
The Health IT Module must be able to respond to requests for patient data (using an ID or other token) for all of the data categories specified in the United States Core Data for Interoperability Standard (USCDI) at one time in a summary record formatted according to the Consolidated CDA Release 2.1 Continuity of Care Document (CCD) template.
The Health IT Module must be able to respond to requests for patient data associated with a specific date as well as with a specific date range.
§ 170.315 (g)(10) Standardized API for patient and population services Application access functionality Provider identified users are able to utilize the standardized API for accessing single and multiple patients. Survey, Internal Testing This measure will survey end users to determine real world interoperability and usability of the Health IT module.
Internal testing will also be done to ensure expected outcomes are met. 		Positive end user feedback will provide a benchmark to determine real-world interoperability and usability of the Health IT Module:
The Health IT Module must be able to respond to requests for patient data (using an ID or other token) for all of the data categories specified in the United States Core Data for Interoperability Standard (USCDI) at one time in a summary record formatted according to the Consolidated CDA Release 2.1
Continuity of Care Document (CCD)
template.
The Health IT Module must be able to respond to requests for patient data associated with a specific date as well as with a specific date range.
§ 170.315 (h)(1) Direct Project Direct Project functionality Provider identified users are able to send and receive health information Survey, Internal Testing This measure will survey end users to determine real world interoperability and usability of the Health IT module. Internal testing will also be done to ensure expected outcomes are met. Positive end user feedback will provide a benchmark to determine real-world interoperability and usability of the Health IT Module:
The Health IT Module can electronically transmit (send and receive) health information to a 3rd party which must be formatted only as a “wrapped” message using the Applicability Statement for Secure Health Transport, Version 1.2.
The Health IT Module can electronically transmit (send and receive) health information to a 3rd party using Direct in accordance with the Implementation Guide (IG) for Delivery Notification in Direct, Version 1.0.
ASSOCIATED CERTIFICATION CRITERIA CARE SETTING JUSTIFICATION
§ 170.315 (b)(1): Transitions of Care Outpatient behavioral health Outpatient (ambulatory) behavioral health providers is able to create a C-CDA that also includes the reason for referral, and the referring or transitioning provider’s name and office contact information.
Residential behavioral health Residential behavioral health providers is able to create a C-CDA Discharge Summary Document that includes reason for referral; and referring or transitioning provider’s name and office contact information; and discharge instructions.
§ 170.315 (b)(6): Data Export Outpatient behavioral health Outpatient (ambulatory) behavioral health providers frequently provide C-CDA documentation to outside entities. Outpatient care C-CDA documentation requires reason for referral; and referring or transitioning provider’s name and office contact information.
Residential behavioral health Residential behavioral health providers frequently provide C-CDA documentation to outside entities. Residential CCDA documentation requires reason for referral; and referring or transitioning provider’s name and office contact information; and discharge instructions.
§ 170.315 (c)(1): Clinical Quality Measures – Record and Export
§ 170.315 (c)(2): Clinical Quality Measures – Import and Calculate
§ 170.315 (c)(3): Clinical Quality Measures – Report 		Outpatient behavioral health QRDA documentation is specific to outpatient behavioral health settings.
Residential behavioral health QRDA documentation is specific to inpatient behavioral health settings.
§ 170.315 (g)(7) Application access – patient selection
§ 170.315 (g)(9) Application access – all data request
§ 170.315 (g)(10) Standardized API for patient and population services 		Outpatient behavioral health Documentation requirements at the time of discharge are different from Residential Behavioral Health setting requirements.
Residential behavioral health Documentation requirements at the time of discharge are different from Outpatient Behavioral Health setting requirements.
§ 170.315 (h)(1) Direct Project Outpatient behavioral health While technically similar, operational processes differ from residential behavioral health programs.
Residential behavioral health While technically similar, operational processes differ from outpatient behavioral health programs.
REAL WORLD TEST PLAN RESULTS 2024

Developer Information

  • Developer Name/Organization: Zoobook Systems LLC
  • Product Name: Zoobook EHR
  • Version Number(s): 2.0
  • Certified Health IT Product List (CHPL) Product Number: 15.04.04.3008.Zoob.02.00.1.171231
  • Report Date: January 20, 2025
  • Real World Testing (RWT) Period: January 1, 2024 – December 31, 2024

1. Executive Summary

Zoobook Systems LLC conducted Real World Testing throughout 2024 to validate that our Zoobook EHR meets interoperability and functionality requirements in production settings. This report presents our 2024 Real World Testing results, covering:

  • Transitions of Care
  • Electronic Health Information (EHI) Export
  • Clinical Quality Measures (CQMs)
  • Application Access (API)
  • Direct Project

Data was collected from a live behavioral health care treatment environment. The results indicate strong performance across all measured criteria, with minimal corrective actions required.

2. Testing Methodology

2.1 Approach/Methods

  • Data Collection: We aggregated usage logs, transaction counts, and error data from live environments.
  • Analysis & Thresholds: Each measure had predefined success thresholds from the Real World Test Plan.
  • Site Involved: A live behavioral health care treatment environment.

2.2 Timeline

  • Testing Period: January – December 2024.
  • Quarterly Reviews: Data was validated and analyzed at the end of each quarter (Q1, Q2, Q3, Q4).
  • Final Compilation: Results finalized in January 2025.

3. Key Measures & Associated Certification Criteria

The table below maps each Real World Test category to relevant certification criteria and metrics.

Category Certification Criteria Metrics Measured Thresholds
1. Transitions of Care 170.315(b)(1)

- % of successful C-CDA (Continuity of Care Document) creations, sends, and receives

- % of successful XDM package processing

- % of readable C-CDA documents with CSS rendering

- Frequency of transmission errors

≥95% success rate

≥90% success for XDM 100% readability with CSS
2. EHI Export 170.315(b)(10)

- % of error-free EHI exports

- Average export completion time

- Role-based access control compliance

- Accuracy of date range configurations

- Batch export efficiency

- Transmission success across endpoints

≥98% success rate

≤2 min/export

≥99% transmission success
3. Clinical Quality Measures 170.315(c)(1)/(c)(2)/(c)(3)

- % of accurate CQM data recording

- % of QRDA-compliant exports

- Compliance of imported files with standards

- Accuracy of CQM calculations

- Success rate for ambulatory and inpatient reporting

≥99% data accuracy

≤1% export error state

100% compliance with CMS guidelines
4. Application Access (API) 170.315(g)(7)/(g)(9)/(g)(10)

- % of successful patient data retrieval (single and bulk)

- Average response time for data requests

- Accuracy of FHIR-based data retrieval

≥99% success for individual requests

≤5 seconds for bulk requests

≤1% error rate
5. Direct Project 170.315(h)(1)

- % of successful Direct messages sent and received

- % of accurate delivery notifications

- Average time for message transmission

≥98% messaging success rate

100% delivery notification compliance

≤3 seconds/message

4. Care Settings

Real World Testing was completed in a live behavioral health care treatment environment, comprised of 163 clients representing a variety of mental health and substance abuse conditions and severities. Treatment protocols involve a wide range of clinical workflows, ensuring Real World Testing reflects varied clinical scenarios.

5. Results & Analysis

5.1 Transitions of Care

  • Certification Criteria: § 170.315(b)(1)
  • Metrics Measured:
    • Percentage of successful C-CDA (Continuity of Care Document) creations, sends, and receives.
    • Frequency of transmission errors over the testing period.
    • Accuracy in handling XDM package and C-CDA attachments (e.g., MIME type recognition).
    • Readability and CSS rendering of C-CDA documents.
  • Methodology:
    • System performance logging to monitor successful vs. failed transmission attempts.
    • Automated testing for XDM attachments and MIME type accuracy.
    • Validation of CSS rendering and readability across sampled C-CDA documents.
  • Expected Outcomes:
    • ≥ 95% success rate for document exchange.
    • ≥ 90% success rate in processing XDM and C-CDA attachments.
    • 100% readability with CSS rendering, with issues affecting < 1% of documents.

5.2 Electronic Health Information (EHI) Export

  • Certification Criteria: § 170.315(b)(10)
  • Metrics Measured:
    • Success rate of error-free EHI exports (target ≥ 98%).
    • Average completion time for exports (target ≤ 2 minutes per export).
    • Role-based access control success for authorized vs. unauthorized users.
    • Accuracy in applying date ranges for exports (target ≥ 98%).
    • Batch export efficiency (target ≤ 1.5x the time for individual exports).
    • Transmission success rate across storage endpoints (target ≥ 99%).
  • Methodology:
    • System log analysis for tracking successful vs. failed exports.
    • Automated timing and access control testing.
    • Validation of exported data integrity and batch processing efficiency.
  • Expected Outcomes:
    • ≥ 98% success rate for error-free exports.
    • ≤ 2 minutes per individual export; ≤ 1.5x for batch exports.
    • ≥ 99% transmission success across storage endpoints.

5.3 Clinical Quality Measures (CQMs)

  • Certification Criteria: § 170.315(c)(1), (c)(2), (c)(3)
  • Metrics Measured:
    • Accuracy rate of recorded CQM data (target ≥ 99%).
    • Success rate of data exports in QRDA format (≤ 1% error rate).
    • Compliance of imported CQM files with §170.205(h)(2) standards.
    • Accuracy of CQM calculations validated against expected outcomes.
    • Reporting success rate for inpatient (Category 1) and ambulatory (Category III) reports (target 100%).
  • Methodology:
    • Automated data integrity checks and file import validation.
    • Calculation consistency checks for sample cases.
    • CMS guideline compliance for QRDA reporting.
  • Expected Outcomes:
    • ≥ 99% data accuracy.
    • 100% compliance with CMS guidelines for reporting formats.
    • ≥ 99% transmission success across storage endpoints.

5.4 Application Access (API)

  • Certification Criteria: § 170.315(g)(7), (g)(9), (g)(10)
  • Metrics Measured:
    • Success rate for patient selection and retrieval (target ≥ 99%).
    • Average response time for individual data requests (≤ 3 seconds).
    • Accuracy of FHIR-based requests for all USCDI data categories (≤ 1% error rate).
    • Success rate for bulk data requests (≥ 98%).
    • Response time for bulk requests (≤ 5 seconds).
  • Methodology:
    • Performance monitoring for patient data selection and retrieval.
    • API load testing for single and bulk requests.
    • Consistency checks for FHIR-based data completeness and USCDI compliance.
  • Expected Outcomes:
    • ≥ 99% success rate for individual data requests.
    • ≤ 5 seconds for bulk requests.

5.5 Direct Project

  • Certification Criteria: § 170.315(h)(1)
  • Metrics Measured:
    • Success rate for Direct messages sent and received (target ≥ 98%).
    • Average time for message transmission (≤ 3 seconds per transaction).
    • Accuracy rate for delivery notifications (target 100%).
    • Rate of confirmed receipt notifications (target ≥ 99%).
  • Methodology:
    • Automated monitoring of message transmission success and failures.
    • Performance logging for transaction timing.
    • Validation of delivery notifications against standards.
  • Expected Outcomes:
    • ≥ 98% success rate for Direct messaging.
    • Delivery notifications achieve ≥ 99% confirmed receipt rate.

6. Standards Updates

  • Standards & Versions:
    • FHIR R4 for Application Access (API)
    • CDA R2 (for C-CDA document exchange in Transitions of Care)
    • DirectTrust/Direct Project standards and certificates

No significant changes to these standards were introduced during 2024 that impacted real world usage. Zoobook EHR remained aligned with the existing versions and updated measure specs as needed.

7. Key Achievements & Challenges

Achievements

  1. High Success Rates Across All Categories: Each tested criterion met or exceeded thresholds.
  2. Reduced Error Rates for CQMs: Streamlined measure library updates significantly decreased mismatch errors.
  3. Improved User Training & Documentation: Proactive user education reduced configuration errors, especially for EHI exports.

Challenges & Mitigations

  1. Occasional Network Interruptions: We introduced retry mechanisms for transitions of care and exports in Q3.
  2. API Misuse by Third Parties: We introduced retr Improved developer portal documentation and added additional logging.
  3. Certificate Management for Direct: Automated renewal reminders reduced downtime and message failures.

8. Future Steps & Continuous Improvement

  1. Enhance FHIR Capabilities: Investigate support for additional FHIR resources (Scheduling, Questionnaire) to expand interoperability.
  2. Refine Direct Messaging Tools: Provide a real-time certificate status dashboard to reduce message failures.
  3. Expand CQM Library: Proactively map measure specifications as soon as CMS releases annual updates to avoid early-year mismatches.