Deepchecks Testing (and CI) Installation

pip install deepchecks -U --user

For installing the nlp / vision submodules or with conda:

• For NLP: Replace deepchecks with "deepchecks[nlp]",
  and optionally install alsodeepchecks[nlp-properties]
• For Computer Vision: Replace deepchecks with "deepchecks[vision]".
• For installing with conda, similarly use: conda install -c conda-forge deepchecks.

Check out the full installation instructions for deepchecks testing here.

Deepchecks Monitoring Installation

To use deepchecks for production monitoring, you can either use our SaaS service, or deploy a local instance in one line on Linux/MacOS (Windows is WIP!) with Docker.
Create a new directory for the installation files, open a terminal within that directory and run the following:

pip install deepchecks-installer
deepchecks-installer install-monitoring

This will automatically download the necessary dependencies, run the installation process
and then start the application locally.

The installation will take a few minutes. Then you can open the deployment url (default is http://localhost),
and start the system onboarding. Check out the full monitoring open source installation & quickstart.

Note that the open source product is built such that each deployment supports monitoring of
a single model.

Jump right into the respective quickstart docs:

• Tabular Quickstarts
• NLP Quickstarts
• Vision Quickstarts

to have it up and running on your data.

Inside the quickstarts, you’ll see how to create the relevant deepchecks object for holding your data and metadata
(Dataset, TextData or VisionData, corresponding to the data type), and run a Suite or Check.
The code snippet for running it will look something like the following, depending on the chosen Suite or Check.

from deepchecks.tabular.suites import model_evaluation
suite = model_evaluation()
suite_result = suite.run(train_dataset=train_dataset, test_dataset=test_dataset, model=model)
suite_result.save_as_html() # replace this with suite_result.show() or suite_result.show_in_window() to see results inline or in window
# or suite_result.results[0].value with the relevant check index to process the check result's values in python

The output will be a report that enables you to inspect the status and results of the chosen checks:

Jump right into the
open source monitoring quickstart docs
to have it up and running on your data.
You’ll then be able to see the checks results over time, set alerts, and interact
with the dynamic deepchecks UI that looks like this:

Deepchecks managed CI & Testing management is currently in closed preview.
Book a demo for more information about the offering.

For building and maintaining your own CI process while utilizing Deepchecks Testing for it,
check out our docs for Using Deepchecks in CI/CD.

• All of the Checks and the framework for customizing them are implemented inside
  the Deepchecks Testing Python package (this repo).
• Each check tests for a specific potential problem. Deepchecks has many pre-implemented checks for
  finding issues with the model’s performance (e.g. identifying weak segments),
  data distribution (e.g. detect drifts or leakages) and data integrity (e.g. find conflicting labels).
• Customizable: each check has many configurable parameters, and custom checks can easily be implemented.
• Can be run manually (during research) or triggered automatically (in CI processes or production monitoring)
• Check results can be consumed by:
  • Visual output report - Saving to HTML(result.save_to_html('output_report_name.html')) or viewing them in Jupyter (result.show()).
  • Processing with code - with python using the check result’s value attribute, or saving a
    JSON output
  • Deepchecks’ UI - for dynamic inspection and collaboration (of test results and production monitoring)
• Optional conditions can be added and customized, to automatically validate check results,
  with a a pass ✓, fail ✖ or warning ! status
• An ordered list of checks (with optional conditions) can be run together in a “Suite”
  (and the output is a concluding report of all checks that ran)