How to Use CleanVision#

What is CleanVision?#

CleanVision is built to automatically detects various issues in image datasets, such as images that are: (near) duplicates, blurry, over/under-exposed, etc. This data-centric AI package is designed as a quick first step for any computer vision project to find problems in your dataset, which you may want to address before applying machine learning.

	Issue Type	Description	Issue Key
1	Light	Images that are too bright/washed out in the dataset	light
2	Dark	Images that are irregularly dark	dark
3	Odd Aspect Ratio	Images with an unusual aspect ratio (i.e. overly skinny/wide)	odd_aspect_ratio
4	Exact Duplicates	Images that are exact duplicates of each other	exact_duplicates
5	Near Duplicates	Images that are almost visually identical to each other (e.g. same image with different filters)	near_duplicates
6	Blurry	Images that are blurry or out of focus	blurry
7	Grayscale	Images that are grayscale (lacking color)	grayscale
8	Low Information	Images that lack much information (e.g. a completely black image with a few white dots)	low_information
9	Odd Size	Images that are abnormally large or small compared to the rest of the dataset	odd_size

The Issue Key column specifies the name for each type of issue in CleanVision code. See our examples which use these keys to detect only particular issue types and specify nondefault parameter settings to use when checking for certain issues.

This notebook uses an example dataset, that you can download using these commands.

wget - nc 'https://cleanlab-public.s3.amazonaws.com/CleanVision/image_files.zip'
unzip -q image_files.zip

Examples#

1. Using CleanVision to detect issues in your dataset#

[3]:

from cleanvision import Imagelab

# Path to your dataset, you can specify your own dataset path
dataset_path = "./image_files/"

# Initialize imagelab with your dataset
imagelab = Imagelab(data_path=dataset_path)

# Find issues
imagelab.find_issues()

Reading images from /home/docs/checkouts/readthedocs.org/user_builds/cleanvision/checkouts/latest/docs/source/tutorials/image_files
Checking for dark, light, odd_aspect_ratio, low_information, exact_duplicates, near_duplicates, blurry, grayscale, odd_size images ...

Issue checks completed. 205 issues found in the dataset. To see a detailed report of issues found, use imagelab.report().

The report() method helps you quickly understand the major issues detected in the dataset. It reports the number of images in the dataset that exhibit each type of issue, and shows example images corresponding to the most severe instances of each issue.

[4]:

imagelab.report()

Issues found in images in order of severity in the dataset

|    | issue_type       |   num_images |
|---:|:-----------------|-------------:|
|  0 | odd_size         |          109 |
|  1 | near_duplicates  |           20 |
|  2 | grayscale        |           20 |
|  3 | exact_duplicates |           19 |
|  4 | odd_aspect_ratio |           11 |
|  5 | dark             |           10 |
|  6 | blurry           |            6 |
|  7 | low_information  |            5 |
|  8 | light            |            5 |

--------------------- odd_size images ----------------------

Number of examples with this issue: 109
Examples representing most severe instances of this issue:

------------------ near_duplicates images ------------------

Number of examples with this issue: 20
Examples representing most severe instances of this issue:

Set: 0

Set: 1

Set: 2

Set: 3

--------------------- grayscale images ---------------------

Number of examples with this issue: 20
Examples representing most severe instances of this issue:

----------------- exact_duplicates images ------------------

Number of examples with this issue: 19
Examples representing most severe instances of this issue:

Set: 0

Set: 1

Set: 2

Set: 3

----------------- odd_aspect_ratio images ------------------

Number of examples with this issue: 11
Examples representing most severe instances of this issue:

----------------------- dark images ------------------------

Number of examples with this issue: 10
Examples representing most severe instances of this issue:

---------------------- blurry images -----------------------

Number of examples with this issue: 6
Examples representing most severe instances of this issue:

------------------ low_information images ------------------

Number of examples with this issue: 5
Examples representing most severe instances of this issue:

----------------------- light images -----------------------

Number of examples with this issue: 5
Examples representing most severe instances of this issue:

The main way to interface with your data is via the Imagelab class. This class can be used to understand the issues in your dataset at a high level (global overview) and low level (issues and quality scores for each image) as well as additional information about the dataset. It has three main attributes:

Imagelab.issue_summary
Imagelab.issues
Imagelab.info

imagelab.issue_summary#

This is a Dataframe containing a comprehensive summary of all detected issue types within your dataset, along with their respective prevalence levels. Each row in this summary includes the following information:

issue_type: The name of the detected issue.

num_images: The number of images exhibiting the identified issue within the dataset.

[5]:

imagelab.issue_summary

[5]:

	issue_type	num_images
0	odd_size	109
1	near_duplicates	20
2	grayscale	20
3	exact_duplicates	19
4	odd_aspect_ratio	11
5	dark	10
6	blurry	6
7	low_information	5
8	light	5

imagelab.issues#

DataFrame assessing each image in your dataset, reporting which issues each image exhibits and a quality score for each type of issue.

[6]:

imagelab.issues.head()

[6]:

	odd_size_score	is_odd_size_issue	odd_aspect_ratio_score	is_odd_aspect_ratio_issue	low_information_score	is_low_information_issue	light_score	is_light_issue	grayscale_score	is_grayscale_issue	dark_score	is_dark_issue	blurry_score	is_blurry_issue	exact_duplicates_score	is_exact_duplicates_issue	near_duplicates_score	is_near_duplicates_issue
/home/docs/checkouts/readthedocs.org/user_builds/cleanvision/checkouts/latest/docs/source/tutorials/image_files/image_0.png	1.0	False	1.0	False	0.806332	False	0.925490	False	1	False	1.000000	False	0.980373	False	1.0	False	1.0	False
/home/docs/checkouts/readthedocs.org/user_builds/cleanvision/checkouts/latest/docs/source/tutorials/image_files/image_1.png	1.0	False	1.0	False	0.923116	False	0.906609	False	1	False	0.990676	False	0.472314	False	1.0	False	1.0	False
/home/docs/checkouts/readthedocs.org/user_builds/cleanvision/checkouts/latest/docs/source/tutorials/image_files/image_10.png	1.0	False	1.0	False	0.875129	False	0.995127	False	1	False	0.795937	False	0.470706	False	1.0	False	1.0	False
/home/docs/checkouts/readthedocs.org/user_builds/cleanvision/checkouts/latest/docs/source/tutorials/image_files/image_100.png	1.0	False	1.0	False	0.916140	False	0.889762	False	1	False	0.827587	False	0.441195	False	1.0	False	1.0	False
/home/docs/checkouts/readthedocs.org/user_builds/cleanvision/checkouts/latest/docs/source/tutorials/image_files/image_101.png	1.0	False	1.0	False	0.779338	False	0.960784	False	0	True	0.992157	False	0.507767	False	1.0	False	1.0	False

There is a Boolean column for each issue type, showing whether each image exhibits that type of issue or not. For example, the rows where the is_dark_issue column contains True, those rows correspond to images that appear too dark.

For the dark issue type (and more generally for other types of issues), there is a numeric column dark_score, which assesses how severe this issue is in each image. These quality scores lie between 0 and 1, where lower values indicate more severe instances of the issue (images which are darker in this example).

One use-case for imagelab.issues is to filter out all images exhibiting one particular type of issue and rank them by their quality score. Here’s how to get all blurry images ranked by their blurry_score, note lower scores indicate higher severity:

[7]:

blurry_images = imagelab.issues[imagelab.issues["is_blurry_issue"] == True].sort_values(
    by=["blurry_score"]
)
blurry_image_files = blurry_images.index.tolist()

Visualize the blurry images

[8]:

imagelab.visualize(image_files=blurry_image_files[:4])

A shorter way to accomplish the above task is to specify an issue type in imagelab.visualize(). This will show images ordered by the severity of this issue within them.

[9]:

imagelab.visualize(issue_types=["blurry"])

imagelab.info#

This is a nested dictionary containing statistics about the images and other miscellaneous information stored while checking for issues in the dataset. Beware: this dictionary may be large and poorly organized (it is only intended for advanced users).

Possible keys in this dict are statistics and a key corresponding to each issue type

[10]:

imagelab.info.keys()

[10]:

dict_keys(['statistics', 'dark', 'light', 'odd_aspect_ratio', 'low_information', 'blurry', 'grayscale', 'odd_size', 'exact_duplicates', 'near_duplicates'])

imagelab.info['statistics'] is also a dict containing statistics calculated on images while checking for issues in the dataset.

[11]:

imagelab.info["statistics"].keys()

[11]:

dict_keys(['brightness', 'aspect_ratio', 'entropy', 'blurriness', 'color_space', 'size'])

You can see size statistics for the dataset below. Here we observe, both the 25th and 75th percentile are 256 for the dataset, hence images that are further away from this range are detected as oddly sized.

[12]:

imagelab.info["statistics"]["size"]

[12]:

count     607.000000
mean      280.830152
std       215.001908
min        32.000000
25%       256.000000
50%       256.000000
75%       256.000000
max      4666.050578
Name: size, dtype: float64

Duplicate sets#

imagelab.info can also be used to retrieve which images are near or exact duplicates of each other.

issue.summary shows the number of exact duplicate images but does not show how many such sets of duplicates images exist in the dataset. To see the number of exact duplicate sets, you can use imagelab.info

[13]:

imagelab.info["exact_duplicates"]["num_sets"]

[13]:

You can also get exactly which images are there in each (exact/near) duplicated set using imagelab.info.

[14]:

imagelab.info["exact_duplicates"]["sets"]

[14]:

[['/home/docs/checkouts/readthedocs.org/user_builds/cleanvision/checkouts/latest/docs/source/tutorials/image_files/image_142.png',
  '/home/docs/checkouts/readthedocs.org/user_builds/cleanvision/checkouts/latest/docs/source/tutorials/image_files/image_236.png'],
 ['/home/docs/checkouts/readthedocs.org/user_builds/cleanvision/checkouts/latest/docs/source/tutorials/image_files/image_170.png',
  '/home/docs/checkouts/readthedocs.org/user_builds/cleanvision/checkouts/latest/docs/source/tutorials/image_files/image_299.png'],
 ['/home/docs/checkouts/readthedocs.org/user_builds/cleanvision/checkouts/latest/docs/source/tutorials/image_files/image_190.png',
  '/home/docs/checkouts/readthedocs.org/user_builds/cleanvision/checkouts/latest/docs/source/tutorials/image_files/image_197.png'],
 ['/home/docs/checkouts/readthedocs.org/user_builds/cleanvision/checkouts/latest/docs/source/tutorials/image_files/image_288.png',
  '/home/docs/checkouts/readthedocs.org/user_builds/cleanvision/checkouts/latest/docs/source/tutorials/image_files/image_289.png'],
 ['/home/docs/checkouts/readthedocs.org/user_builds/cleanvision/checkouts/latest/docs/source/tutorials/image_files/image_292.png',
  '/home/docs/checkouts/readthedocs.org/user_builds/cleanvision/checkouts/latest/docs/source/tutorials/image_files/image_348.png',
  '/home/docs/checkouts/readthedocs.org/user_builds/cleanvision/checkouts/latest/docs/source/tutorials/image_files/image_491.png'],
 ['/home/docs/checkouts/readthedocs.org/user_builds/cleanvision/checkouts/latest/docs/source/tutorials/image_files/image_30.png',
  '/home/docs/checkouts/readthedocs.org/user_builds/cleanvision/checkouts/latest/docs/source/tutorials/image_files/image_55.png'],
 ['/home/docs/checkouts/readthedocs.org/user_builds/cleanvision/checkouts/latest/docs/source/tutorials/image_files/image_351.png',
  '/home/docs/checkouts/readthedocs.org/user_builds/cleanvision/checkouts/latest/docs/source/tutorials/image_files/image_372.png'],
 ['/home/docs/checkouts/readthedocs.org/user_builds/cleanvision/checkouts/latest/docs/source/tutorials/image_files/image_379.png',
  '/home/docs/checkouts/readthedocs.org/user_builds/cleanvision/checkouts/latest/docs/source/tutorials/image_files/image_577.png'],
 ['/home/docs/checkouts/readthedocs.org/user_builds/cleanvision/checkouts/latest/docs/source/tutorials/image_files/image_548.png',
  '/home/docs/checkouts/readthedocs.org/user_builds/cleanvision/checkouts/latest/docs/source/tutorials/image_files/image_7.png']]

The rest of this notebook demonstrates more advanced/customized workflows you can do with CleanVision.

2. Using CleanVision to detect specific issues#

It might be the case that only a few issue types are relevant for your dataset and you don’t want to run it through all checks to save time. You can do so by specifying issue_types as an argument.

issue_types is a dict, where keys are the issue types that you want to detect and values are dict which contains hyperparameters. This example uses default hyperparameters, in which case you can leave the hyperparameter dict empty. To find keys for issue types check the above table that lists all issue types supported by CleanVision.

[15]:

# Initialize imagelab with your dataset
imagelab = Imagelab(data_path=dataset_path)

# specify issue types to detect
issue_types = {"dark": {}}

# Find issues
imagelab.find_issues(issue_types)

# Show a report of the issues found
imagelab.report()

Reading images from /home/docs/checkouts/readthedocs.org/user_builds/cleanvision/checkouts/latest/docs/source/tutorials/image_files

Checking for dark images ...

Issue checks completed. 10 issues found in the dataset. To see a detailed report of issues found, use imagelab.report().
Issues found in images in order of severity in the dataset

|    | issue_type   |   num_images |
|---:|:-------------|-------------:|
|  0 | dark         |           10 |

----------------------- dark images ------------------------

Number of examples with this issue: 10
Examples representing most severe instances of this issue:

3. Check for additional types of issues using the same instance#

Suppose you also want to check for blurry images after having already detected dark images in the dataset. You can use the same Imagelab instance to incrementally check for another type of issue like blurry images.

[16]:

issue_types = {"blurry": {}}

imagelab.find_issues(issue_types)

imagelab.report()

Checking for blurry images ...

Issue checks completed. 16 issues found in the dataset. To see a detailed report of issues found, use imagelab.report().
Issues found in images in order of severity in the dataset

|    | issue_type   |   num_images |
|---:|:-------------|-------------:|
|  0 | dark         |           10 |
|  1 | blurry       |            6 |

----------------------- dark images ------------------------

Number of examples with this issue: 10
Examples representing most severe instances of this issue:

---------------------- blurry images -----------------------

Number of examples with this issue: 6
Examples representing most severe instances of this issue:

4. Save and load#

CleanVision also has a save and load functionality that you can use to save the results and load them at a later point in time to see results or run more checks.

For saving, specify force=True to overwrite existing files.

[17]:

save_path = "./results"
imagelab.save(save_path)

Saved Imagelab to folder: ./results
The data path and dataset must be not be changed to maintain consistent state when loading this Imagelab

For loading a saved instance, specify dataset_path to help check for any inconsistencies between dataset paths in the previous and current run.

[18]:

imagelab = Imagelab.load(save_path, dataset_path)

Successfully loaded Imagelab

5. Check for an issue with a different threshold#

You can use the loaded imagelab instance to check for an issue type with a custom hyperparameter. Here is a table of hyperparameters that each issue type supports and their permissible values.

threshold- All images with scores below this threshold will be flagged as an issue.

hash_size - This controls how much detail about an image we want to keep for getting perceptual hash. Higher sizes imply more detail.

hash_type - Type of perceptual hash to use. Currently whash, phash, ahash, dhash and chash are the supported hash types. Check here for more details on these hash types.

	Issue Key	Hyperparameters
1	light	threshold (between 0 and 1)
2	dark	threshold (between 0 and 1)
3	odd_aspect_ratio	threshold (between 0 and 1)
4	exact_duplicates	N/A
5	near_duplicates	hash_size (int, only power of 2 for whash), hash_types (whash, phash, ahash, dhash, chash)
6	blurry	threshold (between 0 and 1)
7	grayscale	threshold (between 0 and 1)
8	low_information	threshold (between 0 and 1)

[19]:

issue_types = {"dark": {"threshold": 0.2}}
imagelab.find_issues(issue_types)

imagelab.report()

Checking for dark images ...
Issue checks completed. 14 issues found in the dataset. To see a detailed report of issues found, use imagelab.report().
Issues found in images in order of severity in the dataset

|    | issue_type   |   num_images |
|---:|:-------------|-------------:|
|  0 | dark         |            8 |
|  1 | blurry       |            6 |

----------------------- dark images ------------------------

Number of examples with this issue: 8
Examples representing most severe instances of this issue:

---------------------- blurry images -----------------------

Number of examples with this issue: 6
Examples representing most severe instances of this issue:

Note the number of images with dark issue has reduced from the previous run.

6. Run CleanVision for default issue types, but override hyperparameters for one or more issues#

[20]:

imagelab = Imagelab(data_path=dataset_path)

# Check for all default issue types
imagelab.find_issues()

# Specify an issue with custom hyperparameters
issue_types = {"odd_aspect_ratio": {"threshold": 0.2}}

# Run find issues again with specified issue types
imagelab.find_issues(issue_types)


# Pass list of issue_types to imagelab.report() to report only those issue_types
imagelab.report(["odd_aspect_ratio", "low_information"])

Reading images from /home/docs/checkouts/readthedocs.org/user_builds/cleanvision/checkouts/latest/docs/source/tutorials/image_files
Checking for dark, light, odd_aspect_ratio, low_information, exact_duplicates, near_duplicates, blurry, grayscale, odd_size images ...

Issue checks completed. 205 issues found in the dataset. To see a detailed report of issues found, use imagelab.report().
Checking for odd_aspect_ratio images ...
Issue checks completed. 194 issues found in the dataset. To see a detailed report of issues found, use imagelab.report().
Issues found in images in order of severity in the dataset

|    | issue_type       |   num_images |
|---:|:-----------------|-------------:|
|  6 | low_information  |            5 |
|  8 | odd_aspect_ratio |            0 |

------------------ low_information images ------------------

Number of examples with this issue: 5
Examples representing most severe instances of this issue:

7. Customize report#

Report can also be customized in various ways to help with the analysis. For example, you can change the num_images to show for each issue type

[21]:

# Change verbosity
imagelab.report(issue_types=["dark"], num_images=10)

Issues found in images in order of severity in the dataset

|    | issue_type   |   num_images |
|---:|:-------------|-------------:|
|  4 | dark         |           10 |

----------------------- dark images ------------------------

Number of examples with this issue: 10
Examples representing most severe instances of this issue:

You may want to exclude issues from your report which are prevalent in say more than 50% of the dataset and are not real issues but just how the dataset is, for example dark images in an astronomy dataset may not be an issue. You can use the max_prevalence parameter in report to exclude such issues. In this example all issues present in more than 3% of the dataset are excluded.

[22]:

imagelab.report(max_prevalence=0.03)

Removing odd_size from potential issues in the dataset as it exceeds max_prevalence=0.03
Removing near_duplicates from potential issues in the dataset as it exceeds max_prevalence=0.03
Removing grayscale from potential issues in the dataset as it exceeds max_prevalence=0.03
Removing exact_duplicates from potential issues in the dataset as it exceeds max_prevalence=0.03
Issues found in images in order of severity in the dataset

|    | issue_type       |   num_images |
|---:|:-----------------|-------------:|
|  4 | dark             |           10 |
|  5 | blurry           |            6 |
|  6 | low_information  |            5 |
|  7 | light            |            5 |
|  8 | odd_aspect_ratio |            0 |

----------------------- dark images ------------------------

Number of examples with this issue: 10
Examples representing most severe instances of this issue:

---------------------- blurry images -----------------------

Number of examples with this issue: 6
Examples representing most severe instances of this issue:

------------------ low_information images ------------------

Number of examples with this issue: 5
Examples representing most severe instances of this issue:

----------------------- light images -----------------------

Number of examples with this issue: 5
Examples representing most severe instances of this issue:

8. Visualize specific issues#

Imagelab provides imagelab.visualize that you can use to see examples of specific issues in your dataset.

num_images and cell_size are optional arguments, that you can use to control number of examples of each issue type and size of each image in the grid respectively.

[23]:

issue_types = ["grayscale"]
imagelab.visualize(issue_types=issue_types, num_images=3, cell_size=(3, 3))

Advanced: Create your own issue type#

You can also create a custom issue type by extending the base class IssueManager. CleanVision can then detect your custom issue along with other pre-defined issues in any image dataset! Here’s an example of a custom issue manager, which can also be found here

[24]:

from typing import Any, Dict, List, Optional

import numpy as np
import pandas as pd
from PIL import Image
from tqdm.auto import tqdm

from cleanvision.dataset.base_dataset import Dataset
from cleanvision.issue_managers import register_issue_manager
from cleanvision.utils.base_issue_manager import IssueManager
from cleanvision.utils.utils import get_is_issue_colname, get_score_colname

ISSUE_NAME = "custom"


@register_issue_manager(ISSUE_NAME)
class CustomIssueManager(IssueManager):
    """
    Example class showing how you can self-define a custom type of issue that
    CleanVision can simultaneously check your data for alongside its built-in issue types.
    """

    issue_name: str = ISSUE_NAME
    visualization: str = "individual_images"

    def __init__(self) -> None:
        super().__init__()
        self.params = self.get_default_params()

    def get_default_params(self) -> Dict[str, Any]:
        return {"threshold": 0.4}

    def update_params(self, params: Dict[str, Any]) -> None:
        self.params = self.get_default_params()
        non_none_params = {k: v for k, v in params.items() if v is not None}
        self.params = {**self.params, **non_none_params}

    @staticmethod
    def calculate_mean_pixel_value(image: Image.Image) -> float:
        gray_image = image.convert("L")
        return np.mean(np.array(gray_image))

    def get_scores(self, raw_scores: List[float]) -> "np.ndarray[Any, Any]":
        scores = np.array(raw_scores)
        return scores / 255.0

    def mark_issue(self, scores: pd.Series, threshold: float) -> pd.Series:
        return scores < threshold

    def update_summary(self, summary_dict: Dict[str, Any]) -> None:
        self.summary = pd.DataFrame({"issue_type": [self.issue_name]})
        for column_name, value in summary_dict.items():
            self.summary[column_name] = [value]

    def find_issues(
        self,
        *,
        params: Optional[Dict[str, Any]] = None,
        dataset: Optional[Dataset] = None,
        imagelab_info: Optional[Dict[str, Any]] = None,
        **kwargs: Any,
    ) -> None:
        super().find_issues(**kwargs)
        assert params is not None
        assert imagelab_info is not None
        assert dataset is not None

        self.update_params(params)

        raw_scores = []
        for idx in tqdm(dataset.index):
            image = dataset[idx]
            raw_scores.append(self.calculate_mean_pixel_value(image))

        score_colname = get_score_colname(self.issue_name)
        is_issue_colname = get_is_issue_colname(self.issue_name)

        scores = pd.DataFrame(index=dataset.index)
        scores[score_colname] = self.get_scores(raw_scores)

        is_issue = pd.DataFrame(index=dataset.index)
        is_issue[is_issue_colname] = self.mark_issue(
            scores[score_colname], self.params["threshold"]
        )

        self.issues = pd.DataFrame(index=dataset.index)
        self.issues = self.issues.join(scores)
        self.issues = self.issues.join(is_issue)

        self.info[self.issue_name] = {"PixelValue": raw_scores}
        summary_dict = self._compute_summary(
            self.issues[get_is_issue_colname(self.issue_name)]
        )

        self.update_summary(summary_dict)

9. Run CleanVision with a custom issue#

[25]:

imagelab = Imagelab(data_path=dataset_path)

issue_name = CustomIssueManager.issue_name


# To ensure your issue manager is registered, check list of possible issue types
# issue_name should be present in this list
imagelab.list_possible_issue_types()

Reading images from /home/docs/checkouts/readthedocs.org/user_builds/cleanvision/checkouts/latest/docs/source/tutorials/image_files

[25]:

['odd_size',
 'low_information',
 'light',
 'dark',
 'odd_aspect_ratio',
 'near_duplicates',
 'grayscale',
 'custom',
 'blurry',
 'exact_duplicates']

[26]:

issue_types = {issue_name: {}}
imagelab.find_issues(issue_types)

Checking for custom images ...

Issue checks completed. 211 issues found in the dataset. To see a detailed report of issues found, use imagelab.report().

[27]:

imagelab.report()

Issues found in images in order of severity in the dataset

|    | issue_type   |   num_images |
|---:|:-------------|-------------:|
|  0 | custom       |          211 |

---------------------- custom images -----------------------

Number of examples with this issue: 211
Examples representing most severe instances of this issue:

Beyond the collection of image files demonstrated here, you can alternatively run CleanVision on: Hugging Face datasets, torchvision datasets, as well as files in cloud storage buckets like S3, GCS, or Azure.