{"id":382955,"date":"2024-06-29T04:36:15","date_gmt":"2024-06-29T04:36:15","guid":{"rendered":"http:\/\/savepearlharbor.com\/?p=382955"},"modified":"-0001-11-30T00:00:00","modified_gmt":"-0001-11-29T21:00:00","slug":"","status":"publish","type":"post","link":"https:\/\/savepearlharbor.com\/?p=382955","title":{"rendered":"CLIP from OpenAI: what is it and how you can try it out yourself<\/span>"},"content":{"rendered":"
<\/div>\n
\n
\n
\n
\n

Neural networks (NN) and computer vision\u00a0models\u00a0in particular\u00a0are\u00a0known to perform well in specific tasks, but often fail to generalize to tasks they have not been trained on. A model that performs well on a food data may perform poorly on satellite images.\u00a0<\/p>\n

A new model from\u00a0OpenAI\u00a0named CLIP<\/a> claims to close this gap by a large\u00a0margin.\u00a0The\u00a0paper\u00a0Open AI wrote presenting CLIP\u00a0demonstrates how the model may be used on a\u00a0various classification datasets\u00a0in a zero-shot manner.\u00a0<\/p>\n

In this article, I will explain the key ideas of the\u00a0model\u00a0they proposed\u00a0and show you the code to\u00a0use\u00a0it.\u00a0<\/p>\n

Intuition<\/h2>\n

In a typical classification scenario, one has a set of examples connected to a set of pre-defined categories.\u00a0In such a set, the number of categories is fixed. If you train a model to distinguish between cats and dogs and then later decide to add a new class \u201cbear\u201d, then you will have to add example images with bears and train a new network!<\/p>\n

However, if one were to\u00a0train\u00a0a network that connects an image to an arbitrary text, then you can utilize it with new classes simply by providing text description of that class.\u00a0For this to work successfully, the network must learn good visual representations and good connections between visual\u00a0cues and text.<\/p>\n

How CLIP works\u00a0<\/h2>\n

First,\u00a0let us consider our problem scope. In order to connect images with text we need a dataset of image-text pairs.\u00a0CLIP authors report that they assembled a dataset of 400 million (image, text) pairs from the Internet.\u00a0The model will take\u00a0an\u00a0image as an input and predict text as an output.<\/p>\n

<\/figcaption><\/figure>\n

There are\u00a0different ways of representing text for prediction as shown on the figure below:<\/p>\n

<\/figcaption><\/figure>\n

One can predict text with the correct word order,\u00a0i.e.\u00a0the classifier must output this is a photo of a cat<\/code>.\u00a0Or one can predict a label based on bag of words,\u00a0i.e.\u00a0the order of words is not important and if classifier predicts photo<\/code>, cat<\/code>, then it is correct.\u00a0OpenAI\u00a0suggests a further improvement upon the bag of words method and shows that\u00a0CLIP is 4x more efficient in zero-shot ImageNet accuracy compared to previous methods.\u00a0<\/p>\n

CLIP achieves this by reframing the problem and using the contrastive pre-training. Instead of predicting label text, CLIP is training on predicting how likely this image is to correspond to that text.<\/p>\n

<\/figcaption><\/figure>\n

Input images and texts are\u00a0encoded,\u00a0and their vector representations are used to build a\u00a0similarity\u00a0matrix\u00a0(I*T is an inner product). Now, we know\u00a0(during training)\u00a0that the values on\u00a0the\u00a0diagonal represent\u00a0correct classifications, so their similarity must be higher than those in the same row\/column.\u00a0This approach contrasts what we know go together (diagonal values) to what we know doesn\u2019t go together (non-diagonal values).\u00a0You can see that each row is a classification task: given an input image I,\u00a0predict\u00a0the\u00a0text. Similarly, each column is a classification\u00a0task:\u00a0given an input text T, predict the image.\u00a0During training,\u00a0OpenAI\u00a0used a very large\u00a0size of mini-batches\u00a032768\u00a0(N on the figure above).<\/p>\n

During inference one takes a set of labels, creates texts based on labels and runs these texts through the text encoder. Text embeddings are later\u00a0matched to image representation.\u00a0<\/p>\n

<\/figcaption><\/figure>\n

Classic classification training cares only about the predefined labels.\u00a0If it is successful in findings dogs, then it doesn\u2019t care if it is a photo or a sketch of a dog or a specific breed.\u00a0Whereas\u00a0CLIP training coupled with a large\u00a0dataset\u00a0makes the network learn various aspects of images and point attention to details.<\/p>\n

One detail that is worth mentioning is that CLIP is sensitive to\u00a0words used for image descriptions. Texts \u201ca photo of a\u00a0bird\u201d, \u201ca photo\u00a0of a bird siting near\u00a0bird feeder\u201d, or \u201can image of a bird\u201d all produce different probability paired with the same image:<\/p>\n

<\/figcaption><\/figure>\n

CLIP in a real project scenario<\/h2>\n

To\u00a0illustrate\u00a0the potential of CLIP, I would like to show a real project use case, based on\u00a0one\u00a0of the projects I worked on for a customer,\u00a0an\u00a0image similarity search engine.\u00a0In this project, a user submits an image to model and as a result get a list of images that are visually\u00a0similar to\u00a0the\u00a0query\u00a0image.\u00a0In our case, images\u00a0being searched\u00a0corresponded\u00a0to pages of PDF documents and may contain\u00a0individually or a mix of\u00a0text, tables, embedded photos, empty pages, schemas, diagrams,\u00a0and\u00a0technical drawings.\u00a0For the customer,\u00a0the search return images of interest\u00a0were\u00a0technical drawings.\u00a0Additionally,\u00a0we also knew that what\u00a0user searches would only\u00a0be\u00a0based on\u00a0technical drawings.<\/p>\n

The key characteristics of these\u00a0images are that they\u00a0contain a lot of small details that may be highly relevant for interpretation\u00a0and that they\u00a0may contain irrelevant patterns. Here\u00a0are\u00a0a\u00a0couple of\u00a0examples of\u00a0technical drawings:<\/p>\n

<\/figcaption><\/figure>\n

As always, the devil is in the details,\u00a0and in this\u00a0case\u00a0these are things like the fact\u00a0that\u00a0each image contains textual information block in the bottom-right corner. For example, if there are a lot of technical drawings from Pittsburgh Technical Institute, then\u00a0they all will have a very similar text block.\u00a0Thus, a\u00a0neural network may very quickly begin to anchor to that block.<\/p>\n

The\u00a0ResNet-18\u00a0model\u00a0used\u00a0in production was trained using\u00a0SimCLR<\/a>\u00a0approach.\u00a0SimCLR\u00a0is a\u00a0self-supervised\u00a0contrastive learning method that allows to learn good visual representations without image labels.\u00a0The model\u00a0was trained\u00a0on 100k\u00a0images,\u00a0ca.\u00a050 % of which were technical\u00a0drawings\u00a0and the rest were all the other types of images.<\/p>\n

Following this,\u00a0I benchmarked CLIP\u00a0against\u00a0SimCLR\u00a0for\u00a0visual similarity search. I found\u00a0that image features from a released CLIP-based model taken as zero-shot perform on pair with\u00a0SimCLR-based\u00a0model trained specifically for that data.\u00a0This is truly amazing given that\u00a0technical drawings\u00a0are\u00a0not the typical candidates for publicly available datasets.\u00a0I can\u2019t explain why CLIP is able to perform so well on technical drawings, might it be that examples of such drawings were part of the training dataset.\u00a0<\/p>\n

Another\u00a0result shown\u00a0here is that CLIP was not training\u00a0with image similarity in mind. Yet it learned useful representations that may be used in image similarity scenarios.<\/p>\n

CLIP current limitations<\/h2>\n

CLIP authors are open about its limitations.\u00a0CLIP struggles on\u00a0more abstract or systematic tasks such as counting the number of objects\u00a0and on a more\u00a0complex tasks such as estimating relative distances between objects.\u00a0On such datasets, CLIP is only slightly better than random guessing.\u00a0CLIP also struggles\u00a0with very fine-grained classification,\u00a0such as telling the difference between car models, variants of aircraft, or flower species.<\/p>\n

CLIP model\u00a0itself is data hungry and expensive to train. If pre-trained model doesn\u2019t work well for you, it may\u00a0be not feasible to train your own version.<\/p>\n

While zero-shot CLIP\u00a0tries to reformulate classification task,\u00a0the principles are still the same. And although CLIP\u00a0generalizes well to many image distributions, it still generalizes poorly to\u00a0data that is truly out-of-distribution. One example of this was CLIP\u2019s performance on MNIST\u00a0dataset where CLIP zero-shot accuracy was 88 %.\u00a0Logistic regression on raw pixels\u00a0outperforms CLIP.<\/p>\n

Ability to adapt to new datasets and classes\u00a0is related to text encoder. It is thus limited to choosing from only those concepts\u00a0known to the encoder.\u00a0CLIP model trained with English texts will be of little help\u00a0if used with texts in other languages.<\/p>\n

Finally,\u00a0CLIP\u2019s classifiers can be sensitive to wording\u00a0in label descriptions\u00a0and\u00a0may require\u00a0trial and error\u00a0to perform well.<\/p>\n

Conclusions<\/h2>\n

CLIP training pushes the boundaries of traditional classifier a bit further, and the released pre-trained model allows one to perform various computer vision tasks\u00a0(classification, image feature utilization)\u00a0with good performance and\u00a0without a need of a training set.\u00a0As one\u00a0of the pain points\u00a0when\u00a0working on real projects in data science is data scarcity, where\u00a0there may be a\u00a0lack of ground truth data or the amount of data is limited.\u00a0As\u00a0I\u00a0show\u00a0with an example, pre-trained CLIP-based model allows to kick-start such projects\u00a0\u2013 thus this development is a welcomed addition to the Data Science toolbox.<\/p>\n

I hope you enjoyed a\u00a0presentation\u00a0the key aspects of how CLIP works, and a\u00a0high-level\u00a0demonstration of what it can be used on.\u00a0<\/p>\n

If you have found it interesting, I highly recommend reading the original paper where authors run a lot of different experiments and show how CLIP performs zero-shot classification on a broad range of datasets.<\/p>\n

How to try out CLIP yourself?<\/h2>\n

I\u2019ve prepared a\u00a0Colab\u00a0notebook<\/u><\/a>\u00a0that shows how to interact with CLIP. There you will find not only the basic procedure, but also some insights into how text descriptions affect the outcome. Be sure to check it out!\u00a0<\/p>\n

That notebook uses 16 portrait photos of 3\u00a0people. I wanted to see if CLIP can discriminate these people. It certainly can! However, as CLIP authors point out in\u00a0their\u00a0paper<\/u><\/a>, at the current state of development CLIP may be not the best candidate to perform such tasks, but it is a good way to see how the model works. Its capabilities are anyway look very interesting.<\/p>\n

Acknowledgments<\/h2>\n

Big thanks to Maximilian Warner<\/a> and Alexander Vaagan<\/a> who helped preparing this post.<\/p>\n

References<\/h2>\n