How ML models work with images?

Intro¶

If you read chapter 4 from Practical Deep Learning for Coders
and you have questions about all these numbers from images
like what is $28*28$ or $784$ or why we divide these tensors with image data on 255
then this is a post for you

Usual start of working with images¶

Download MNIST SAMPLE images dataset with FastAI function untar_data
os.getcwd() + '/images' returns path of notebook directory and add
subdir images to it
images will be download in this [notebook dir]\images directory

After we have directory with images we use some commands from FastAI
and received these images in nice batches which we can feed to our model

Here is our first batch of images

What is going on under the hood in the previous code block?¶

Let’s make a simulation on the easy example of two numbers from MNIST image dataset

Images unpacking and loading¶

After downloading and unpacking dataset from FastAI or from other source,
we have a directory which contains all images.

In MNIST dataset case this directory contains
subdirectorieswith the name of the number in image

Make two lists of paths to images of 3 and 7

Let’s make two lists of tensors with 3 and 7 images
For this we will use Image.open(full_path_to_image)
tensor for converting image to tensor
[] for packing these tensors in list

Length of list is a quantity of images in this list

Looking inside the image¶

Each item of these lists is a tensor/array which contains image data
three_tensors[5] means 5th image/tensor in our list

torch.Size([28, 28]) means that there are $28*28$ pixels/numbers in the tensor
and each of these numbers is from 0 to 255
this 0 to 255 number is a color:

• 0 – black color / no color
• 128 – middle shade of grey
• 255 – white color

if we look at first 25 rows and middle 14 columns from 10th to 24th
we will see ASCII picture of 3

(if we output more than 14 columns, the notebook splits the row into two rows
and we won’t see the picture)

There is an interesting way of using pandas.DataFrame to receive
more clear ASCII image from this data

and the same image data but in usual image regime

Preparation image data for ML¶

so we have lists of tensors which contain integers from 0 to 255
for machine learning we need to convert these lists to tensors

torch.stack converts list of tensors into sequence of tensors (tensor of tensors)

so for now we have tensor_threes with 6131 elements
each of them contains an 28*28 image

Changing image data’s range¶

machine learning usually better work with numbers from 0 to 1 or from -1 to 1
so we need to convert our data into the small range
to do this we will convert our data in tensor to float and divide each pixel by 255
tensors use broadcasting so this dividing operation divide not the object tensor
but each element/pixel of this tensor

What’s interesting: after dividing we still can see this image in DataFrame visualization
but with different numbers inside

also we still can see this data in usual image regime

Observation about image data¶

It’s always was very strange for me when I saw these transformations of range
in data before ML training. I always thought that this is some mutiliation of the data.

This visualization shows to us that this changing doesn’t change the data
we changed order of digits but not the relation/proportion among pieces of data

Image editors like range 0..255
Machine learning likes smaller range 0..1
but data is the same in both cases

Does ML models work with info about sides of image?¶

Some models work with information about side sizes, for example CNNs
in which case we have ready tensors with data in good format

Some models want to throw off this information about sides sizes, for example FCN
in which case we need to flatten our image data

If you read the chapter 4 from Practical Deep Learning for Coders
that’s exactly what is going on there: we use linear model there
and need to flatten out this $28*28$ images into the row of 784 pixels

we can do this flattening by using command view

number 784 is a
$side * height = 28 * 28 = 784$

this command slice image by rows and concatenate these rows from left to right

If previous explanation about view function doesn’t make sense
let’s see output of this command on the simplier example

we create “image” with sides 3 * 3 and we need to flat it out

What about color images?¶

All this were about greyscale images. And what about full color images?
The same principles but with a little trick from visual domain.

With mix of red green and blue we can mix any color.

So for storing full color image we can use three images and just combine them
to achieve any color.

These three mixed images have a name channels

Manual crafting of a color image¶

Let’s create tensor with the same sides but with additional dimension

earlier we used this dimension as an index of file
so when we saw this: Shape of tensor_trees: torch.Size([6131, 28, 28])
we knew that this is a tensor with 6131 images

for now we will use this dimension as a channel
so when we see this Shape of clr_tensor: torch.Size([3, 28, 28])
we knew that this is an image tensor with 3 channels and 28 * 28 sides

• R – 0 channel is red
• G – 1 channel is green
• B – 2 channel is blue

It’s still the same dimension from the programming point of view,
we just put another sense of use into this dimension.

Let’s look at our image

It’s just black and it’s ok because we create our tensor and fill it with zeroes
which is black for an image

Now we will fill with ones first 5 rows in 0/Red channel in our tensor

so we should receive a horizontal red line

Next channel 1 is green, we will put horizontal line of ones at the bottom

final blue channel 2 we fill it with vertical line of ones
and here we can see interesting effect of color mixing:
left up corner – mix of red and blue gives magenta
left bottom corner – mix of blue and green gives cyan