Let me start by saying, playing around with OpenCV was a lot of fun. I’m genuinely impressed by it and I consider it a very useful resource to have it on your tool belt. What I tried to achieve in this part is having the game state in a processable manner. Currently what I get from the game is an image. We humans understand it perfectly well but in order to write an algorithm which will make some decision based on the game state. I need to extract the meaning out of it. Here is where OpenCV comes in.

My plan is simple.

1. Cut out each cell the game can have on board into a separate png.
2. Iterate over the cutouts and check where do they turn up in the image.
3. Extract the findings into an array which I can process further

To shortcut testing I loaded up a print screen of the game and analyzed that instead of the image that comes from the game. We will handle wiring everything together in a different post. This I what our input looks like.

The following code might look complicated but the process is actually simple.

Point* points = new Point[SIZE];
int* values = new int[SIZE];
int ind = 0;

int templates[]{
	0, //empty
	1, 2, 3, 4, 
	11,//marked bomb
	14 //undiscovered
};
for (auto &t : templates)
{
	Mat ti= imread("../img/" + to_string(t) + ".png");
	Mat result;	
	matchTemplate(img, ti, result, TM_CCOEFF_NORMED); 
	normalize(result, result, 0, 1, NORM_MINMAX, -1, Mat());
	threshold(result, result, 0.90, 1, THRESH_BINARY);
	Merge(img, tresh, ti.cols, ti.rows, t, points, values, &ind);
}

First we declare all the images that we need to test. That is what the templates variable does (i know it is only till 4 but for my input this is perfectly fine for now). Then we iterate through the templates. We load the template image and we request OpenCV to mach our template with the image we have. Template matching is the process where open CV slides the template over the image and checks how close is the template to the image. The result is another image, each pixel of this image is the outcome of the matching at that specific coordinates. In our example the higher the match the brighter the pixel is. If you are more interested about this you can find a lot better explanation in the documentation. The matched template looks like this.

In this image you can see the output after matching the 1. In order to get the coordinates we need to highlight were our matches are even better. This is where threshold method comes in. With the settings we use it will put a white pixel at every point where in our image the pixel brightness is between 0.9 and 1. After tresholding our image looks like this (you might want to open this separately to see it better.

Now you might be wondering how are we getting the coordinates out of it. This is were our Merge method comes in.

void Merge(Mat ref, Mat in, int w, int h, int val, 
Point* points, int* values, int* ind)
{
	double minVal; double maxVal; Point minLoc; Point maxLoc;
	Scalar b(0, 0, 0);
	while (true)
	{
		minMaxLoc(in, &minVal, &maxVal, &minLoc, &maxLoc, Mat());
		if (maxLoc.x == 0 && maxLoc.y == 0)
			break;

		Point p(maxLoc.x + w / 2, maxLoc.y + h / 2);
		circle(in, maxLoc, 15, b, FILLED);
		circle(ref, p, 15, b, FILLED);

		points[*ind] = p;
		values[*ind] = val;
		(*ind)++;
	}
}

What it does is really simple. It takes in our original image and the threshold image:

1. Get the location of the brightest point.
2. Draw a black circle above it in the threshold image so it will not turn up again
3. Draw a black circle above our input image so it will not interfere with other analysis (1 and 4 can be mistaken easily)
4. Update our output vectors with the found coordinates.

We are almost done, we have to assemble the game matrix from the points and their values. This is done in the following way.

void Extract(Point* p, int* v, int* game)
{
	int minX = 10000, minY = 10000, maxX, maxY, sizeX, sizeY;

	for (int i = 0; i < SIZE; i++)
	{
		if (p[i].x < minX) minX = p[i].x;
		if (p[i].y < minY) minY = p[i].y;
		if (p[i].x > maxX) maxX = p[i].x;
		if (p[i].y > maxY) maxY = p[i].y;
	}
	sizeX = ((maxX - minX) / WIDTH);
	sizeY = ((maxX - minX) / WIDTH);
	minX -= sizeX / 2;
	maxX += sizeX / 2;
	minY -= sizeY / 2;
	maxY += sizeY / 2;
	sizeX = ((maxX - minX) / WIDTH);
	sizeY = ((maxX - minX) / WIDTH);

	for (int i = 0; i < SIZE; i++)
	{
		int x = (p[i].x - minX) / sizeX;
		int y = (p[i].y - minY) / sizeY;
		game[y * WIDTH + x] = v[i];
	}
}

Since the detected points are not in perfect placement on a grid, I have to normalize the points. First we get the min/max coordinates. Since these coordinates are going through the center of the edge cells we compensate against it, otherwise vales from one coordinate can fall on the adjacent coordinate in our result. After that we proceed marking the coordinate with the value for which we did the template matching. And the output is (drum-rolls)

1eeeeeeeeeeeeeeeeeeeeeeeeeeee1
eeeeeeeeeeeeeeeeeeeeeeeeeeeeee
eeeeeeeeeeeeeeeeeeeeeeeeeeeeee
eeeeeeeeeeeeeeeeeeeeeeeeeeeeee
eeeeeeeeeeeeeeeeeeeeeeeeeeeeee
eeeeeeeeeeeeeeeeeeeeeeeeeeeeee
eeeeeeeeeeeeeeeeeeeeeeeeeeeeee
111eeeeeeeeeeeeeeeeeeeeeeeeeee
001eeeeeeeeeeeeeeeeeeeeeeeeeee
012eeeeeeeeeeeeeeeeeeeeeeeeeee
01eeeeeeeeeeeeeeeeeeeeeeeeeeee
01eeeeeeeeeeeeeeeeeeeeeeeeeeee
011112eeeeeeeeeeeeeeeeeeeeebbb
01b101eeeeeeeeeeeeeeeeeebb4b42
011101eeeeeeeeeeeeeeeeee322110
000001eeeeeeeeeeeeeeeeee100000

We came a long way since our input image, but this is already something that we can analyze, which we will do next time. See ya!