Capture video data from screen in Python

67
December 03, 2019, at 01:30 AM

Is there a way with Python (maybe with OpenCV or PIL) to continuously grab frames of all or a portion of the screen, at least at 15 fps or more? I've seen it done in other languages, so in theory it should be possible.

I do not need to save the image data to a file. I actually just want it to output an array containing the raw RGB data (like in a numpy array or something) since I'm going to just take it and send it to a large LED display (probably after re-sizing it).

Answer 1

There is an other solution with mss which provide much better frame rate. (Tested on a Macbook Pro with MacOS Sierra)

import numpy as np
import cv2
from mss import mss
from PIL import Image
mon = {'top': 160, 'left': 160, 'width': 200, 'height': 200}
sct = mss()
while 1:
    sct.get_pixels(mon)
    img = Image.frombytes('RGB', (sct.width, sct.height), sct.image)
    cv2.imshow('test', np.array(img))
    if cv2.waitKey(25) & 0xFF == ord('q'):
        cv2.destroyAllWindows()
        break
Answer 2

You will need to use ImageGrab from Pillow (PIL) Library and convert the capture to numpy array. When you have the array you can do what you please with it using opencv. I converted capture to gray color and used imshow() as a demonstration.

Here is a quick code to get you started:

from PIL import ImageGrab
import numpy as np
import cv2
img = ImageGrab.grab(bbox=(100,10,400,780)) #bbox specifies specific region (bbox= x,y,width,height *starts top-left)
img_np = np.array(img) #this is the array obtained from conversion
frame = cv2.cvtColor(img_np, cv2.COLOR_BGR2GRAY)
cv2.imshow("test", frame)
cv2.waitKey(0)
cv2.destroyAllWindows()

you can plug an array there with the frequency you please to keep capturing frames. After that you just decode the frames. don't forget to add before the loop:

fourcc = cv2.VideoWriter_fourcc(*'XVID')
vid = cv2.VideoWriter('output.avi', fourcc, 6, (640,480))

and inside the loop you can add:

vid.write(frame) #the edited frame or the original img_np as you please

UPDATE
the end result look something like this (If you want to achieve a stream of frames that is. Storing as video just a demonstration of using opencv on the screen captured):

from PIL import ImageGrab
import numpy as np
import cv2
while(True):
    img = ImageGrab.grab(bbox=(100,10,400,780)) #bbox specifies specific region (bbox= x,y,width,height)
    img_np = np.array(img)
    frame = cv2.cvtColor(img_np, cv2.COLOR_BGR2GRAY)
    cv2.imshow("test", frame)
    cv2.waitKey(0)
cv2.destroyAllWindows()

Hope that helps

Answer 3

You can try this=>

import mss
import numpy
with mss.mss() as sct:
    monitor = {'top': 40, 'left': 0, 'width': 800, 'height': 640}
    img = numpy.array(sct.grab(monitor))
    print(img)
Answer 4

based on this post and others posts, i made something like this . Its taking a screenshot and writing into a video file without saving the img.

import cv2
import numpy as np
import os
import pyautogui
output = "video.avi"
img = pyautogui.screenshot()
img = cv2.cvtColor(np.array(img), cv2.COLOR_RGB2BGR)
#get info from img
height, width, channels = img.shape
# Define the codec and create VideoWriter object
fourcc = cv2.VideoWriter_fourcc(*'mp4v')
out = cv2.VideoWriter(output, fourcc, 20.0, (width, height))
while(True):
 try:
  img = pyautogui.screenshot()
  image = cv2.cvtColor(np.array(img), cv2.COLOR_RGB2BGR)
  out.write(image)
  StopIteration(0.5)
 except KeyboardInterrupt:
  break
out.release()
cv2.destroyAllWindows()
Answer 5

With all of the above solutions, I was unable to get a usable frame rate until I modified my code in the following way:

import numpy as np
import cv2
from mss import mss
from PIL import Image
bounding_box = {'top': 100, 'left': 0, 'width': 400, 'height': 300}
sct = mss()
while True:
    sct_img = sct.grab(bounding_box)
    cv2.imshow('screen', np.array(sct_img))
    if (cv2.waitKey(1) & 0xFF) == ord('q'):
        cv2.destroyAllWindows()
        break

With this solution, I easily get 20+ frames/second.

For reference, check this link: OpenCV/Numpy example with mss

Answer 6

I tried all of the above but it did not give me the real-time screen update. You can try this. This code is tested and worked successfully and also give you a good fps output. You can also judge this by each loop time it's needed.

import numpy as np
import cv2
from PIL import ImageGrab as ig
import time
last_time = time.time()
while(True):
    screen = ig.grab(bbox=(50,50,800,640))
    print('Loop took {} seconds',format(time.time()-last_time))
    cv2.imshow("test", np.array(screen))
    last_time = time.time()
    if cv2.waitKey(25) & 0xFF == ord('q'):
        cv2.destroyAllWindows()
        break
Answer 7

You can try this .this code is working for me.I tested Linux

import numpy as np
import cv2
from mss import mss
from PIL import Image
sct = mss()
while 1:
    w, h = 800, 640
    monitor = {'top': 0, 'left': 0, 'width': w, 'height': h}
    img = Image.frombytes('RGB', (w,h), sct.grab(monitor).rgb)
    cv2.imshow('test', np.array(img))
    if cv2.waitKey(25) & 0xFF == ord('q'):
        cv2.destroyAllWindows()
        break

make sure that following package is installed

Pillow, opencv-python, numpy, mss

Answer 8

I've tried ImageGrab from PIL and it gave me 20fps which is ok but using win32 libraries gave me +40fps which is amazing!

I used this code by Frannecklp but it didn't work just fine so I needed to modify it:

-Firstly pip install pywin32 in case using the libraries

-import the libraries like this instead:

import cv2
import numpy as np
from win32 import win32gui
from pythonwin import win32ui
from win32.lib import win32con
from win32 import win32api

for geting a simple image screen do:

from grab_screen import grab_screen
import cv2
img = grab_screen()
cv2.imshow('frame',img)

and for getting frames:

while(True):
#frame = grab_screen((0,0,100,100))
frame = grab_screen()
cv2.imshow('frame',frame)
if cv2.waitKey(1) & 0xFF == ord('q') or x>150:
    break
READ ALSO
Powerset algorithm in Python: difference between + and append on lists

Powerset algorithm in Python: difference between + and append on lists

I’m working through the powerset problem in Python

44
Django - Multiple user types (ability to switch between them)

Django - Multiple user types (ability to switch between them)

I am doing some pre-planning on a project I am working on and my approach to itA part of the app is the ability to have multiple user types

52
Function to find number of 'ordered combinations' with no particular length of a list Python

Function to find number of 'ordered combinations' with no particular length of a list Python

While variants of this question have been asked numerous times on this site, I haven't found any information on how to do 'ordered combinations' with a given list

63
jQuery submit function add data several times

jQuery submit function add data several times

I have this function I use to store items (voucher codes) based on some rules

50