Contents
INTRODUCTION
Background
Fast transportation systems and rapid transit systems are nerves of economic developments for any nation. Mismanagement and traffic congestion results in long waiting times, loss of fuel and money. It is therefore utmost necessary to have a fast, economical and efficient traffic control system for national development. The monitoring and control of city traffic is becoming a major problem in many countries. With the ever increasing number of vehicles on the road, the Traffic Monitoring Authority has to find new methods of overcoming such a problem. One way to improve traffic flow and safety of the current transportation system is to apply automation and Intelligent control methods. There are many ways to apply automation and intelligent control methods, some of the most used automation techniques are magnetic loop detectors buried in the road, infra-red and radar sensors on the road sides, but their disadvantage are that traffic personnel are required to supervise the traffic 24 hours a day. In contrast, an automated and intelligent systems are more effective. After our research we found two major ways in which we can implement our goal,a) Convolutional Neural Network, b) Background Substraction technique using openCV. While comparing the two, Convolutional Neural Network or CNN requires high speed processors to compute real time data, the processor is required to compute data from multiple lanes and give a real time output for best results, which made using CNN complex.
While Background Subtraction technique processes the video feed in a way where detecting vehicles was much easier, less complex and least hectic for the processor. The background subtraction method subtracts the background from an image giving us our required foreground. Processing the image for better quality and noise reduction, background subtraction was ideal for our project.But no technique is perfect by itself as the real time situations are generally continuously changing and the system has to adapt itself to change in the continuously changing circumstances. In this system traffic light are connected to logic so as to figure out which particular lane to be given priority, how much time to be given to each lane, according to the traffic density conditions on the road.Proposed system will be functioning based on traditional system along with automated signaling. System will have artificial vision with the help of digital camera. The camera is controlled by a micro controller to face each lane and capture video. Maximum and minimum time limit will be maintained to prevent over waiting of vehicle in queue. Controls of the signal will be routed through the micro controller. A real time traffic conditions can be viewed by the general public through the mobile application so as to skip high traffic areas. The mobile application also calculates and predicts an estimated time of arrival of public transportation at their destination.
Objectives
The main objectives of this project are:
• Smart traffic management with image processing
• Real time information sharing system with mobile app developed using Flutter.
Problem Statements
Traffic congestion at major junctions in city areas are causing long wait time and high stress to the public. No information sharing system, sharing traffic condition,public vehicle location and estimated time of arrival.
Scope and Application
The transportation management system has following scope and applications:
•Smart traffic system
•Information sharing system
Organization of Report
The following block diagram presents the chapters included in the report presented. It shows the total overview and the organizational flow of the report.CHAPTER 1 consists of background portion along with objectives and problem statements with the scope and application of project. CHAPTER 2 contains the literature review of the project. CHAPTER 3 contains the requirement analysis for the project. CHAPTER 4 contains the system designs used in the project.CHAPTER 5 contains methodology followed for the development of the project.CHAPTER 6 containing the result and analysis of further implementations of the project. CHAPTER 7 contains the discussion part about how development process was carried out. CHAPTER 8 presents the conclusion of the project, its limitations and future enhancements possible for further development of the project.
LITERATURE REVIEW
The projects addressing similar problem like in this project, have been already done in national and international level but each of such projects are provided with different features but here in this project all such special features from different projects are combined into a single project.“Detection and classification of highway lanes using vehicle motion trajectories” [1] paper describes a low-level object tracking system that produces accurate vehicle motion trajectories that can be further analyzed to detect lane centers and classify lane types. Accompanying techniques for indexing and retrieval of anomalous trajectories are also derived. The predictive trajectory merge-and-split algorithm is used to detect partial or complete occlusions during object motion and incorporates a Kalman filter that is used to perform vehicle tracking. “A real-time computer vision system for measuring traffic parameters” [2]uses a feature-based tracking approach for the task of tracking vehicles under congestion. Instead of tracking entire vehicles, vehicle sub-features are tracked to make the system robust to partial occlusion.“Real-time traffic monitoring” [3]applies apply recent real-time trading techniques along with scene specific tuning of the dynamics to enable the tracker to accurately predict target location and thus reduce the amount of search and/or image processing required. “Image analysis and rule-based reasoning for a traffic” [4] paper presents an approach for detecting vehicles in urban traffic scenes by means of rule-based reasoning on visual data. The strength of the approach is its formal separation between the low-level image processing modules and the high-level module,which provides a general-purpose knowledge-based framework for tracking vehicles in the scene. The image-processing modules extract visual data from the scene by spatio-temporal analysis during daytime, and by morphological analysis of headlights at night.“A survey of video processing techniques for traffic applications” [5] paper presents an overview of image processing and analysis tools used in these applications and we relate these tools with complete systems developed for specific traffic applications. More specifically, this paper categorizes processing methods based on the intrinsic organization of their input data (feature-driven,area-driven, or model-based) and the domain of processing (spatial/frame or temporal/video). “Smart Traffic Control System Using Image Processing” [6]journal uses matlab and it aims to prevent heavy traffic congestion. Moreover, for implementing project imaging processing technique has been used. A web camera present in the traffic lane captures image of the road. Then these images are efficiently processes to know the traffic density. “Moving Vehicle Detection for Measuring Traffic Count Using OpenCV” [7],in this paper is designed and implemented with Visual C++ software with Intel’s OpenCV video stream processing system to realize the real-time automatic vehicle detection and vehicle counting. This paper presents inexpensive, portable and Computer Vision based system for moving vehicle detection and counting. Image from video sequence are taken to detect moving vehicles, so that background is extracted from the images. The extracted background is used in subsequent analysis to detect and classify moving vehicles as light vehicles, heavy vehicles and motorcycle. “Smart Traffic Control Using OpenCV” [8]project has been implemented using OpenCV software and aims to have smart ways for traffic management which will ultimately be a cost effective solution. The system has a camera facing the road where traffic is to be controlled. Then ta captured image is processed to9know the traffic density. According to the processed data from the OpenCV, an8051 micro controller will send command to traffic LED timer to show particular time on the signal to manage traffic. “Automated vehicle counting using image processing and machine learning”[9] paper presents a low-cost procedure to perform automatic vehicle counting using remote video cameras with an automatic counting algorithm. The procedure utilizes a Raspberry Pi micro-computer to detect when a car is in a lane, and generate an accurate count of vehicle movements. The method utilized in this paper used background subtraction to process the images and a machine learning algorithm to provide the count. This method avoids fatigue issues that are encountered in manual video counting and prevents the disruption of road ways that occurs when installing pneumatic tubes. OpenCV (Open source computer vision) is a library of programming functions mainly aimed at real-time computer vision. Originally developed by Intel, it was later supported by Willow Garage then It seez (which was later acquired by Intel. The library is cross-platform and free for use under the open-source BSD license.
REQUIREMENT ANALYSIS
Hardware Requirement
It consists of different hardware components that are required to build a system.The hardware requirements for this project are Raspberry PI, Camera, and LED
Tools Used
•MongoDB Atlas
•Flutter 2.0
•Linux (Emedded/ PC)
•Multiprocessing (Python)
•Node (Express.js)
•OpenCV
•Python 3.7
•Sockets (Networking)
System Requirement
The system requirements are as follows:
•Processor – quad-core
•RAM – 6 GB
•Internet Connection
Raspberry Pi 3
A Raspberry Pi is a credit card-sized computer originally designed for education,inspired by the 1981 BBC Micro. The Raspberry Pi is slower than a modern laptop or desktop but is still a complete Linux computer and can provide all the expected abilities that implies, at a low-power consumption level.
Camera
A camera is an optical instrument to capture still images or to record moving images, which are stored in a physical medium such as in a digital system or on photographic film. A camera consists of a lens which focuses light from the scene,and a camera body which holds the image capture mechanism.
LED
A light-emitting diode (LED) is a semiconductor light source that emits light when current flows through it. Electrons in the semiconductor recombine with electron holes, releasing energy in the form of photons. The color of the light(corresponding to the energy of the photons) is determined by the energy required for electrons to cross the band gap of the semiconductor. We have used Green,Yellow and Red colored LED to implement a traffic light.
Software Requirement
It consists of different software components that are required to build a system.Our sensor node consists of combination of Back-end, Front-end and Image Processing.
Back-end
The back end is a REST API. REST is an acronym for Representational State Transfer. It is a software architectural style that defines a set of constraints to be used for creating web based database. The framework used is Express js. The data base is hosted in mlab.com, a cloud hosting service with MongoDB. This database is accessed through a client and data is stored in key-value pair (JSON). MongoDB is a cross-platform document-oriented database program. It is classified as a NoSQL, it uses JSON-like documents with schema. It supports field, range query and regular expression searches. It has load balancing and data replication features over multiple machines for storing files.
Front-end
The front end is a mobile application developed using Flutter. Flutter is a cross platform application development framework. It was developed by Google Inc. It uses dart programming langugage. It also uses various plugins that are developed by flutter developers as well as third party plugins. This mobile application is a front end which allows user to read the traffic density and status as processed from the camera video.It is a data relaying portal which shares the Traffic Location information of all installations, user information, public transportation locations, Emergency vehicle information.
Image Processing
Image processing is a method to convert an image into digital form and perform some operations on it, in order to get an enhanced image or to extract some useful information from it. It is a type of signal dispensation in which input is image, like video frame or photograph and output may be image or characteristics associated with that image. Usually Image Processing system includes treating images as two dimensional signals while applying already set signal processing methods to them. It is among rapidly growing technologies today, with its applications in various aspects of a business. Image Processing forms core research area within engineering and computer science disciplines too.Image processing basically includes the following three steps:•Importing the image with optical scanner or by digital photography. 15•Analyzing and manipulating the image which includes data compression and image enhancement and spotting patterns.•Output is the last stage in which result can be altered image or report that is based on image analysis. In our project (python bindings for) OpenCV was used for the image processing tasks.
SYSTEM DESIGN
System Design
Our system is mostly software with some parts hardware. The system block diagram is shown as follows:
Hardware Design
Our hardware consists of micro controller connected to individual surveillance cameras. The Micro controller process the feed from camera for the required information, and accordingly helps control the traffic lights.
Software Design
Software design consists of three different parts, Back-end, Image Processing and Front-end.
Back-end Design
The back end is a REST API, and the front end is a Mobile Application made using Flutter.
Image Processing Design
Front-end Design
Traffic Routing Mechanisms
METHODOLOGY
Methodology
Our project is divided into four parts:
a)Back-end
b)Front-end
c)Image Processing
d)Hardware Interfacing
Back-end
It stores data such as: Traffic Location and status, User Information, Provides basic authentication mechanism.
Front-End
It is a data relaying portal which shares the Traffic Location information of all installations, user information, public transportation locations.
Image processing
Image processing, involves the following steps:
i.Image acquisition
ii.Pre-processing
◦Region of Interest Selection
◦Blurring and Thresholding
iii.Foreground extraction
◦Background segmentation
iv.Background extraction
◦average images
v.Object detection and tracking
◦find Contours
◦Track each vehicle
vi.Lane Detection
◦Edge Detection
◦Line Detection
i) Image acquisition
It is the action of retrieving an image from some source, usually a hardware-based source, so it can be passed through whatever processes need to occur afterward. In our project we have used OpenCV to capture images.
ii) Preprocessing:
Preprocessing an image inovolve removing low-frequency background noises,normalizing the intensity of the individual particles images, removing reflections,and masking portions of images. Image pro processing is the technique enhancing data images prior to computational processing.-Region Of Interest (ROI) selection: They are samples within a data set identified for a particular purpose. The lanesare seperated for the background.28-Blurring and thresholding:Blurring: Image blurring is achieved by convolving the image with a low-pass filter kernel. It is useful for removing noises. It actually removes high frequency content (eg: noise, edges) from the image.Thresholding: If a pixel value is greater than threshold value, it is assigned one value else it is assignment another value. The source image should be grayscale image, and and a threshold value is used to classify the pixel values.
iii) Foreground Extraction
Foreground -background sereration is a segmentation task whose aim is to splitthe image into foreground and background. It uses Gradient based algorithm which selects a few points in the foreground of an image and a few points in the background, these points are called seed points. The way it works is by spreading region labels outwards from the seed points, but stopping at object boundaries. After the stage of image pre-processing object localization is required which may make use of this technique. Detecting foreground to separate these changes taking place in the foreground of the background.
iv) Background extraction:
Many applications do not need to know about the evolution of movement in a video sequence, but only require the information of changes in the scene, because an image’s regions of interest are objects(humans, cars, text etc.) in its foreground.It is a set of techniques that typically analyze the video sequences in real time and are recorded with a stationary camera. The stationary background is extracted for the detection of lanes in the road.
v) Object detection and tracking Object tracking is a process of taking an initial set of object detection, creating a unique ID for each of the initial detectors and then tracking each of the objects as they move around frames in a video, maintaining the assignment of unique Ids. It allows us to apply a unique ID to track each object making it possible to continue objects in a video. [10]Object detection is a computer technology related to computer vision and image processing that deals with detecting instances of semantic objects of a certain class(such as humans, buildings, or cars) in digital images and videos. Well-researched domains of object detection include face detection and pedestrian detection.Object detection has applications in many areas of computer vision, including image retrieval and video surveillance.
For Traffic Congestion Detection, same above technique of foreground detection is used. But during STOP traffic signal, vehicles are stationary.Back ground Subtract MOG needs static background and moving foreground for detection.So we create a static background by finding the average of n input frames. This way only the background image is obtained and all the noise i.e. vehicles and people is removed. After obtaining the static background processing is done to track and count vehicles.
vi) Lane DetectionLane detection involves the following steps:
Edge detection
Edge detection includes a variety of mathematical methods that aim at identifying points in a digital image at which the image brightness changes sharply or, more formally, has discontinuities. The points at which image brightness changes sharply are typically organized into a set of curved line segments termed edges.We have used the edge detection technique to detect the edges of the object (ie.Vehicles) for an accurate detection, as well as the edges of a road.
Line detection
In image processing, line detection is an algorithm that takes a collection of n edge points and finds all the lines on which these edge points lie. The most popular line detectors are the Hough transform and convolution based techniques.In our project we have used the line detection to divide the traffic lanes so as to figure out the incoming traffic and outgoing ones.
iv) Normalization
It is a method which reduces the redundancy and duplication of data.Normalization helps to reduce number of bounding boxes of an object. It calculates the average and displays only one box consisting of the required object.Where as line normalization reduces the number of detected lines and only shows the accurate one.
Hardware interfacing
Information exchange between a program running on the Raspberry Pi and a partner program running on a remote computer system becomes important when the Raspberry Pi is a front end of a measurement system and transfers sensor data in real-time to a control station or when a remote PC sends commands to a Raspberry Pi based robot (remote control mode). In most situations the data link is established via TCP/IP and the well-established client/server technology based on socket programming is used.
Algorithms
Main loop:
•Step 1: Start
•Step 2: Capture image.
•Step 3: Image pre-processing.
•Step 4: Vehicles detection and tracking.
•Step 5: Has lanes been detected?
◦If yes Goto Step 7
◦Else Goto Step 6
•Step 6: Lane detection and tracking.
•Step 7: Process the data (extract vehicle count and density).
•Step 8: Set value in Queue.
•Step 9: Go to step 2
Vehicle detection and tracking:
•Step 1: Start
•Step 2: Initialize track Dict, id-count.
•Step 3: Get frame.
•Step 4: Apply background segmentation (returns contours).
•Step 5: If track Dict is empty
◦Register each detected contour with a unique id and it’s centroid.34
◦Else go to step 6.
•Step 6: for each contour detected.
◦Compare the detected centroid with every tracked centroids.
◦If the difference is less than a THRESHOLD then the new centroid is just previous centroid in new position, so only update the centroid.
◦Else register the contour with a unique id & it’s centroid.
•Step 7: Goto Step 3.
Lane Detection:
•Step 1: Start
•Step 2: Capture frame.
•Step 3: RGB to Grayscale conversion.
•Step 4: Canny edge detector.
•Step 5: Hough line detector.
•Step 6: Optimize detected lines.
•Step 7: Stop
Watcher loop:
watcher loop is the main controller. It acts as a Master controller where all slaves (camera feed processes) send the data. Each slave processes the above mentioned algorithms steps. The watcher loop is as follows:
•Step 1: Start
•Step 2: Initialize priority Dict with a collection of density and count of each lane belonging to each direction.
•Step 3: Compare count and density of each lane and assign priority accordingly.
•Step 4: If the priority of the lane is HIGH turn the traffic light GREEN.
◦Change LOW priority lane to RED.
•Step 4: Has the waiting time elapsed?
◦If yes go to step 3 else wait.
Priority Algorithm
•Step 1: Start
•Step 2: Initialize priority Dict with initial values.
•Step 3: Compare major flow lane and stopped lane density values.
•Step 4: If the density of current major flow lane is still greater than stop lane then allocate it a time period (T+t).
•Step 5: Change the major flow lane to STOP and stop lane to GO.
•Step 6: Update the Queue with Count and Density.
•Step 7: Go to step 2
Density calculation during stop
•Step 1: Start
•Step 2: Is the traffic light RED or GREEN?
◦If GREEN: Calculate density and count. Go to step 6.
◦If RED: Go to step 3.
•Step 3: Subtract current frame with static background frame.
•Step 4: Threshold the output image to get contours.
•Step 5: Calculate the overall area/ volume of contours.
•Step 6: Update the Queue.
•Step 7: Go to step 2
Density calculation flowchart
Algorithms used:
a)Background Subtraction: Back ground Subtract MOG is the algorithm used in our project. It is a Gaussian Mixture-based Background/Foreground Segmentation algorithm. It was introduced in the paper “Anim proved adaptive background mixture model for real-time tracking with shadow detection” by P. Kadew Tra Ku Pong and R. Bowden in 2001. Ituses a method to model each background pixel by a mixture of K Gaussian distributions (K=3 to 5). The weights of the mixture represent the time proportions that those colors stay in the scene. The probable background colors are the ones which stay longer and more static. [11]b)Contour Detection: It finds contours in a binary image. The function retrieves contours from the binary image using the algorithm. The contours are a useful tool for shape analysis and object detection and recognition. c)Edge Detection: It is a multi-stage algorithm where Noise reduction is the most important. As all images are susceptible to noise, a Gaussiam filter isused to remove the noise. It then finds the intensity gradient of the image,the deduced points undergoes Non-maximum Suppression, which eliminates the unnecessary points, and fnally Hysteresis Thresholding is done to determine the actual edges.[12]d)Line Detection (Hough line Transform Algorithm): It creates a 2D array which holds two parameters (rho, theta). When first point of the line is detected the cell (rho, theta) = 1, on detecting the second point the cell value = 2 and so on. When the cell with the maximum value is seen, it means that there is a line in the image at a distance of rho and at an angletheta degrees.e)Object tracking / Centroid tracking: It accepts bounding box coordinates from the detection process, then computes the centroids of the boxes. It also computes the Euclidean distance between new bounding boxes and39existing objects. It updates the changing coordinates of the existing objects as it moves between the frames. It registers all the objects underan id, finally de registers the old objects and continues to appoint id to the new ones. This process helps in the counting of vehicles in the traffic.
RESULT AND ANALYSIS
Result
Our team divided work sectors wise: Image processing, Back-end , mobile application and model implementations. The following results were achieved from this project:
•Traffic density and flow calculated for each individual lanes.
•Traffic data was successfully shared using mobile application and back end.
Implementations
Our project can be implemented:
•On various major roads with surveillance cameras.
•It can be used for information sharing among users.
DISCUSSION
Discussion
Using various technologies, we were able to identify moving vehicles in a road.So, by collecting various data such as in-going and out-going of vehicles from a traffic junction we could determine the weight of a lane (vehicles flowing through the lane). Using this weight we were able to set priorities and assign time slot to each lane(stop time or flow time). Further more, this data collected from traffic centers can be used to predict traffic flow, determine best path for emergency vehicles and so on.The UI serves as an information dissemination center. Public can see information about public vehicles, where their location is and their ETA.
CONCLUSION AND FUTURE ENHANCEMENTS
Conclusion
Hence, various technologies e.g. Image Processing (OpenCV), User Application(Andriod/ IOS) and RESTful API, can be used to solve the problems of Traffic Congestion Management.
Limitations
The following limitations were seen which can be modified for future:
•Customized board for better image processing: Raspberry Pi is used for general purpose, but our system requires NPU.
•Artificially intelligent model for vehicle detection and tracking.
•Low computation power of Raspberry Pi.
Future Enhancements
The prototype is made for a simple four way pathway, more complex road ways requires further coding.
•Development of Artificially intelligent system (learns with experience).
•Mobile Applications for informing users about approaching emergency vehicles, and location of public vehicles.
•A system can be introduced which helps traffic police supervise rule breaking vehicle.
•Hardware with high processing capacity, preferably with NPU.
APPENDIX
PROGRAMMING
TASK
LANES
import cv2
from utils import roi
import numpy as np
import math
from globals import MAX_LINE_GAP, MIN_LINE_LENGTH, THETA
class LaneDetector:
'''
Detects lanes from given input frame.
Parameters
@param rho: Number
Distance resolution in pixels of the Hough grid.
@param lThresh: Number
Minimum number of votes/intersections in Hough grid cell.
'''
def __init__(self, height, width, rho=1, lThresh=150, sig=0.33, sThresh=7):
self.rho = rho
self.sig = sig
self.lThresh = lThresh
self.sThresh = sThresh
self.height = height
self.width = width
self.frame = None
self.points = None
self.kernel = np.ones((15, 15), np.uint8)
self.count = 0
def update(self, frame):
'''
Called on each frame update.
Parameters
@param frame: List
Numpy array of an image.
'''
self.count += 1
if self.count > 150:
if self.canny(frame):
if self.drawLanes():
self.points.sort()
self.averagePoints()
return self.points
else:
print("failed ...")
return False
def canny(self, frame):
v = np.median(frame)
lower = int(max(0, (1.0 - self.sig) * v))
upper = int(min(255, (1.0 + self.sig) * v))
self.frame = roi(cv2.Canny(frame, lower, upper))
return True
def drawLanes(self):
'''
Draws possible lanes from an edged frame.
'''
closing = cv2.morphologyEx(self.frame, cv2.MORPH_CLOSE, self.kernel)
lines = cv2.HoughLinesP(closing, self.rho, THETA, self.lThresh,
np.array([]), MIN_LINE_LENGTH, MAX_LINE_GAP)
points = []
for line in lines:
for x1, y1, x2, y2 in line:
points.append([(x1, y1), (x2, y2)])
self.points = points
return True
def averagePoints(self):
head = self.points[0]
buffer = []
temp = []
for point in self.points[1:]:
if point[0][0] - head[0][0] < 20:
temp.append(point)
head = point
else:
if len(temp) > 0:
buffer.append(temp)
temp = []
else:
buffer.append([point])
head = point
self.points = list(np.average(items, axis=0) for items in buffer)
TRACKER
from scipy.spatial import distance as dist
from collections import OrderedDict
import numpy as np
class CentroidTracker():
def __init__(self, maxDisappeared=1):
self.nextObjectID = 0
self.objects = OrderedDict()
self.disappeared = OrderedDict()
self.averageDensity = 0
self.maxDisappeared = maxDisappeared
def register(self, centroid):
self.objects[self.nextObjectID] = centroid
self.disappeared[self.nextObjectID] = 0
self.nextObjectID += 1
def deregister(self, objectID):
del self.objects[objectID]
del self.disappeared[objectID]
def count(self):
count = self.nextObjectID - 1
return count
def density(self, area=None):
if area is None:
return self.averageDensity
self.averageDensity = area
def update(self, rects):
if len(rects) == 0:
# loop over any existing tracked objects and mark them
# as disappeared
for objectID in self.disappeared.keys():
self.disappeared[objectID] += 1
# if we have reached a maximum number of consecutive
# frames where a given object has been marked as
# missing, deregister it
if self.disappeared[objectID] > self.maxDisappeared:
self.deregister(objectID)
# return early as there are no centroids or tracking info
# to update
return self.objects
# initialize an array of input centroids for the current frame
inputCentroids = np.zeros((len(rects), 2), dtype="int")
# loop over the bounding box rectangles
for (i, (startX, startY, endX, endY)) in enumerate(rects):
# use the bounding box coordinates to derive the centroid
cX = int((startX + endX) / 2.0)
cY = int((startY + endY) / 2.0)
inputCentroids[i] = (cX, cY)
# if we are currently not tracking any objects take the input
# centroids and register each of them
if len(self.objects) == 0:
for i in range(0, len(inputCentroids)):
self.register(inputCentroids[i])
# otherwise, are are currently tracking objects so we need to
# try to match the input centroids to existing object
# centroids
else:
# grab the set of object IDs and corresponding centroids
objectIDs = list(self.objects.keys())
objectCentroids = list(self.objects.values())
# compute the distance between each pair of object
# centroids and input centroids, respectively -- our
# goal will be to match an input centroid to an existing
# object centroid
D = dist.cdist(np.array(objectCentroids), inputCentroids)
# in order to perform this matching we must (1) find the
# smallest value in each row and then (2) sort the row
# indexes based on their minimum values so that the row
# with the smallest value as at the *front* of the index
# list
rows = D.min(axis=1).argsort()
# next, we perform a similar process on the columns by
# finding the smallest value in each column and then
# sorting using the previously computed row index list
cols = D.argmin(axis=1)[rows]
# in order to determine if we need to update, register,
# or deregister an object we need to keep track of which
# of the rows and column indexes we have already examined
usedRows = set()
usedCols = set()
# loop over the combination of the (row, column) index
# tuples
for (row, col) in zip(rows, cols):
# if we have already examined either the row or
# column value before, ignore it
# val
if row in usedRows or col in usedCols:
continue
# otherwise, grab the object ID for the current row,
# set its new centroid, and reset the disappeared
# counter
objectID = objectIDs[row]
self.objects[objectID] = inputCentroids[col]
self.disappeared[objectID] = 0
# indicate that we have examined each of the row and
# column indexes, respectively
usedRows.add(row)
usedCols.add(col)
# compute both the row and column index we have NOT yet
# examined
unusedRows = set(range(0, D.shape[0])).difference(usedRows)
unusedCols = set(range(0, D.shape[1])).difference(usedCols)
# in the event that the number of object centroids is
# equal or greater than the number of input centroids
# we need to check and see if some of these objects have
# potentially disappeared
if D.shape[0] >= D.shape[1]:
# loop over the unused row indexes
for row in unusedRows:
# grab the object ID for the corresponding row
# index and increment the disappeared counter
objectID = objectIDs[row]
self.disappeared[objectID] += 1
# check to see if the number of consecutive
# frames the object has been marked "disappeared"
# for warrants deregistering the object
if self.disappeared[objectID] > self.maxDisappeared:
self.deregister(objectID)
# otherwise, if the number of input centroids is greater
# than the number of existing object centroids we need to
# register each new input centroid as a trackable object
else:
for col in unusedCols:
self.register(inputCentroids[col])
# return the set of trackable objects
return self.objects
UTILS
import os
import cv2
import math
import numpy as np
from globals import *
def save(frame, dir, name="untitled"):
'''
Saves specified frame into a directory.
Parameters
----------
@params: frame: Numpy Array
@params: dir : String
@params: name : String
'''
if dir is not None:
return cv2.imwrite(os.path.join(dir, name), frame)
return cv2.imwrite(name, frame)
def video(dir, name="processed.mp4"):
try:
images = [img for img in os.listdir(dir) if img.endswith(".png")]
frame = cv2.imread(os.path.join(
FILE_DIR, "data", "processed", images[0]))
height, width, layers = frame.shape
video = cv2.VideoWriter(name, 0, 1, (width, height))
for image in images:
video.write(cv2.imread(os.path.join(
FILE_DIR, "data", "processed", image)))
return True
except Exception as e:
return False
def getCap(device=0):
'''
Gets media capture from a device.
Parameters
----------
@params: device: Integer
@returns: cap: Device capture
'''
cap = cv2.VideoCapture(device)
if cap.isOpened():
return cap
def getBBoxes(contour, objects, frame):
'''
Draws rectangles around contours.
Parameters
----------
@param: contour: List
@param: frame : Numpy Array
@param: color : RGB tuple
@return frame: List
Numpy array of an image.
'''
for (objectID, centroid), c in zip(objects.items(), contour):
text = "ID {}".format(objectID)
x, y, w, h = cv2.boundingRect(c)
cv2.putText(frame, text, (centroid[0] - 10, centroid[1] - 10),
CV_FONT, 0.4, TEXT_COLOR, 1, CV_AA)
cv2.circle(frame, (centroid[0], centroid[1]), 4, TEXT_COLOR, -1)
cv2.rectangle(frame, (x, y), (x+w, y+h), TEXT_COLOR, 2)
# rect = cv2.minAreaRect(c)
# box = cv2.boxPoints(rect)
# box = np.int0(box)
# frame = cv2.drawContours(frame, [box], 0, (0, 0, 255), 2)
return frame
def smoothContours(contours, thresh=1):
'''
Combines multiple bounding boxes into a single one.
Parameters
----------
@param: contour: List
@param: thresh: Int
@return contour: List
Array of smoothed contours.
'''
if contours is None:
return
try:
contours, weights = cv2.groupRectangles(contours, thresh)
except Exception as e:
pass
# if width of the contour is very high than it's
# height then it is probably an error.
contours = list(x for x in contours if cv2.boundingRect(x)
[2]/2.5 <= cv2.boundingRect(x)
[3] and cv2.contourArea(x) > 50)
return contours
def findHull(contour):
'''
A function to calculate convex hull of contours.
Parameters
----------
@param: contour: List
@return hull: List
Array of convex hulls.
'''
return list(cv2.convexHull(x) for x in contour)
def approxCnt(contours):
smooth = smoothContours(contours)
return findHull(smooth)
def getBoxes(contour):
'''
Gets bounding boxes from contours.
Parameters
----------
@param: contour: List
@returns: boxes: List
'''
if contour is None:
return []
boxes = []
area = 0
for c in contour:
area += cv2.contourArea(c)
x, y, w, h = cv2.boundingRect(c)
boxes.append((x, y, x+w, y+h))
return boxes, area # list(cv2.boundingRect(c) for c in contour)
def roi(frame):
'''
Extracts a region of interest from given frame.
Parameters
----------
@param frame: List
Numpy array of an image.
@return frame: List
Numpy array of an image.
'''
imshape = frame.shape
vertices = np.array([[(0, imshape[0]/1.3),
(350, 390),
(850, 390),
(imshape[1], imshape[0]),
(0, imshape[0])]
], dtype=np.int32)
mask = np.zeros(imshape, dtype=np.uint8)
cv2.fillPoly(mask, vertices, (255, 255, 255))
return cv2.bitwise_and(mask, frame)
def compare(array):
'''
Compares each and every element of an array with other.
Parameters
----------
@params: array: List
'''
for i in range(len(array)):
for j in range(i + 1, len(array)):
return array[i]-array[j]
EXTRACTOR
import cv2
import numpy as np
from globals import BG_IMG_DATA
class Extractors:
'''
Implements foreground, background extraction and image subtraction.
Parameters
----------
@params : Frame
A numpy array of input image.
'''
def __init__(self, frame):
self.bg_sub = cv2.bgsegm.createBackgroundSubtractorMOG()
self.kernel = np.ones((7, 7), np.uint8)
self.bg_avg = np.float32(frame)
self.frame = frame
def update(self, frame, jobType="bg"):
self.frame = frame
if jobType == "fg":
return self.extractForeground()
return self.extractBackground()
def extractForeground(self):
'''
Extracts foreground from a supplied frame.
Parameters
----------
@params : Frame
A numpy array of input image.
@return : Frame
Foreground extracted frame.
'''
sub = self.bg_sub.apply(self.frame)
closing = cv2.morphologyEx(sub, cv2.MORPH_CLOSE, self.kernel)
return closing
def extractBackground(self):
'''
Extracts background from a supplied frame.
Parameters
----------
@params: Frame
A numpy array of imput image.
@return : Frame
Background extracted frame.
'''
cv2.accumulateWeighted(self.frame, self.bg_avg, 0.01)
res = cv2.convertScaleAbs(self.bg_avg)
return res
def subtractor(self):
return cv2.subtract(cv2.imread(BG_IMG_DATA), self.frame)
APP.PY
import cv2 as cv
import asyncio
from tasks import LaneDetector
from extractor import Extractors
from tracker import CentroidTracker
from utils import roi, getBoxes, getCap, approxCnt, getBBoxes
from globals import VID_DATA_DIR, TEXT_COLOR, CV_FONT, CV_AA, ROI_AREA
print('using OpenCV {}'.format(cv.__version__))
cap = getCap('{}/one.mp4'.format(VID_DATA_DIR))
tracker = CentroidTracker()
ex = Extractors(roi(cap.read()[1]))
WIDTH = int(cap.get(cv.CAP_PROP_FRAME_WIDTH))
HEIGHT = int(cap.get(cv.CAP_PROP_FRAME_HEIGHT))
lanes = LaneDetector(HEIGHT, WIDTH)
async def detectVehicles(frame):
sub = ex.update(frame, "fg")
contours, _ = cv.findContours(
sub, cv.RETR_EXTERNAL, cv.CHAIN_APPROX_SIMPLE
)
hulls = approxCnt(contours)
boxes, area = getBoxes(hulls)
objects = tracker.update(boxes)
if len(boxes) > 0:
tracker.density(int(area))
else:
tracker.density(area)
frame = getBBoxes(hulls, objects, frame)
return frame, sub
async def detectLanes(frame):
background = ex.update(frame)
return lanes.update(background)
async def main():
while cap.isOpened():
start = cv.getTickCount()
_, frame = cap.read()
detection, ret = await detectVehicles(roi(frame))
averagedLines = await detectLanes(frame)
if averagedLines is not False:
for points in averagedLines:
try:
cv.line(detection, (int(points[0][0]),
int(points[0][1])),
(int(points[1][0]),
int(points[1][1])), (255, 0, 0), 5)
except Exception as e:
print(e)
end = cv.getTickCount()
fps = 'FPS: '+str(int(1/((end - start)/cv.getTickFrequency())))
cv.putText(detection, fps, (20, 50), CV_FONT,
0.8, TEXT_COLOR, 1, CV_AA)
cv.putText(detection, "Count: {}".format(tracker.count()), (20, 80),
CV_FONT, 0.8, TEXT_COLOR, 1, CV_AA)
cv.putText(detection, "Density: {:.3f}%".format(tracker.density()*100 /
ROI_AREA), (20, 115),
CV_FONT, 0.8, TEXT_COLOR, 1, CV_AA)
cv.imshow('frame', detection)
if cv.waitKey(1) & 0xFF == ord('q'):
break
cv.destroyAllWindows()
cap.release()
if __name__ == '__main__':
asyncio.run(main())
GLOBALS.PY
import math
from numpy import pi
from os.path import dirname, abspath, join
from cv2 import FONT_HERSHEY_SIMPLEX, LINE_AA
ROI_AREA = 534632-100000
CV_FONT = FONT_HERSHEY_SIMPLEX
TEXT_COLOR = (255, 255, 255)
CV_AA = LINE_AA
FILE_DIR = dirname(abspath(__file__))
VID_DATA_DIR = join(FILE_DIR, 'data', 'video')
IMG_DATA_DIR = join(FILE_DIR, 'data', 'image')
BG_IMG_DATA = join(FILE_DIR, 'data', 'image', 'background.jpg')
MIN_LINE_LENGTH = 200 # minimum number of pixels making up a line
MAX_LINE_GAP = 60 # maximum gap in pixels between connectable line segments
THETA = pi/180 # angular resolution in radians of the Hough grid
YOUTUBE LINK
This project is done by :
ASHISH KAFLE
NEHA MANANDHAR
PRABESH ADHIKARI
ROHIT KUINKEL
