INTRODUCTION

In this digital world we need every possible thing around us to be automatic and secure which reduces human effort. There are increasing electronic circuits that makes todays life easier and simpler. Nowadays, energy crisis is a big problem faced by everyone. So, there is a need to conserve energy. The aim of this is to make IOT BASED SMART HOME WITH FACE RECOGNITION SECURITY for Secured home automation. In today’s world, face recognition is an important part for the purpose of security and surveillance. Hence there is a need for an efficient and cost effective system. This paper details the design and development of IOT based security surveillance system in buildings with Wi-Fi network connectivity. Upon detecting the face, the controller enables the camera for capturing the event, alerts the user by placing the live video of that event on webpage.

BACKGROUND

IOT or Internet of Things is an upcoming technology that allows us to control hardware devices through the internet. Here, we propose to use IOT in order to control home appliances, thus automating modern homes through the internet. This system uses three loads to demonstrate as house lighting and a fan. Our user friendly interface allows a user to easily control these home appliances through the internet. For this system we use CPU as a processor. Camera is used for face recognition. When an unknown person enters the home i.e. If the camera doesn’t recognize the face of the person entering the home then the alerts the administrator via SMS, notification alert is used to notify the admin when he is inaccessible to the in. After getting the notification, the admin will get access to the internet and see the live streaming of the activities of that person.

SCOPE OF THE PROJECT

This automation prototype is meant to demonstrate the proper functionality and scalability of the system, not to include every desired system feature. There are countless features and devices that the system could eventually incorporate, but in order to maintain proper project scope, only a single user interface have been designed.

Future scope for the home automation systems involves making homes even smarter. Standardization enables smart homes that can control appliances, lighting, environment, energy management and security as well as the expandability to connect with other networks.

Some application areas:

1. Private rooms where security is mandatory.

2. Houses where handicapped people live.

OBJECTIVES

Some objectives of our project are as follows:

1. Lower Security Risks

Safe homes are the most important aspect of any house. No matter how many locks you use for your house, the risk of burglars and thieves breaking into your house is always there. But having a smart home will lower your security risks by providing security alerts when something happens wrong. Then there is the additional catch or the fear of losing your keys. Mostly in case of those members of your family, who frequently lose their keys or forget to lock some of the important doors of your house, it is always a great idea to install the smart home security system to increase your home safety.

With the help of smart home devices such as smart security cameras, you will be able to take care of the security of your home. Even when you are not at home, you can lock the door of your house.

If someone has reached your home and you haven’t reached yet, you can open the door for him or her with the help of smart cameras & security password and your friend or family member can wait inside. You can have motion detectors in your doors, gates and garage which can be open through one click of a smartphone.

2. Save Energy

Energy saving is probably the primary concern for every house owner. With convenient appliances and so many electronic gadgets, the amount of energy consumption is greatly increased. Hence, you would always want to make sure that there is no amount of energy wastage and your home needs to fulfill the need of energy efficiency. 

With smart home products, you can ensure that there is no waste of energy usage. The machines, devices and appliances can all be managed and turned off immediately after usage, when you have a smart home product.

Not just this, controlling the lighting of your house will be a lot easier. Since lighting wastes the maximum amount of energy, you can save a lot with the smart appliances. This is a great advantage of this facility.

If you are not in a room or have left a room thinking that you would come back and hence, kept the light bulb or fans turned on or if you simply forgot to turn the lights off, you can benefit hugely with the smart home system it will help you to increase energy savings.

3. Maintenance and Service Awareness

When your devices and appliances need servicing for maintaining its performance, must be known in advance. However, monitoring this is not always easy. In that case, you can rely on smart home system. This will notify you whenever you need to take care of the appliances and devices.

4. Customization

It is very convenient with smart home automation systems to customize the appliances and smart devices in your home as per your needs. When you want to increase the temperature of your heating system or want to decrease the temperature in your air conditioner, or if you want to turn the water heater on before you reach home or want to turn off everything while lying in the bed, you can do them with just a click on your mobile phone. These features are completely different than our traditional homes.

5. Ease of Using 

Along with all these facilities and benefits, another important smart technology advantage is the ease of using this. You can control all these things, without any trouble or difficulty. You can easily manage these things. Even if you are not a very tech savvy person, you will be able to understand how these things work very easily, with some following of the manuals. With smart home tech you can feel connected home 24/7 even if you are not at home.

Smart homes are the new and the in thing with Artificial Intelligence. If you want to upgrade yourself and your house according to the global standards and want to become a smart homeowner, then smart system setup is what you can do to your house. Initial cost of setup is a little high but when you look into the long run you save a lot of money with the energy transition process only. 

LIMITATIONS

There are many disadvantages of smart homes as well.

1. Dependency on Internet

The basic requirement for the smart home system is the internet. Without a good and strong internet connection, you will not be able to take control of this. If there is no internet connection for some reason, there is no other way through which you can access and control your system.

2. Dependency on Professional

The basic requirement for the smart home system is the internet. Without a good and strong internet connection, you will not be able to take control of this. If there is no internet connection for some reason, there is no other way through which you can access and control your system. so it is very important to pick a good quality internet service provider.

In case there is a problem with the smart home system, you cannot simply call a handyman or someone similar to repair or manage the bug. You will have to depend on the professionals. Only the company professionals can help you to handle the problems. A professional only able to take care of your smart home technologies that lead to high dependency on professionals.

CHALLENGES FACED BY FACE RECOGNITION SYSTEM

Face Recognition has always been one of the most fascinating and intriguing technologies as it deals with human faces. As said, opportunities and challenges go hand in hand. Growing commercial interest for face recognition is encouraging, but it also turns out to be a challenging endeavour when it comes to problems associated which have played continuously with its quality of delivery. These challenges arise when the situations are non-cooperative and causes the varied facial appearance/expressions.

1. Illumination

Illumination stands for light variations.  The slight change in lighting conditions cause a significant challenge for automated face recognition and can have a significant impact on its results. If the illumination tends to vary, the same individual gets captured with the same sensor and with an almost identical facial expression and pose, the results that emerge may appear quite different.

Illumination changes the face appearance drastically. It has been found that the difference between two same faces with different illuminations is higher than two different faces taken under same illumination.

• Pose

Facial Recognition Systems are highly sensitive to pose variations. The pose of a face varies when the head movement and viewing angle of the person changes. The movements of head or differing POV of a camera can invariably cause changes in face appearance and generate intra‐class variations making automated face recognition rates drop drastically. It becomes a challenge to identify the real face when the rotation angle goes higher. It may result in faulty recognition or no recognition if the database only has the frontal view of the face.

• Occlusion

Occlusion means blockage, and it occurs when one or other parts of the face are blocked and whole face is not available as an input image. Occlusion is considered one of the most critical challenges in face recognition system.

It occurs due to beard, moustache, accessories (goggle, cap, mask, etc.), and it is prevalent in real-world scenario. The presence of such components makes the subject diverse and hence making automated face recognition process a tough nut to crack.

• Expression

Face is one of the most crucial biometrics as its unique features play a crucial role in providing human identity and emotions. Varying situations cause different moods which result in showing various emotions and eventually change in facial expressions.

Different expressions of the same individual are another significant factor that needs to be taken into account. Human expressions are particularly macro-expressions which are happiness, sadness, anger, disgust, fear, surprise. Micro-expressions are the one which shows the rapid facial patterns and happen involuntarily.

Macro and micro expressions find their place on someone’s face due to changes in one’s emotional state and in the wake of such emotions- which are many- the efficient recognition becomes difficult.

•  Ageing

Face appearance/texture changes over a period of time and reflect as ageing, which is yet another challenge in facial recognition system. With the increasing age, the human face features, shapes/lines, and other aspects also change. It is done for visual observation and image retrieval after a long period.For accuracy checking, the dataset for a different age group of people over a period of time is calculated. Here, the recognition process depends on feature extraction, basic features like wrinkles, marks, eyebrows, hairstyles, etc.

LITERATURE REVIEW

Face Recognition is getting increasingly popular and most of us are already using it without even realizing it. Be it a simple Facebook Tag suggestion or Snapchat Filter or an advanced airport security surveillance, Face Recognition has already worked its magic in it. China has started using Face Recognition in schools to monitor student’s attendance and behaviors. Retail Stores have started using Face Recognition to categorize their customers and isolate people with history of fraud. With a lot more changes underway, there is no doubt that this technology would be seen everywhere in the near future. It is important to understand that Face Detection and Face Recognition are two different things. In Face Detection only the Face of a person is detected the software will have no Idea who that Person is. In Face Recognition the software will not only detect the face but will also recognize the person. Now, it should be clear that we need to perform Face Detection before performing Face Recognition. A video feed from a webcam is nothing more than a long sequence of still images being updated one after the other. And each of these images is just a collection of pixels of different values put together in its respective position. So how can a program detect a face from these pixels and further recognize the person in it? There are a lot of algorithms behind it and since we are using the OpenCV library it very simple to perform face Recognition without getting deeper into the concepts.

IOT or Internet of Things is an upcoming technology that allows us to control hardware devices through the internet. Here, we propose to use IOT in order to control home appliances, thus automating modern homes through the internet. This system uses three loads to demonstrate as house lighting and a fan. Our user friendly interface allows a user to easily control these home appliances through the internet.

HARDWARE REQUIREMENT

NodeMCU 

The NodeMCU (Node MicroController Unit) is an open-source software and hardware development environment built around an inexpensive System-on-a-Chip (SoC) called the ESP8266. The ESP8266, designed and manufactured by Espressif Systems, contains the crucial elements of a computer: CPU, RAM, networking (WiFi), and even a modern operating system and SDK. That makes it an excellent choice for the Internet of Things (IoT) projects of all kinds. However, as a chip, the ESP8266 is also hard to access and use. You must solder wires, with the appropriate analog voltage, to its pins for the simplest tasks such as powering it on or sending a keystroke to the “computer” on the chip. You also have to program it in low-level machine instructions that can be interpreted by the chip hardware. This level of integration is not a problem using the ESP8266 as an embedded controller chip in mass-produced electronics. It is a huge burden for hobbyists, hackers, or students who want to experiment with it in their own IoT projects.

 Web Camera

A webcam is a video camera that feeds or streams an image or video in real time to or through a computer network, such as the Internet. Webcams are typically small cameras that sit on a desk, attach to a user’s monitor, or are built into the hardware .In our project, we use webcam for the face recognition purpose.

Processor

A processor (CPU) is the logic circuitry that responds to and processes the basic instructions that drive a computer. The CPU is seen as the main and most crucial integrated circuitry (IC) chip in a computer, as it is responsible for interpreting most of computers commands.

 LED Light(LOAD)

Household lights are used as loads in this project for the demonstration.

SOFTWARE REQUIREMENT

Aurdino IDE

The Arduino Integrated Development Environment (IDE) is a crossplatform application (for Windows, macOS, Linux) that is written in functions from C and C++.It is used to write and upload programs to Arduino compatible boards, but also, with the help of third-party cores, other vendor development boards. The source code for the IDE is released under the GNU General Public License, version 2. The Arduino IDE supports the languages C and C++ using special rules of code structuring. The Arduino IDE supplies a software library from the Wiring project, which provides many common input and output procedures. User-written code only requires two basic functions, for starting the sketch and the main program loop, that are compiled and linked with a program stub main() into an executable cyclic executive program with the GNU toolchain, also included with the IDE distribution. The Arduino IDE employs the program avrdude to convert the executable code into a text file in hexadecimal encoding that is loaded into the Arduino board by a loader program in the board’s firmware. By default, avrdude is used as the uploading tool to flash the user code onto official Arduino boards.

VS Code

Visual Studio Code is a source-code editor that can be used with a variety of programming languages, including Java, JavaScript, Go, Node.js, Python and C++. It is based on the Electron framework, which is used to develop Node.js Web applications that run on the Blink layout engine. Visual Studio Code employs the same editor component (codenamed “Monaco”) used in Azure DevOps (formerly called Visual Studio Online and Visual Studio Team Services).

Instead of a project system, it allows users to open one or more directories, which can then be saved in workspaces for future reuse. This allows it to operate as a language-agnostic code editor for any language. It supports a number of programming languages and a set of features that differs per language. Unwanted files and folders can be excluded from the project tree via the settings. Many Visual Studio Code features are not exposed through menus or the user interface but can be accessed via the command palette.

Visual Studio Code can be extended via extensions, available through a central repository. This includes additions to the editor and language support. A notable feature is the ability to create extensions that add support for new languages, themes, and debuggers, perform static code analysis, and add code linters using the Language Server Protocol.

Visual Studio Code includes multiple extensions for FTP, allowing the software to be used as a free alternative for web development. Code can be synced between the editor and the server, without downloading any extra software.

Visual Studio Code allows users to set the code page in which the active document is saved, the newline character, and the programming language of the active document. This allows it to be used on any platform, in any locale, and for any given programming language.

Web Browser

A web browser is application software for accessing the World Wide Web. When a user requests a web page from a particular website, the web browser retrieves the necessary content from a web server and then displays the page on the user’s device. The purpose of a web browser is to fetch content the web and display it on a user’s device. Virtually all URLs on the web start with either http: or https: which means the browser will retrieve them with the Hypertext Transfer Protocol (HTTP).

FEASIBILITY STUDY

The hardware and software used are highly feasible for the system. The easy availability, simple operation and cheap maintenance of the devices used fulfill the need of the system.

 Technical 

The devices used in this system are easily available and require interfacing with microcontroller. So it is quite fair to say that the devices are technically feasible for the proposed system.

Economical 

The devices used in the system are economic in price and are easily available. This system is very beneficial as it provides best result at low price. The device can be used in public sectors like railway station, department stores, restaurants and many advance networking companies. Also the maintenance of the device is very cheap.

METHODOLOGY

To realize the proposed IOT BASED SMART HOME WITH FACE RECOGNITION SECURITY it consists of Node MCU, LED, Webcam, Server, Processor and Operating System. The whole system here is divided into two main sections i.e. home automation and security system integrated there in the system. For home automation Node MCU is the integral part. The loads of the house are all connected in the Node MCU centrally and for ac system realization relay modules can also be integrated in the system. Here all the devices of the house can be controlled via the web application. Buttons are provided to each electrical appliances individually and the user can control all appliances via those buttons in the web page.

On the other hand, for the security system face recognition system is integrated. Here in this system CPU plays the integral part which is fed with the face recognition program. The webcam integrated with the processor gets the human face detected by the camera in front of the door. If the detected face is un known, then the owner of the house gets notified by the system via the notification system. As the owner gets notified then the owner of the house login to the web interface system and see the activity around through the live streaming in the web browser. From the given system owner can turn on the alert system integrated in the system and also take the necessary required action.

 Block Diagram

 Block Diagram

Algorithm

Algorithm for Face Recognition System

Step 1: Start    

Step 2: Take Samples and store in dataset.

Step 3: Test Samples from data set and make array for specific data.

Step 4: Run Recognition program.

Step 5: Compare the image obtained from webcam and compare it with tested data.

Step 6: If image does not match with the tested data:

If data matches with sample data:

 go to step 5

Step 7: Stop

 Algorithm for Home automation system

Step 1: Start

Step 2: Open Web Page

Step 3: Enter Id and password for login.

Step 4: If id and password matches with the database:

Enter inside the control page.

Control the home appliances using the buttons.

Step 5: Exit the page.

Step 6: Stop

Flowchart

 Flowchart for Face Recognition System

 Flowchart for Home Automation System

 Website Preview 

IMPLEMENTATION AND DESIGN

 Interfacing Node MCU with load

The main aim of our project here is to interface node MCU with the home appliances. Here node MCU is fed with the program to control home appliances. The loads of the home are all centrally connected to the node MCU. As soon as the node MCU is supplied with the power then our system will start. Now the user can control all the devices connected in the system remotely through the web app. In this way all the home appliances can be controlled via the web server.

Code format and Description

The codes written for Arduino are known as sketches. They are written in C++.Every sketch needs two void type functions, setup () and loop (). A void type function doesn’t return any value. The setup () method is ran once at the just after the Arduino is powered up and the loop () method is ran continuously afterwards. The setup () is where we want to do any initialization steps, and in loop () we want to run the code we want to run over and over again. The code was then written in the Arduino IDE in the following syntax: void setup () { } void loop () { } In order to go forward with the programming, we have to choose the library files to be used. In this code we used following library files:

ESP8266WiFi.h: This library provides ESP8266 specific Wi-Fi routines that we are calling to connect to the network. The actual connection to Wi-Fi is initialized by calling: begin (“network-name”, “pass-to-network”)

ESP8266mDNS.h: Start the mDNS server and register the service

WiFi.h: Used to connect to a WiFi network with strongest WiFi signal (RSSI)

Understanding the Sketch

 Face Recognition Program

The folder format of the program is:

Step 1: The program is executed to take the face sample from the web cam. The programmed file to take the face sample is :

To execute this program, we use terminal and command:

After executing this program, the instructed data samples are taken and stored inside data set folder.

Step 2: The samples taken through the Step 1 process is trained using the training

To execute the training program we use terminal

After executing this program, the data samples are stored as an array of number as trainer.yml format

Step 3: After the samples is trained through step 2, we are ready to run the face-recognition program.

To execute this program, we use terminal.

After executing this program, we can now recognize the human face comparing with the trained sample faces

Home Automation Program

Before starting the setup function, we include the header files which is already explained in code format section above. Along with header files, we define the input pins for the loads that are to be connected in the node MCU.

Here we have integrated webpage into the nodeMCU program.

Here we have started and started sending and receiving data through our own created network.

CODING

NODEMCU PROGRAMMING

Conversation opened. 1 read message.

Skip to content
Using Gmail with screen readers
Meet
New meeting
Join a meeting
Hangouts
project video, face recognition portion and Arduino code for IoT
Inbox
Sunil Pokharel <sunilpokhrel.09@gmail.com>
	
AttachmentsMay 30, 2021, 6:27 PM
	
to me
 2021-05-12 17-04-49.mp4

 SUNIL POKHREL 

	

	


3 Attachments
	
	
	

/*    
      
 */     



#ifdef ESP8266
#include <ESP8266WiFi.h>
#include <ESP8266mDNS.h>
#elif defined(ESP32)
#include <WiFi.h>
#include <ESPmDNS.h>
#else
#error "Board not found"
#endif


#define LED1 D1
#define LED2 D2

char webpage[] PROGMEM = R"=====(
<!DOCTYPE html>
<html lang="en">
<head>
    <meta charset="UTF-8">
    <meta http-equiv="X-UA-Compatible" content="IE=edge">
    <meta name="viewport" content="width=device-width, initial-scale=1.0">
    <title>Button control</title>
</head>
<body style="padding: 0; margin: 0; width: 100%; height: 100vh;">
    <h1 style="display: flex; justify-content: center; font-family: 'Segoe UI', Tahoma, Geneva, Verdana, sans-serif;">Controlling the house system remotely</h1>
    <div class="l1" style="padding-bottom: 30px; display: flex; justify-content: center;">
        <button style="width: 100px; height: 30px; background-color: green; color: white; border-radius: 5px; border-style: none; margin-right: 10px; cursor: pointer;" onclick="window.location = 'http://'+location.hostname+'/led1/on'">Light ON </button>
        <button style="width: 100px; height: 30px; background-color: rgb(255, 51, 0); color: white; border-radius: 5px; border-style: none; cursor: pointer;" onclick="window.location = 'http://'+location.hostname+'/led1/off'">Light OFF</button>
    </div>
    <div class="l2" style="display: flex; justify-content: center;">
        <button style="width: 100px; height: 30px; background-color: green; color: white; border-radius: 5px; border-style: none; margin-right: 10px; cursor: pointer;" onclick="window.location = 'http://'+location.hostname+'/led2/on'">Buzzer ON</button>
        <button style="width: 100px; height: 30px; background-color: rgb(255, 51, 0); color: white; border-radius: 5px; border-style: none; cursor: pointer;" onclick="window.location = 'http://'+location.hostname+'/led2/off'">Buzzer OFF</button>
    </div>
</body>
</html>
)=====";

// ipaddress/led1/on
//ipaddress/led1/off

// ipaddress/led2/on
//ipaddress/led2/off
#include <ESPAsyncWebServer.h>

AsyncWebServer server(80); // server port 80

void notFound(AsyncWebServerRequest *request)
{
  request->send(404, "text/plain", "Page Not found");
}

void setup(void)
{

  Serial.begin(74880);
  pinMode(LED1,OUTPUT);
  pinMode(LED2,OUTPUT);

  WiFi.softAP("testwifi", "");
  Serial.println("softap");
  Serial.println("");
  Serial.println(WiFi.softAPIP());


  if (MDNS.begin("ESP")) { //esp.local/
    Serial.println("MDNS responder started");
  }



  server.on("/", [](AsyncWebServerRequest * request)
  { 

  request->send_P(200, "text/html", webpage);
  });

   server.on("/led1/on", HTTP_GET, [](AsyncWebServerRequest * request)
  { 
    digitalWrite(LED1,HIGH);
  request->send_P(200, "text/html", webpage);
  });
  server.on("/led1/off", HTTP_GET, [](AsyncWebServerRequest * request)
  { 
    digitalWrite(LED1,LOW);
  request->send_P(200, "text/html", webpage);
  });

    server.on("/led2/on", HTTP_GET, [](AsyncWebServerRequest * request)
  { 
    digitalWrite(LED2,HIGH);
  request->send_P(200, "text/html", webpage);
  });
  server.on("/led2/off", HTTP_GET, [](AsyncWebServerRequest * request)
  { 
    digitalWrite(LED2,LOW);
  request->send_P(200, "text/html", webpage);
  });

  server.onNotFound(notFound);

  server.begin();  // it will start webserver
}


void loop(void)
{
}

codearduinopart.txt
Displaying codearduinopart.txt.

PYTHON PROGRAMMING

 RESULT AND ANALYSIS

As we have controlled the home appliances via the web application where all the devices in the home appliances can be controlled via the buttons switch in the web page. Here the server connects to the hotspot created by the node MCU and the data are sent and received via port 80.

At the same time for security system face recognition system is implemented in the system. The web camera fitted in the system is in constant surveillance of the home and as soon as some unknown person comes in front of the system it detects and sends notification to the owner of the house. The owner can login to the system and monitor along with that he can turn on the buzzer or take necessary action. In this way home automation and security system using IOT can be implemented.

The error may occur while the user tryto login to the system using wrong credentials. The system may sometimes show error while recognizing the face.

 CONCLUSION

The technology used in our project can be used for any organizatons, colleges, malls, etc. by making slight modification in the program and system. By using more datasets and appropriate algorithm the system can be made more efficient and faster. With greater efficiency, we can control home appliances and security system can be implemented in the home. Time consumption and electrical wastage is reduced. The operating software required is free and easily available and system can be implemented with cheaper cost. This system can be used in various applications for future scope.

PROJECT BY:

Anil Kumar Panta (https://www.facebook.com/anil.panta.10)

Rajan Adhikari

Sudip Poudel

Sunil Pokhrel(https://www.facebook.com/sunil.pokharel.395454)