8 Must-have features for IoT Products
What is ThingsBoard IoT Platform?
ThingsBoard is an open-source IoT platform for monitoring, data processing, visualization of data along with device management.
Why Thingsboard?
It supports industry-standard IoT protocols – MQTT, CoAP, and HTTP. ThingsBoard combines scalability, fault-tolerance, and performance to capture the device data for processing and monitoring. It provides its gateway server that can communicate with the attached devices.
Build around Netty Framework due to which it provides support for many protocols and applications. New Hardware protocols can be added by just adding Inbound and Outbound channel handlers for new protocols with the netty framework.
Thingsboard features
1) Security
Support for provisioning and management of devices with credentials management. Custom security rules for each protocol can be applied.
2) Dashboard and Data Visualization
Built-in support for 100+ widgets components. Collect and visualize data from devices and assets in Dashboard with widgets.
3) Telemetry
Analyze incoming telemetry and trigger alarms with complex event processing. Support event storage Api to capture Telemetry metrics.
4) Rest API and RPC Support
Data workflow can be designed using Rest API and RPC Requests.
5) Push Device Data
Support to push Device data to other systems in real-time.
6) Integration with Different Message Queues
Different connectors are available to connect with different implementations of message queues. The multiple message queue implementations: Kafka, RabbitMQ, AWS SQS, Azure Service Bus, and Google Pub/Sub are supported.
7) Hybrid Database support
Stores all entities in SQL databases and telemetry data in the NoSQL database.
8) Rule Engine
In-built rule engine, to configure rules for device state and different notifications or alarms can be generated based on the telemetry of the device. Data processing rules can be changed at runtime based on the device state.
9) Deployment, Standard and Cluster mode
On-premise and cloud deployments are supported along with Standard and Cluster mode are supported
10) Alarms Management
Create and Manage alarms related to devices, assets, and customers, etc. Rule policy can be applied to raise alarms for device state changes.
Use Cases:
Thingsboard IoT platform supports broad use cases specifically in fields of Fleet tracking, Smart Farming, Smart Metering, Smart Energy, Smart City, and Commercial Home Automation Solution.
1) Smart Farming
Smart farming is a combination of hardware (IoT) and software (SaaS) to capture the data and use data to manage all the operations on the farm, both pre and post-harvest.
2) Smart Metering
Smart Metering solutions are used to study consumption patterns.
Alarms are integrated with smart metering with rules to identify any abnormalities or energy offenders
3) Smart Energy
Smart Energy is syncing different energy producers with energy consumers to optimize energy consumption based on data points.
4) Fleet management
Different types of fleet management solutions can be derived
- Vehicle leasing, financing, maintenance, licensing, tracking, and diagnostics.
- Supply chain management.
- Driver management, Speed management, Fuel management
Workflow :
Conclusion :
Thingsboard is a good choice to develop IoT Solutions. Its source code is available with Apache License.
Why should we use MQTT over HTTP?
MQTT (MQ Telemetry Transport) is a lightweight message protocol that provides telemetry information to clients with network limitations in a simple way. The protocol mainly uses a publish-subscribe communication pattern, which is mainly used for M2M(Machine to machine) communication and plays a crucial role in the Internet of things (IoT). The MQTT protocol is a great choice for wireless networks that may experience different levels of latency due to occasionally unreliable connections or bandwidth constraints.
This protocol was designed by Andy Stanford-Clark and Arlen Nipper in 1999 mainly for connecting sensors on oil pipelines with communications satellites, with an emphasis on minimum bandwidth usage.
Working of MQTT :
In the MQTT protocol, the connected devices are known as “clients,” which communicate with a server known as the “broker.” The broker handles the task of data transmission and communication between clients
Whenever a client wants to distribute particular information,it publishes that information to a particular topic, then the broker sends the information to any clients that have subscribed to that topic. Any client can be a publisher, or a subscriber, or both.
Advantages of MQTT:
- Efficient data transmission
- Quick to implement as it is a lightweight protocol
- Low network usage due to minimization of data packets
- Fast and efficient message delivery
- Uses very small amounts of power, which is good especially for the wireless devices
- Reduction of network bandwidth in communication.
Why MQTT over HTTP?
MQTT over HTTP:
| MQTT | HTTP |
| Fast as the data packet size is small | Slow as it used bigger data packet size |
| The connection is always open between broker and client | Open and close the connection every time for each request |
| Low power consumption | Higher power consumption |
| Mainly useful for sending short messages. | Mainly useful for sending lengthy messages. |
| It provides easy management for large applications. | Not easy to manage for large applications |
MQTT Broker:
It is a server that handles the communication between the clients. A broker can have many clients connected to it, which can publish and subscribe all the time.
Types of MQTT Brokers:
Mosquitto:
Mosquitto is a lightweight MQTT broker written in the C language. It supports TLS and there are plugins for authorization using a database, But it has some disadvantage as:
- It does not support clustering so it is difficult in terms of scalability.
- It cannot take advantage of multi-core CPUs.
RabbitMQ:
It is a very popular message broker which is written in ‘Erlang’ that supports MQTT via a plugin. It supports TLS and clustering is also fine. Authorization cannot be done using a database directly but we can create an HTTP REST wrapper over the database and that can be used for authorization. This broker supports the AMQP protocol natively but, the MQTT implementation is missing some important features such as QoS2.
VerneMQ:
It is a distributed, high-performance MQTT message broker built-in Erlang. VerneMQ can scale horizontally and vertically to support a huge number of concurrent publishers and subscribers. These subscribers can connect to any node and can also receive messages from any other node.Also, as VerneMQ is mainly written in Erlang, it is capable of working on a multi-threaded processor and also can take the benefit of multi-core processors. It is also an active community on GitHub and Docker images are also available for installation.
Eclipse Paho:
This is mainly an umbrella project for several MQTT implementations in different programming languages. It is an open-source MQTT client implementation and is also maintained by a huge community. Paho API comes with support with different languages and platforms like C/C++, Android, Arduino, Python, Java, and Rust. To use Paho in our IoT project we just have to import the appropriate library and begin using various functions provided by the library for our requirements. There are also various GUI tools available which include RCP application, Eclipse plugin, and Java Swing application.
Conclusion:
MQTT Protocol is a lightweight and fast protocol for using especially in IoT. In IoT, the main concern is minimum battery and network bandwidth usage as the devices are wireless and run on battery, which can be achieved efficiently using MQTT.
Are IoT devices vulnerable?
The Internet of Things (IoT) is a collection of devices that are connected to the Internet. Following are the types of ‘things’ connected to the internet:
- Things that collect information and send it on the internet – sensors allow machines to make sense of the world.
- Things that receive information and take action on it. – printers, door locks, electric switches etc
- Things that do both.
What is IoT security?
Any device that’s connected to the Internet can be vulnerable to cyberattacks. IoT security is the act of securing Internet of Things devices and the networks connecting them. With IoT, data is always on the move. It is being transmitted, stored, and processed. IoT device security must protect systems, networks, and data from a large no of IoT security attacks, which target four types of vulnerabilities:
- Communication attacks, the insecure data transmission between IoT devices and servers.
- Lifecycle attacks put the integrity of the IoT device at risk as control shifts to the malware.
- Attacks on the device software.
- Physical attacks, which target the chip in the device or the device physically
Risks
1. The Rise Of Botnets and Ransomware
Increasing the amount of botnets among IoT devices causes a security threat to IoT applications. Hackers can remotely control internet-connected devices and use them for illegal purposes using Botnets. Many organisations do not have solutions to track this.
A ransomware can hack an IoT-enabled camera that captures confidential information from home or the work office. The attackers can encrypt the webcam system and consumers will not be able to access any information.
2. Large number of IoT Devices
Till now the security was for computers and mobile devices, but with IoT a lot many no of devices are connected through on the internet. All these devices are not secure and can be hacked , disabled. They have limited processing capacity, storage and memory which creates a problem for securing these devices. Thus IoT devices mean increased security vulnerabilities across the enterprise, and it is a growing challenge for security professionals.
3. Infrequent Updates for IoT devices
Software updates ensure that computers and mobile devices are as secure as can be. The devices may have outdated hardwares and softwares. Some IoT devices may not have the amount of software updates like other technologies. Some of them may not have even one update.
These products may be secure at the time of purchase but become vulnerable to attacks when the hackers find some bugs or security issues. When these issues are not fixed by releasing regular updates for hardware and software, the devices remain vulnerable to attacks.
4. Lack Of Encryption – Unreliable Communication
Though encryption is used to prevent hackers from accessing data while communicating between devices, it is also one of the leading IoT security challenges. The lack of storage and processing capabilities of devices unlike on a traditional computer increases the possibility of attacks. Unless an enterprise secures the data communication over the internet, the data leak or manipulation pose a threat to the system.
5. Weak Default Passwords
Many IoT devices come with original default weak passwords. It is recommended that you change the passwords regularly, some administrators fail to take this simple step. A weak and default credentials could leave an IoT device vulnerable to an attack. Hackers can get access to devices easily with username and default passwords.
6. Unreliable Threat Detection Methods
When there is a large scale of IoT devices, it becomes difficult to monitor all of them. Since the number of devices is increasing significantly, the number of things to be managed is increasing even more. Hence, many devices keep on operating without the understanding of the users that they have been hacked. Hackers can breach common devices such as printers and cameras.The apps, services, and protocols used by IoT devices should be monitored for security.
7. Phishing Attacks
Phishing is already a security concern across all enterprise technologies, and IoT devices are also vulnerable to these attacks. Hackers could send a signal to an IoT device that triggers exposure of important data. Although it is commonly found security attacks, and it can be stopped by giving education to users.
8. Prediction and prevention of attacks
Hackers are proactively finding out new techniques for security threats. Security professionals need to prevent IoT security breaches before they occur. In such a scenario, there is a need for learning to predict and prevent new threats. A robust management system should be present to monitor activity and provide insights into potential threats ahead of time.
Some Preventive measures
- Implement automatic antivirus updates
- Require strong login credentials
- Deploy end-to-end encryption
- Make sure the device and software updates are installed on time
- Disable the unused features of the devices
- Be proactive and predict the threats and fix it.
Conclusion
We have experts in handling IoT Devices and Securing them. Please feel free to connect with us in case of any issues.
Async Log4j in IoT Environment
Security of IoT environment is the need of time to identify unauthorized access and reconfiguration of IoT devices. IoT networks required logging to analyze the data flow and allow different tools to do forensic analysis of logs in-case of security breaches.
We are now living in a world of IoT devices. Many IoT Platforms are developed in current years to support different IoT devices. Each IoT platform needs a logging mechanism to inform the user about the state of the system in case of high availability.
We have recently moved one of the customer’s ( IoT platform) sync log4j logging to async log4j logging mechanism.
We have decided on the following points for log4j to analyze the behavior of the system along with performance:
1) System throughput performance
2) Configuration of Log4j
3) Support to add logs in different storage e.g Files, Database asynchronously.
4) Support to add for Custom level. Need to add a new log level Audit.
Log4j supports async mode by the in-built async plugin. To enable it we need to add either existing and new appenders/loggers inside that async tag in log4j.xml
We have created Async appenders and added in appenders list inside log4j2.xml
<Async name="asyncAppender" level="info"> <AppenderRef ref="fileAppender"/> </Async>
As an async log4j implementation using lock-free structures, we have seen system performance increased to 25% in async with file appender only.
Testing system running 20 threads simultaneously creating different log level log-events at speed of 400 logs per second. Each log message size is around 4KB.
We have created two async appenders – One for file appender and one for console appender.
As the console is very slow in response, a console logger or appender should be run in an independent async mode without another appender, so that its slowness doesn’t impact other appenders.
No much change needed in the existing configuration of log4j to enable async mode.
We have also added a Custom level “AUDIT” and for enabling custom level in log4j2.xml we have added the following lines in log4j2.
<CustomLevels> <CustomLevel name="AUDIT" intLevel="150" /> </CustomLevels>
Level of the custom level set’s between error and fatal level.
To add custom ( AUDIT) log in code, we have used the following method
LOGGER.log(Level.getLevel(“AUDIT”), “Audit Message {}”, log);
To add error level logs in the database, we have again added async appender along with JDBC appender with a connection pooling system.
It is also running in different async threads.
Log4j documentation provides a detailed explanation of configuration parameters.
As the Log4j community added many improvements in recent years to increase the handling of logs, we have used log4j library version 2.13.1,
Conclusion
By using async mode log4j, we have improved the performance and throughput of an IoT platform.
Mass Notification System Integration (Everbridge)
What is a Mass Notification System?
A Mass Notification System is a platform that protects, alert & sends messages to inform employees and the people when there is an emergency.
Mass Notification System provides a secure, scalable, and reliable emergency management application, by;-
The Right Message
Concise, accurate, timely, and well-directed messages that communicate what to do in response to a threat or emergency.
To the Right People
Directing live or recorded messages to the entire community or to specific floors of a building.
At the Right Time
Efficiently managing people before, during and after a disaster.
Everbridge Mass Notification System:
Everbridge Mass Notification System is a fully-featured Emergency Notifications Software designed to serve SMEs, Enterprises. Everbridge Mass Notification System provides end-to-end solutions.
Everbridge console :
Giving power to users:
- Manage settings, limits, and defaults through a user-friendly admin.
- Define role-based access control to manage users and contacts separately.
- Customize portals to allow contacts to update and manage their accounts.
Rest API:
REST suggests create an object of the data requested by the client and sending the values of the object in response to the user.
REST is an architectural style as well as an approach for communications purposes that are often used in various web services development.
METHODS OF REST API:
When I say CRUD operations, I mean that we create, read, update and delete a resource. For these actions, you can actually use the HTTP methods, which are nothing but the REST API Methods.
Steps to configure Everbridge:
1) How to configure an Everbridge Account:
- Register your company with Everbridge and create a user for API access.
- Once support activates the user account with API access, you’re ready to create the API key.
2) How to create an Everbridge API Key:
- Get the username and password of the API user account
- Encode and decode the username and password using Base64EncoderDecoder.
- Make sure the username and password are case correct.
3) Test the Key:
- Use api.everbridge.net to test your API key.
- Write a username and password into the “credentials” field.
- Write your Organization ID into the “organizationId” field.
- Select HeartBeats time.
- Click on the “ok” button.
- You should get a “successful message”.
API Calls
1) GET Method:
1) Retrieve a specific organization:
EverbridgeBaseUrl+"/organizations/{organizationId}"
2) Retrieve all incident templates for an organization:
EverbridgeBaseUrl+"/incidentTemplates/{organizationId}"
Response:
{
"message": "OK",
"data":
{
"id": 393354571204451,
"name": "Test Template",
"templateStatus": "Live"
}
}
3) Retrieve all Variables for an organization:
EverbridgeBaseUrl+"/incidentVariableItems/{organizationId}/?pageNumber="+pageNumber
2) POST Method:
1) Launch an Incident:
EverbridgeBaseUrl+"/incidents/{organizationId}”
2) Initiate an immediate notification:
EverbridgeBaseUrl+"/notifications/{organizationId}”
Benefits of Everbridge Mass Notification System:
1) Secure, reliable & scalable platform.
2) Send consistent and error-free messages quickly.
3) Communicate and respond more quickly to disruptive events.
4) The user and contact list are managed by Everbridge.
5) Facilitates the creation of email templates and variables which can be used for notifications.
Conclusion:
This article explains one of the most important components i.e. Everbridge Mass Notification System, How it works, Integration with benefits of using an Everbridge MNS for business. For any queries regarding the integration of mass notification system do connect with us.
How to Tune Database Performance?
What does Tuning mean?
Database tuning is the process of optimizing the performance of a database. In the enterprise, this usually means the maintenance of a large database management system (DBMS ) and Avoiding transactions that create bottlenecks or avoiding queries that run for minutes to hours unnecessarily. This includes optimizing the performance of the database as well as the underlying hardware.
How to Tune Database Performance?
- While using the SELECT statement use column names so that it will fetch whatever information is required and avoid using * in your SELECT queries because it would put a load on the system unnecessarily.
- Use column names instead of the DISTINCT keyword to get unique results.
- Avoid typecasting as it will lead to performance downgrade.
- Create joins with INNER JOIN instead of using WHERE condition.
- Use WHERE instead of HAVING to define filters
- Do Your Constraints in Application Code Rather Than the Database
- Create indexes carefully on all the tables where you have frequent search operations. Avoid indexing the tables where you have less number of search operations and more number of insert and update operations.
- Use wildcard search wisely. If you want records starting with string neo you should use “neo%” instead of “%neo%” it will effectively prevent the entire filter from being used.
- To make a database fast and reliable you can use the RAID method in the creation of the database. RAID stands for Redundant Array of Independent Disks. If data are stored on one disk, the entire database is completely reliant on that one disk; if it were to fail, the database would not exist anymore. Another drawback to having it on a single disk is the read/write time. One hard disk can only be so fast. If a lot of I/O data is processed, it can be a time-consuming process. What RAID does is it partitions and replicates the data onto several autonomous disks. It makes reading and writing faster because instead of one disk trying to find all the data, the task is separated into parts, and each hard disk does part of the job.
- Avoid the use of logical operators such as OR in a query if not required as it slows down the query.
- Defragmenting the database can help to increase the performance of the databases.
- For queries that are executed on a regular basis, try to use procedures. A procedure is a potentially large group of SQL statements. Procedures are compiled by the database engine and then executed. Unlike an SQL statement, the database engine need not optimize the procedure before it is executed.
- When performing batch transactions, perform COMMIT after a reasonable number of records creation as opposed to making them after each record creation.
- Use response time analysis: With response time analysis, we can optimize database tuning by identifying root causes and prioritizing actions based on the impact of poor database.
Tuning Tools:
Oracle: tkprof
MySQL : Solarwinds DB Performance Analyzer, QL Power Tools, Idera SQL, Diagnostic Manager.
SQL Server: SQL Server Management Studio, SQL Server.
Conclusion:
While DBAs must consider various territories for performance tuning, execution tuning begins from the database and application configuration from the design stage. Databases and applications that are structured in light of execution tuning point of view are considerably more scalable than applications planned without performance tuning thought.
Machine Learning Algorithms: Beginners Guide Part 2
In Blog Machine Learning Algorithms:Beginners Guide Part 1, we have seen machine learning algorithms and its 3 broad categories. In this section, let’s learn the sub-types of Machine learning algorithms.
A question may arise in your mind, when using a wide variety of machine learning algorithms, “which algorithm should I use?” The answer to the question varies depending on many factors, including:
- The size, quality, and nature of data.
- The available computational time.
- The urgency of the task.
- What you want to do with the data.
For 3 broad categories, there are many subtypes of algorithms. Lets see, Top 10 Machine Learning Algorithms.
A: SUPERVISED LEARNING
1. DECISION TREES:
It is a decision support tool that uses a tree-like graph or model of decisions and their possible consequences, including chance-event outcomes, resource costs, and utility.
A decision tree is the minimum number of yes/no questions that one has to ask, to assess the probability of making a correct decision. It allows approaching the problem in a structured and systematic way to show a logical conclusion.
2. NAIVE BAYES CLASSIFICATION:
This classifier is based on applying Bayes’ theorem with naive (strong) independence assumptions between the features. Following is equation for Naive Bayes:
P(A|B) is posterior probability, P(B|A) is likelihood, P(A) is class prior probability, and P(B) is predictor prior probability.
Some of real world examples are:
- To mark an email as spam or not spam
- Classify a news article about technology, politics, or sports
- Check a piece of text expressing positive emotions, or negative emotions?
- Used for face recognition software.
3. LINEAR REGRESSION:
The representation is a linear equation that combines a specific set of input values (x) the solution to which is the predicted output for that set of input values (y). As such, both the input values (x) and the output value are numeric.
While doing linear regression our objective is to fit a line through the distribution which is nearest to most of the points. Hence reducing the distance (error term) of data points from the fitted line.
4. LOGISTIC REGRESSION:
Logistic regression measures the relationship between the categorical dependent variable and one or more independent variables by estimating probabilities using a logistic function, which is the cumulative logistic distribution.
In general, regressions can be used in real-world applications such as:
- Credit Scoring
- Predicting the revenues of a certain product
5. SUPPORT VECTOR MACHINES:
Support Vector Machine (SVM) is a supervised machine learning algorithm that can be used for both classification or regression challenges. However, it is mostly used in classification problems. In this algorithm, we plot each data item as a point in n-dimensional space (where n is a number of features you have) with the value of each feature being the value of a particular coordinate. Then, we perform classification by finding the hyperplane that differentiates the two classes.
6. ENSEMBLE METHODS:
Ensemble methods is a machine learning technique that combines several base models in order to produce one optimal predictive model.
B: UNSUPERVISED LEARNING
7. CLUSTERING ALGORITHMS:
Clustering is the task of grouping a set of objects such that objects in the same group (cluster) are more similar to each other than to those in other groups.
K-means is a popularly used unsupervised machine learning algorithm for cluster analysis. K-Means is a non-deterministic and iterative method. The algorithm operates on a given data set through a pre-defined number of clusters, k. The output of the K Means algorithm is k clusters with input data partitioned among the clusters.
8. PRINCIPAL COMPONENT ANALYSIS:
Principal Component Analysis (PCA) is used to make data easy to explore and visualize by reducing the number of variables. This is done by capturing the maximum variance in the data into a new coordinate system with axes called ‘principal components. Each component is a linear combination of the original variables and is orthogonal to one another.
9. SINGULAR VALUE DECOMPOSITION:
Singular value decomposition is a method of decomposing a matrix into three other matrices:
Where:
- A is an m × n matrix
- U is an m × n orthogonal matrix
- S is an n × n diagonal matrix
- V is an n × n orthogonal matrix
For a given m * n matrix M, there exists a decomposition such that M = U?V, where U and V are unitary matrices, and ? is a diagonal matrix.
10. INDEPENDENT COMPONENT ANALYSIS:
Independent Component Analysis(ICA) is a much more powerful technique that is capable of finding the underlying factors of sources when these classic methods fail completely. Its applications include digital images, document databases, economic indicators, and psychometric measurements. ICA defines a generative model for the observed multivariate data, which is typically given as a large database of samples.
Last Note: Although there are many other Machine Learning algorithms, these are the most popular ones. If you’re a beginner to Machine Learning, these would be a good starting point to learn.
References :
