Have Any Questions?

+1 781-640-0588

Neova Tech Solutions Logo
Neova Tech Solutions Logo

July, 2020

Apache-kafta-1.2-min
July 20, 2020

Can Kafka be used for Video Streaming?

Kafka was developed around 2010 at LinkedIn by a team that included Jay Kreps, Jun Rao, and Neha Narkhede. Apache Kafka is a distributed publish-subscribe messaging system in which multiple producers send data to the Kafka cluster and which in turn serves them to consumers. In the publish-subscribe model, message producers are called publishers, and one who consumes messages is called as subscribers. Kafka has a robust queue that handles a high volume of data and passes data from one point to another. Kafka prevents data loss by persisting messages on disk and replicating data in the cluster.

Kafka Architecture:

Topic: A stream of messages of a particular type is called a topic.

Producer: A Producer is a source of data for the Kafka cluster. It will publish messages to one or more Kafka topics. 

Consumer: A Consumer consumes records from the Kafka cluster. Multiple consumers consume or read messages from topics parallelly. 

Brokers: Kafka cluster may contain multiple brokers. A broker acts as a bridge between producers and consumers. A Kafka cluster may contain 10, 100, or 1,000 brokers if needed. Each Kafka broker has a unique identifier number.

Record: Messages Sent to the Kafka are in the form of records. It is a key-value pair.

ZooKeeper: It is used to track the status of Kafka cluster nodes. It also maintains information about Kafka topics, partitions, etc.

Kafka Cluster: A Kafka cluster is a system that comprises different brokers, topics, and their respective partitions. Data is written to the topic within the cluster and read by the cluster itself.

Who uses Kafka ?

A lot of companies adopted Kafka over the last few years. I will list some of the companies that use Kafka.

1) Netflix

Netflix uses Kafka clusters together with Apache Flink for distributed video streaming processing.

2) Pinterest

Pinterest uses Kafka to handle critical events like impressions, clicks, close-ups, and repins. According to Kafka summit 2018, Pinterest has more than  2,000 brokers running on Amazon Web Services, which transports about 800 billion messages and more than 1.2 petabytes per day, and handles more than 15 million messages per second during the peak hours.

3) Uber

Uber requires a lot of real-time processing. Uber collects event data from the rider and driver apps. Then they provide this data for processing to downstream consumers via Kafka.

4) LinkedIn

Apache Kafka originates at LinkedIn. Linked uses Kafka for monitoring, tracking, and user activity tracking, newsfeed, and stream data.

5) Swiftkey

Swiftkey uses Kafka for analytics event processing.

Apart from the above-listed companies, many companies like Adidas, Line, The New York Times, Agoda, Airbnb, Oracle, Paypal, etc use Kafka.

Why can Apache Kafka be used for video streaming?

  1. High throughput – Kafka handles large volume and high-velocity data with very little hardware. It also supports message throughput of thousands of messages per second.
  2. Low Latency – Kafka handles messages with very low latency in the range of milliseconds.
  3. Scalability – As Kafka is a distributed messaging system that scales up easily without any downtime. Kafka handles terabytes of data without any overhead. It can scale up to handling trillions of messages per day.
  4. Durability – As Kafka persists messages on disks this makes Kafka a highly durable messaging system. Also one of another reason for durability is message replication due to which messages are never lost.

Other reasons to consider Kafka for video streaming are reliability, fault tolerance, high concurrency, batch handling, real-time handling, etc.

Neova has expertise in message broker services and can help build micro-services based distributed applications that can leverage the power of a system like Kafka.

References :

  1. https://kafka.apache.org/powered-by
  2. https://kafka.apache.org/documentation/
  3. https://blog.softwaremill.com/who-and-why-uses-apache-kafka-10fd8c781f4d

July 20, 2020
0
IOT
July 7, 2020

8 Must-have features for IoT Products

As we step into 2021, businesses have to be extra vigilant about protecting their and
Read More
July 7, 2020
0
Thingsboard-IoT-min
July 5, 2020

What is ThingsBoard IoT Platform?

ThingsBoard is an open-source IoT platform for monitoring, data processing, visualization of data along with device management.

Why Thingsboard?

It supports industry-standard IoT protocols – MQTT, CoAP, and HTTP.  ThingsBoard combines scalability, fault-tolerance, and performance to capture the device data for processing and monitoring. It provides its gateway server that can communicate with the attached devices. 

Build around Netty Framework due to which it provides support for many protocols and applications. New Hardware protocols can be added by just adding Inbound and Outbound channel handlers for new protocols with the netty framework.  

Thingsboard features

1) Security  

Support for provisioning and management of devices with credentials management. Custom security rules for each protocol can be applied.

2) Dashboard and Data Visualization

Built-in support for 100+ widgets components. Collect and visualize data from devices and assets in Dashboard with widgets.

3) Telemetry 

Analyze incoming telemetry and trigger alarms with complex event processing. Support event storage Api to capture Telemetry metrics.  

4) Rest API and RPC Support

Data workflow can be designed using Rest API and RPC Requests.

5) Push Device Data

Support to push Device data to other systems in real-time. 

6) Integration with Different Message Queues

Different connectors are available to connect with different implementations of message queues. The multiple message queue implementations: Kafka, RabbitMQ, AWS SQS, Azure Service Bus, and Google Pub/Sub are supported.

7) Hybrid Database support

Stores all entities in SQL databases and telemetry data in the NoSQL database. 

8) Rule Engine 

In-built rule engine, to configure rules for device state and different notifications or alarms can be generated based on the telemetry of the device. Data processing rules can be changed at runtime based on the device state.

9) Deployment, Standard and Cluster mode

On-premise and cloud deployments are supported along with Standard and Cluster mode are supported

10) Alarms Management

Create and Manage alarms related to devices, assets, and customers, etc. Rule policy can be applied to raise alarms for device state changes.

Use Cases:

Thingsboard IoT platform supports broad use cases specifically in fields of Fleet tracking, Smart Farming, Smart Metering, Smart Energy, Smart City, and Commercial Home Automation Solution. 

1) Smart Farming

Smart farming is a combination of hardware (IoT) and software (SaaS) to capture the data and use data to manage all the operations on the farm, both pre and post-harvest. 

2) Smart Metering

Smart Metering solutions are used to study consumption patterns. 

Alarms are integrated with smart metering with rules to identify any abnormalities or energy offenders

3) Smart Energy

Smart Energy is syncing different energy producers with energy consumers to optimize energy consumption based on data points.

4) Fleet management

Different types of fleet management solutions can be derived

  1. Vehicle leasing, financing, maintenance, licensing, tracking, and diagnostics. 
  2. Supply chain management.
  3. Driver management, Speed management, Fuel management

Workflow :

Conclusion :

Thingsboard is a good choice to develop IoT Solutions. Its source code is available with Apache License. 

July 5, 2020
0
MQTT over HTTP
July 3, 2020

Why should we use MQTT over HTTP?

MQTT (MQ Telemetry Transport) is a lightweight message protocol that provides telemetry information to clients with network limitations in a simple way. The protocol mainly uses a publish-subscribe communication pattern, which is mainly used for M2M(Machine to machine) communication and plays a crucial role in the Internet of things (IoT). The MQTT protocol is a great choice for wireless networks that may experience different levels of latency due to occasionally unreliable connections or bandwidth constraints.

This protocol was designed by Andy Stanford-Clark and Arlen Nipper in 1999 mainly for connecting sensors on oil pipelines with communications satellites, with an emphasis on minimum bandwidth usage.

Working of MQTT :

In the MQTT protocol, the connected devices are known as “clients,” which communicate with a server known as the “broker.” The broker handles the task of data transmission and communication between clients 

Whenever a client wants to distribute particular information,it publishes that information to a particular topic, then the broker sends the information to any clients that have subscribed to that topic. Any client can be a publisher, or a subscriber, or both.

Advantages of MQTT:

  1. Efficient data transmission 
  2. Quick to implement as it is a lightweight protocol
  3. Low network usage due to minimization of data packets
  4. Fast and efficient message delivery
  5. Uses very small amounts of power, which is good especially for the wireless devices
  6. Reduction of network bandwidth in communication.

Why MQTT over HTTP?

MQTT over HTTP:

MQTTHTTP
Fast as the data packet size is smallSlow as it used bigger data packet size
The connection is always open between broker and clientOpen and close the connection every time for each request
Low power consumptionHigher power consumption
Mainly useful for sending short messages.Mainly useful for sending lengthy messages.
It provides easy management for large applications.Not easy to manage for large applications

MQTT Broker:

It is a server that handles the communication between the clients. A broker can have many clients connected to it, which can publish and subscribe all the time.

Types of MQTT Brokers:

Mosquitto:

Mosquitto is a lightweight MQTT broker written in the C language. It supports TLS and there are plugins for authorization using a database, But it has some disadvantage as:

  1. It does not support clustering so it is difficult in terms of scalability.
  2. It cannot take advantage of multi-core CPUs.

RabbitMQ:

It is a very popular message broker which is written in ‘Erlang’ that supports MQTT via a plugin. It supports TLS and clustering is also fine. Authorization cannot be done using a database directly but we can create an HTTP REST wrapper over the database and that can be used for authorization. This broker supports the AMQP protocol natively but, the MQTT implementation is missing some important features such as QoS2.

VerneMQ:

It is a distributed, high-performance MQTT message broker built-in Erlang. VerneMQ can scale horizontally and vertically to support a huge number of concurrent publishers and subscribers. These subscribers can connect to any node and can also receive messages from any other node.Also, as VerneMQ is mainly written in Erlang, it is capable of working on a multi-threaded processor and also can take the benefit of multi-core processors. It is also an active community on GitHub and Docker images are also available for installation.

Eclipse Paho:

This is mainly an umbrella project for several MQTT implementations in different programming languages. It is an open-source MQTT client implementation and is also maintained by a huge community. Paho API comes with support with different languages and platforms like C/C++, Android, Arduino, Python, Java, and Rust. To use Paho in our IoT project we just have to import the appropriate library and begin using various functions provided by the library for our requirements. There are also various GUI tools available which include RCP application, Eclipse plugin, and Java Swing application.

Conclusion:

MQTT Protocol is a lightweight and fast protocol for using especially in IoT. In IoT, the main concern is minimum battery and network bandwidth usage as the devices are wireless and run on battery, which can be achieved efficiently using MQTT.

July 3, 2020
0
Atmosphere Framework
July 1, 2020

Atmosphere Framework: A complete walk through

Atmosphere:

The Atmosphere Framework contains client and server-side components for building Asynchronous Web Applications. Realtime client-server framework for the JVM, supporting WebSocket and cross-browser fallbacks support. The Atmosphere Framework supports all major Browsers and Servers. The Atmosphere framework is the most popular asynchronous application development framework for enterprise Java.
Atmosphere’s Java Client is called wAsync.

Why to choose Atmosphere Framework?

Based on the requirement we need a framework that supports bi-directional communication so, we have selected Atmosphere Framework. The main advantage is that it provides two-way communication between client and server over a single TCP connection. Generally, a client will send a request to the server then the server returns the response. For every request from the same client to the same server, a new connection needs to be opened. WebSocket will maintain a single connection between the client and the server. It also keeps the connection alive with all it’s clients until they disconnect.

Tomcat Configuration Steps:

Used runtime – native as atmosphere dependency for maven tool and tomcat -v9 as a server

Step 1: Download and Install Tomcat.

Step 2: Create an Environment Variable JAVA_HOME.

Step 3: Configure the Tomcat Server.

Step 4: Start Tomcat Server.

Step 5: Develop and Deploy an App.

Maven Tool:

A maven is a build tool that provides different dependencies. Here dependency means the external libraries required to build a project.

It does the following:

  1. Generates source code (if auto-generated code is used).
  2. Generates documentation from source code.
  3. Compiles source code.
  4. Packages compiled code into JAR or ZIP file.
  5. Installs the packaged code in a local repository, server repository, or central repository.

How to Broadcast Message from JAVA server to JAVA Client in Atmosphere Framework:

  1. Used Atmosphere Framework
  2. Used runtime-native as atmosphere dependency for maven tool
  3. Tomcat-v9 as a server

Long polling :

 Long polling is the simplest way of having a persistent connection with a server that doesn’t use any specific protocol like WebSocket or Server Side Events. Easy to implement, and delivers messages without delays.

The Flow:

  1. A request is sent to the server.
  2. The server doesn’t close the connection until it has a message to send.
  3. When a message appears – the server responds to the request with it.

Server:

Broadcasting the messages to clients-

BroadcasterFactory.getDefault().lookup(“URL to broadcast”, true).scheduleFixedBroadcast(message, 2, TimeUnit.SECONDS);

Client:

We have used the AsyncHttpProvider library to establish a connection with the Server in async mode.

AtmosphereRequest request = atmosphereResource.getRequest();

String IncomingMessage = request.getReader().readLine();

To use Atmosphere, add the following dependency:

 1.

<dependency>
<groupId>org.atmosphere</groupId>
<artifactId>atmosphere-runtime</artifactId>
<version>2.4.21</version>
</dependency

 2.

<dependency>
<groupId>org.atmosphere</groupId>
<artifactId>atmosphere-spring</artifactId>
<version>2.4.3</version>
</dependency>

Benefits of Atmosphere:

  1. Provide High Availability.
  2. Scalability.
  3. Fault Tolerance.

Conclusion:

The Atmosphere framework makes the development easier to build applications. The Atmosphere Framework is portable and can be deployed to any web server.

July 1, 2020
0

Headquarters

4701 Patrick Henry Drive, Bldg.16, Suite 106, Santa Clara, California 95054

+1 781-640-0588sales@neovatechsolutions.com

Development Center

P3-603, Pentagon Tower, Magarpatta City, Hadapsar, Pune, MH 411028

+91 960-702-3233jobs@neovasolutions.com

COPYRIGHT 2022 © NEOVA TECH SOLUTIONS INC.

Privacy Policy