Machine Learning with Python
The Future of AI in Natural Language Processing
Why should we use MQTT over HTTP?
MQTT (MQ Telemetry Transport) is a lightweight message protocol that provides telemetry information to clients with network limitations in a simple way. The protocol mainly uses a publish-subscribe communication pattern, which is mainly used for M2M(Machine to machine) communication and plays a crucial role in the Internet of things (IoT). The MQTT protocol is a great choice for wireless networks that may experience different levels of latency due to occasionally unreliable connections or bandwidth constraints.
This protocol was designed by Andy Stanford-Clark and Arlen Nipper in 1999 mainly for connecting sensors on oil pipelines with communications satellites, with an emphasis on minimum bandwidth usage.
Working of MQTT :
In the MQTT protocol, the connected devices are known as “clients,” which communicate with a server known as the “broker.” The broker handles the task of data transmission and communication between clients
Whenever a client wants to distribute particular information,it publishes that information to a particular topic, then the broker sends the information to any clients that have subscribed to that topic. Any client can be a publisher, or a subscriber, or both.
Advantages of MQTT:
- Efficient data transmission
- Quick to implement as it is a lightweight protocol
- Low network usage due to minimization of data packets
- Fast and efficient message delivery
- Uses very small amounts of power, which is good especially for the wireless devices
- Reduction of network bandwidth in communication.
Why MQTT over HTTP?
MQTT over HTTP:
| MQTT | HTTP |
| Fast as the data packet size is small | Slow as it used bigger data packet size |
| The connection is always open between broker and client | Open and close the connection every time for each request |
| Low power consumption | Higher power consumption |
| Mainly useful for sending short messages. | Mainly useful for sending lengthy messages. |
| It provides easy management for large applications. | Not easy to manage for large applications |
MQTT Broker:
It is a server that handles the communication between the clients. A broker can have many clients connected to it, which can publish and subscribe all the time.



Types of MQTT Brokers:
Mosquitto:
Mosquitto is a lightweight MQTT broker written in the C language. It supports TLS and there are plugins for authorization using a database, But it has some disadvantage as:
- It does not support clustering so it is difficult in terms of scalability.
- It cannot take advantage of multi-core CPUs.
RabbitMQ:
It is a very popular message broker which is written in ‘Erlang’ that supports MQTT via a plugin. It supports TLS and clustering is also fine. Authorization cannot be done using a database directly but we can create an HTTP REST wrapper over the database and that can be used for authorization. This broker supports the AMQP protocol natively but, the MQTT implementation is missing some important features such as QoS2.
VerneMQ:
It is a distributed, high-performance MQTT message broker built-in Erlang. VerneMQ can scale horizontally and vertically to support a huge number of concurrent publishers and subscribers. These subscribers can connect to any node and can also receive messages from any other node.Also, as VerneMQ is mainly written in Erlang, it is capable of working on a multi-threaded processor and also can take the benefit of multi-core processors. It is also an active community on GitHub and Docker images are also available for installation.
Eclipse Paho:
This is mainly an umbrella project for several MQTT implementations in different programming languages. It is an open-source MQTT client implementation and is also maintained by a huge community. Paho API comes with support with different languages and platforms like C/C++, Android, Arduino, Python, Java, and Rust. To use Paho in our IoT project we just have to import the appropriate library and begin using various functions provided by the library for our requirements. There are also various GUI tools available which include RCP application, Eclipse plugin, and Java Swing application.
Conclusion:
MQTT Protocol is a lightweight and fast protocol for using especially in IoT. In IoT, the main concern is minimum battery and network bandwidth usage as the devices are wireless and run on battery, which can be achieved efficiently using MQTT.
Performance Optimization Using Caching Mechanism
Let’s try an exercise. Try to remember any memories about your childhood. Now try to remember the same memory again after some time. You shall be able to remember it much quicker the second time. This happens because as soon as you recalled the memory the first time, your brain saved this information in your recent memory.
Caching works exactly the same way. Cachingenables us to store data in memory for quick access. When the data is needed again, applications can fetch the data from the cache instead of fetching it from the source. In addition, caching can also make data available when the data source is unavailable. Therefore, caching is one of the most popular performance optimization techniques which not only saves time but also increases the overall performance of an application or website.
There are two main areas for implementing caching mechanism
1. Application Caching
2. Website caching
Application cache is used to store and fetch web resources that are fetched from the web server in the first request. Getting resources from the cache is more efficient and faster than getting them from the source. This means it can be loaded instantly as no retrieving is needed and the resource will be ready to use.
In application caching, we can use the SQLDependency Class which establishes a relationship between an item stored in an ASP.NET application’s cache object and either a specific SQL Server database table or the results of a SQL query.
We can initialize dependency like:
SqlDependency = new SqlCacheDependency("DB_Name", "Column_name");While using this class there is a possibility of
DatabaseNotEnabledForNotificationException
therefore, we need to implement try and catch blocks with the code to handle exceptions.
The CacheDependency class monitors the dependency relationships so that, when any of the dependencies change, the cached item associated with it will be automatically removed. We can implement this class as:
CacheDependency dep = new CacheDependency(fileName, dt);
cache.Insert("key", "value", dep);
if (dep.HasChanged)
{
Response.Write("The dependency has changed and the item will be removed");
}
2. Website caching means the resources from site sources are fetched in the first request and stored in the cache.
There are two main types of caching mechanisms
1. Client-side Caching
When a user visits a site for the first time, the browser stores items like CSS files and images for a specific time. Browser caching allows these saved items to be immediately available upon the next request, rather than placing a new request which needs to be sent back to the server. It will reduce the number of requests and results in faster web page loading time and improves the user experience. Browser caching is useful for reducing download time for static content like CSS, js, and images.
We can also use Local storage which enables us to store data locally in a browser. This data has no expiration date and will persist even after the browser window is closed. The following are some methods for using local storage:
- setItem() — Add key and value to local storage
- getItem() — Retrieve a value by the key
- removeItem() — Remove an item by key
- clear() — Clear all storage
For example, we can use local storage in Javascript like:
localStorage.setItem("key", value);
localStorage.removeItem(“key”);
localStorage.getItem(“key”)
localStorage.clear()2. Server-Side Caching
This type of caching can serve many visitors from a centralized cache whenever a user visits the website. This helps us to reduce the load on the server and improves web page loading time for users. This type of cache acts on behalf of the web server and serves the users by providing them the required data without hitting app servers and database servers.
Caching is implemented through “Cache-Control” which is an HTTP header directive which is sent out in every instance of the HTTP response header. Although caching is helpful in many ways but, for security reasons, any sensitive data such as history and user account information should never be cached.
ASP.NET provides the following different types of caching :
Output Caching
Rendering a web page may involve some complex operations like accessing the database and initializing complex controls. Output caching allows us to bypass the round trip to the server, by caching data in the memory. It stores a copy of the completely rendered HTML page or part of the page which is sent to the client from the server. When the client requests for this page next time, instead of fetching the page from the server, the cached copy of the page is used to save time.
Syntax for implementation of Output Cache directive is:
<%@ OutputCache=" " Duration="10" VaryByParam="*" %>
The option VaryByParam helps you to specify the variables that would require a new cache entry. Other possible options include VaryByCustom, VaryByHeader and Location.
Data Caching
Data caching means caching data from a data source. When the cache expires fresh data is obtained from the data source and the cache is refilled.
The main aspect of data caching is caching the data source. These controls are derived from the abstract class DataSourceControl and have the following properties for implementing caching:
- CacheDuration – Sets the number of seconds for which caching will be done
- CacheExpirationPolicy – Defines the cache behavior when the cache has expired.
- EnableCaching – Specifies whether to cache data or not.
Object Caching
Object caching is to cache the objects on a page, such as data-bound controls. The cached data is stored in the server memory. It provides more flexibility than other caching techniques. The object can be of any type – a data type, a web control, a class or database object and any item can be added to the cache by assigning a key name like:
Cache["key"] = item;
REDIS Cache
Redis is an open-source NoSQL, high-speed memory data structure store. It is quick and it resides entirely in the memory with very small performance overhead. It is free for both commercial and non-commercial use under the BSD license.
It maintains data structures such as Sets, hashes, lists, strings with range queries, bitmaps, and geospatial indexes. It also has built-in support for replication, transactions, and also data persistence.
Redis allows the user to store a huge amount of data without the limitation of relational databases. Redis is written in ANSI C.
Redis is a reliable choice if your application requires to store and recover a huge amount of data and if the application is hosted on multiple servers.
The following are the benefits:
- High performance and application caching
- Simple Usage
- Very easy configuration with any application
- Supports lists of data types(hashes, lists, sets, etc)
- It provides Web UI for viewing web content
In order to install Redis, we need Cloud or Dedicated Server. The hardware requirements include 4 core CPU,16GB RAM, 1GBps Network Connection, and 2 storage devices.
Integration of Redis cache using C#.Net
Step 1: We need to download NuGet package StackExchange.Redis which will be our connector to Redis cache
public class MyRedisConnectorHelper
{
static class Redis
{
static string redisHost = "127.0.0.1";
static int redisPort = 6397;
static string redisPassword = "admin123";
static string redisConnection = string.Format("{0}:{1},password={2}",redisHost, redisPort, redisPassword);
static string connectionString;
connectionString=redisConnection,syncTimeout=30000,connectTimeout=30000,ssl=False,abortConnect=False, allowAdmin=true";
static Lazy multiplexer = CreateMultiplexer();
public static ConnectionMultiplexer Connection
{
get
{
multiplexer.Value.PreserveAsyncOrder = false;
return multiplexer.Value;
}
}
private static Lazy CreateMultiplexer()
{
return new Lazy(() => ConnectionMultiplexer.Connect
(connectionString));
}
}
}
Step 2: We can use the Redis Cache Manager as:
public class RedisCacheManager : ICacheManager, IDisposable
{
private static IDatabase db;
private readonly RetryPolicy _retryPolicy;
public RedisCacheManager(ILogger logger)
{
db = Redis.Connection.GetDatabase();
var retryStrategy = new FixedInterval(4, TimeSpan.FromSeconds(3));
_retryPolicy = new RetryPolicy(retryStrategy);
}
public T Get(string key, ref bool hasValue)
{
var serializedItem = _retryPolicy.ExecuteAction(() => db.StringGet(key));
var item = JsonConvert.DeserializeObject(serializedItem);
if (item == null)
return default(T);
return item;
}
public void Set(string key, object data, int cacheTime)
{
var expiresIn = TimeSpan.FromMinutes(cacheTime);
var serializedItem = JsonConvert.SerializeObject(data);
_retryPolicy.ExecuteAction(() => db.StringSet(key,serializedItem,expiresIn));
}
}
Another caching option we can use is Memcached. It could be preferable when caching small and static data, such as HTML code snippets. Its internal memory management is more efficient in the simplest use cases because it consumes less memory for metadata. Strings are ideal for storing data that is only read, because strings require no processing. String is the only data type supported by Memcached.
Advantages of Redis over Memcached
1. Data Types Supported:
Memcached supports data records of the simple key-value structure. The data types supported by Redis include String, List, Hash, Set, and SortedSet.
2. Memory Management Scheme
The unique thing about Redis is that all of the data storage is not in memory. Redis may swap out values that are no longer in use. These values are then made persistent into the disk and are erased from the memory. This feature of Redis can maintain much more data than system memory capacity. Memcached has a high memory management efficiency but its major defect is that it causes space waste.
3. Support of data persistence
Redis supports memory data persistence, whereas Memcached does not support the data persistence operation.
Redis with Java application:
To use Redis with Java we need to download the jar named jedis.jar and include it into the classpath.
We also need Jedis which is a client library in Java for implementing a Redis data store. It is small, fast, and fully compatible with Redis 2.8.x and above.
Now we will see how we can use Redis in-memory data store and Jedis client library in a Java application :
Step 1: We need to add maven dependency for Jedis client library like:
<dependency>
<groupId>redis.clients</groupId>
<artifactId>jedis</artifactId>
<version>3.2.0</version>
<type>jar</type>
</dependency>
Step 2: We can write a simple Java application to connect to a Redis server like:
package com.eduonix.redis.example
import redis.clients.jedis.Jedis;
public class RedisJavaExample {
public static void main(String[] args) {
//Connect to Redis server using localhost
Jedis jedis = new Jedis("localhost");
System.out.println("Connection is successful");
System.out.println("Getting response: " + jedis.ping());
}
}Step 3. Now we will store a value in Redis cache as:
package com.eduonix.redis.example
import redis.clients.jedis.Jedis;
public class RedisCacheExample {
public static void main(String[] args) {
Jedis jedis = new Jedis("localhost");
System.out.println("Connection successful");
jedis.set("Test", "Test data 1");
//Retrieve the value from the cache
System.out.println("Stored value: "+ jedis.get("Test"));
}
}
String is the most common value stored in Redis cache and it is stored as a key-value pair.
Conclusion:
We all know that performance is a key factor when it comes to websites for providing the best experience to end-users. Caching helps with the same by acting as a mediator between server and end-user by providing data on demand quickly. There are various caching mechanisms that can be used to speed up websites or applications. Although caching may seem like an afterthought for small applications, it is very crucial for complex applications.
Implementation of Advance Logging Mechanism
The purpose of logging is to track the errors in any application, in a centralized manner. Logging should not be used only in big applications but also in smaller applications, especially if they play a crucial role with respect to the overall system. Log files can show any sort of event within an application or system, such as a failure, error, or a state transformation. When something inevitably goes wrong, such changes in state help indicate which transformation actually caused an error in the system.
Logging is mainly developed and used by system administrators on the operational level, for getting a high-level view of the complete system. A perfect log file should not be noisy and it should not contain distracting information. Log files should log only what is absolutely necessary and beneficial for error tracing.
Now, a common question that comes to every developer’s mind: “What exactly should I be logging?”
This depends very much on the nature of the application which is being developed. Logging everything can be helpful as you have a huge amount of data to analyze when you have an error in the system. But it is not at all helpful if it needs to be inspected by a single person.
Instead, developers should be thinking about, what we can call as 3 R’s of logging:
Logging the Right events in the Right Format for the Right user.
Some of the best practices for logging are:
- Add a logging framework before you write the code.
- The logging framework must support logging level modifications without restarting the application.
- Logging should not be an afterthought.
- We should avoid temporary log statements during development and also avoid ‘console.writeline’ statements.For example,

If we write logger function in such a way, then, in case if server or DB fails, then there will be an exception and logging will not be successful.
- Write meaningful log messages and also add context to those messages.
- Use timestamps and logtype for every logged event.
Common Logging Levels :
Here are some common logging levels, listed in order from least severe to most severe. When logging from your application, follow these levels:
- TRACE
At this level, you are basically looking to capture every possible detail about the application’s behavior. This is likely to consume huge resources in production.
- DEBUG
From this level, you’re getting into “noisy” territory and formatting more information than you’d want in normal production situations.
- INFO
This level corresponds to normal application behavior. You need not care too much about these entries during normal operations, but they generally provide the skeleton of what happened.
- WARN
This log level indicates that you might have a problem or an unusual situation is detected
- ERROR
An error is a serious issue and it represents the failure of something important in your application. In this, the application itself is not failing but, this will require attention probably sooner than later, but the application can still keep running.
- FATAL
Fatal represents catastrophic situations in your application. Your application may stop to prevent some kind of serious problem, if possible.
There are many tools available in the market which any developer can use according to his/her requirement for a particular application. Some of these tools are:
- Log4j ( Mainly used for Java )
- Log4j2 ( Mainly used for Java )
Logging in Java requires using one of the available logging frameworks. These frameworks provide the methods and configuration necessary to create and send log messages. Java provides a built-in framework known as java.util.logging package.
Abstraction layers such as SLF4J decouple the logging framework from your application, allowing us to change logging frameworks on demand. The abstraction layer mainly provides a generic API and determines which logging framework to bind to at runtime in an application.
The Java logging API mainly consists of 3 core components:
- Loggers are responsible for capturing events and passing them to appropriate Appenders.
- Appenders are responsible for recording log events to a destination file.
- Layouts are responsible for formatting the data in a log.
Logging frameworks are mainly configured through configuration files. These files are loaded by the logging framework at runtime. Although we can configure logging frameworks by code, using a configuration file is the preferred as it centralizes all configuration settings in a single location.
Loggers are objects that are responsible to trigger log events. Loggers are created in the code of a Java application, and they generate events before passing them to an Appender. A class can have multiple independent Loggers responding to different events.Loggers provide several methods for recording log events. But, before you log an event, you need to assign a level to the event. Log levels determine the severity of the log.

Appenders forward logs from Loggers to an output directory or file. The log messages are formatted using a Layout before being sent to the destination.
Types of Appenders:
1. ConsoleAppender:
It is one of the most common Appenders, which simply displays log entries on the console. The ConsoleAppender is used as the default Appender and comes with basic settings. Sample Log4j2 configuration:

2. FileAppenders:
FileAppenders help to write log entries into a file. They are responsible for opening and closing files along with appending entries to files and also locking files to prevent data corruption. Sample Log4j2 configuration:

3. SyslogAppender:
SyslogAppenders send log entries to a Syslog server. It is a service that runs on a device and collects logs from the device’s operating system and various processes and services. Sample Log4j2 configuration:

If you are running an application in production, FileAppender or SyslogAppender will let you read old logs from a file.
Layouts convert the details of a log entry from one data type to another. Logging frameworks provide Layouts for plain text, HTML, syslog, XML, JSON and serialized.Layouts are generally configured using a configuration file, although starting with Java 7, SimpleFormatters can be configured using a system property.
To use a different Layout with java.util.logging,you can set the Appender’s formatter property to Layout of your choice.
JSON format for logging:
JSON is quite portable and versatile, with most modern languages supporting it by default or through libraries. Its portability makes it ideal for log entries. Using JSON, a log can be read by many JSON interpreters. As the data is already structured, parsing a JSON log is easier than parsing a plain text log. This makes it easier to filter and analyze your logs.
By using Log4j, you can embed a JSON string into a JSON Layout by setting the JSONLayout’s ‘objectMessageAsJsonObject’ attribute to true in the XML file:

ThreadContext creates logs by adding unique data stamps to individual log entries. These stamps are managed on a per-thread basis and last till the thread is destroyed.This can be useful when you want to trace specific requests, transactions, or users through a very complex system.
NDC, or Nested Diagnostic Context, is based on the idea of stack. Information can be pushed onto and popped from the stack. The values in the stack can then be accessed by a Logger without having to explicitly pass a value to the logging method.
The following example uses Log4j and NDC and to associate a userName with a log entry.
ThreadContext is a static class, so we can access methods without initializing an object.
We can store the current value of userName and sessionId of the user, in the stack using ThreadContext.push(userName) and ThreadContext.push(sessionId).
ThreadContext.pop() removes individual items from the stack .
ThreadContext.removeStack() allows Java to reclaim the memory used by the stack, thus preventing memory leaks.
For example,
import org.apache.logging.Log4j.ThreadContext;
String userName = “ABC”;
String password = “12345”;
ThreadContext.push(userName);
ThreadContext.push(sessionId);
logger.info(“Login successful for user”);
ThreadContext.pop();
ThreadContext.remove();
Log4j can extract values from the NDC through the ’%x’ conversion character by using PatternLayout class
<PatternLayout pattern=”%x %-5p – %m%n” />
The code above gives the output as:
“ABC 12345 INFO – Login successful for user”
Markers allow us to mark individual log entries with unique data tokens. Markers can be used to group together entries and trigger actions. You can also combine Markers with ThreadContext in orderto improve the ability to search and filter log data.
For example, if we have a class that connects to a server.Suppose an exception occurs during communication to the server, we can log the exception as a fatal error.
import org.apache.logging.Log4j.Marker;
import org.apache.logging.Log4j.MarkerManager;
//Some Code
final static Marker SERVER_ERROR = MarkerManager.getMarker(“SERVER_ERROR”);
//Some Code
logger.fatal(SERVER_ERROR , “An exception occurred at server side.”);
To display marker in the log output, add the ‘%marker’ pattern to your PatternLayout method:
<PatternLayout pattern=”%p %marker: %m%n” />
The code above gives the output as:
[FATAL] DATABASE_ERROR: An exception occurred.
- Serilog ( Mainly used for .NET )
- NLog ( Mainly developed for .NET )
NLog is one of the best-performing and popular logging frameworks for .NET. Setting up NLog is quite simple. Developers can edit the ‘NLog.config’ file to set up intended targets and configure a target by updating the ‘NLog.config’ .When we modify ‘NLog.config’ file no code change or recompile is required
Here’s an example of how we can log using NLog:

.NET Core also supports a logging API that works with a variety of built-in and third-party logging providers known as ILogger. The responsibility of the ILogger interface is to display a log message in the specified log level. This interface mainly aggregates logging patterns to a single method.

The ILogger interface includes methods for logging to the file or DB. There are many extension methods that are available for logging.
public interface ILogger
{
void Log<TState>(LogLevel logLevel, EventId eventId, TState state); bool IsEnabled(LogLevel logLevel);
IDisposable BeginScope<TState>(TState state);
}
The ‘ILoggerProvider’ manages and creates a logger, specified by the logging category.
public interface ILoggerProvider : IDisposable
{
ILogger CreateLogger(string categoryName);
}
With the help of ‘ILogger’ interface we can write a log to a file or DB. We can plug third-party tools like Serilog or Nlog for logging the logs to a file.
Conclusion:
Logging data for any application starts with choosing what to log and how to log the events in an application. The additional time spent in creating a good logging structure pays off when an error or unwanted event occurs at run-time in an application. Always keep in mind that inefficient and noisy logs can sometimes be accepted, but a good log structure and log data not only save time but also helps us to create better solutions for any problems in the application.


















