Tuesday, August 11, 2015

MongoDB restore

To use Mongo DB mongorestore command:

put db dump file in following hierarchy:
dump/<db-name>/<dump-files>

Then go parent of dump folder and type:
mongorestore


It will restore all backup in respective db, depends on db-name where dump files are kept.

Monday, May 18, 2015

Rest Http Status Code, Header (Request-Response)

HTTP codes used in Rest:
 
Http Response Code HTTP Method Response Body Contents Response code Description
200 GET, PUT, DELETE Resource No error, operation successful.
201 Created POST Resource that was created Successful creation of a resource.
202 Accepted POST, PUT, DELETE N/A The request was received.
204 No Content GET, PUT, DELETE N/A The request was processed successfully, but no response body is needed.
301 Moved Permanently GET XHTML with link Resource has moved.
303 See Other GET XHTML with link Redirection.
304 Not Modified conditional GET N/A Resource has not been modified.
400 Bad Request GET, POST, PUT, DELETE Error Message Malformed syntax or a bad query.
401 Unauthorized GET, POST, PUT, DELETE Error Message Action requires user authentication.
403 Forbidden GET, POST, PUT, DELETE Error Message Authentication failure or invalid Application ID.
404 Not Found GET, POST, PUT, DELETE Error Message Resource not found.
405 Not Allowed GET, POST, PUT, DELETE Error Message Method not allowed on resource.
406 Not Acceptable GET Error Message Requested representation not available for the resource.
408 Request Timeout GET, POST Error Message Request has timed out.
409 Resource Conflict PUT, PUT, DELETE Error Message State of the resource doesn't permit request.
410 Gone GET, PUT Error Message The URI used to refer to a resource.
411 Length Required POST, PUT Error Message The server needs to know the size of the entity body and it should be specified in the Content Length header.
412 Precondition failed GET Error Message Operation not completed because preconditions were not met.
413 Request Entity Too Large POST, PUT Error Message The representation was too large for the server to handle.
414 Request URI too long POST, PUT Error Message The URI has more than 2k characters.
415 Unsupported Type POST, PUT Error Message Representation not supported for the resource.
416 Requested Range Not Satisfiable GET Error Message Requested range not satisfiable.
500 Server Error GET, POST, PUT Error Message Internal server error.
501 Not Implemented POST, PUT, DELETE Error Message Requested HTTP operation not supported.
502 Bad Gateway GET, POST, PUT, DELETE Error Message Backend service failure (data store failure).
505 GET Error Message HTTP version not supported.

Request header in Rest:
 
Request Header Description
Accept Request a particular content type. Valid values: application/json, application/xml.
Accept-Encoding Request content encoding such as gzip.
Accept-Language Request a particular language. Supported where localized strings are generated, such as Updates event descriptions. Affects sort order. The lang query parameter will override this header.
Authorization Include appropriate authorization token.
Host Formats the host portion of embedded links to other social data.
If-Modified-Since Conditional GET.
If-Unmodified-Since Conditional GET.

Response header in Rest:

Response Header Description
Content-Encoding Included if content has been encoded (gzip).
Content-Language Language selected.
Content-Length. Size of response, in bytes, unless content or transfer encoding has been applied.
Content-Type Type of response.
Last-Modified The last time the resource data was modified.
Location URI of a newly created resources.
Transfer-Encoding The form of encoding used to transfer the entity to the user.


Sources: google,yahoo

Wednesday, April 15, 2015

Hadoop Terminology

  • Hadoop Ecosystem: It consists of Hadoop core project(HDFS, MapReduce) and other open source projects (Hive, Pig, Imapala, Sqoop etc) working on top of Hadoop to help in data analysis
  • PIG: High level language to  analyses large data set, internal it uses MR
  • Hive: Offers a SQL like language on top of MR
  • Imapala: develops as a way to query data in hadoop like SQL with out using MR, with low latency compare to Hive.
  • Sqoop: To migrate data from RDBMS to Hadoop
  • Flume:  To move data from external resource (logs etc) to Hadoop
  • Hbase: real time db on hdfs
  • Hue: GUI Frontend to cluster
  • Oozie: workflow management
  • Mahoot: Machine learning library

Thursday, January 15, 2015

Link between SSL, HTTPS, Certificate, Browser, Server

I have tried to draw a link between SSL, HTTPS, Certificate, Browser, Server using simple language in following steps: 

 Process of obtaining Certificate:
  •  A web site operator obtains a certificate by applying to a certificate provider (a CA that presents as a commercial retailer of certificates) with a certificate signing request. 
  • The certificate provider generate a certificate with public and private key.
  • Now web site operator send a request to certificate Authority to sign the request. For security reasons the private key is not part of the request and is not sent to the certificate authority. 
  • The certificate provider signs the request, thus producing a public certificate. 
  • During web browsing, this public certificate and public key is served to any web browser that connects to the web site and proves to the web browser that the provider believes it has issued a certificate to the owner of the web site.
 SSL flow:
    • End-user enter a secure site to connects to
    • Browser identify it site is secure, it request SSL certificate from the site.
    • SSL certificate contains following information:
      • Domain name
      • Company name
      • City
      • State
      • Country
      • Expiration date
      • issuing Authority
    • Browser verifies following three items:
      • Certificate is not expired
      • Certificate has been issued by the Authority that browser trusts
      • Certificate has been issued to browser from the site for which it has been generated
    • If any of the above condition does not meet then browser shows error for the certificate about the same. ex: This Connection is Untrusted / Certificate Error
    • With certificate Browser also receives a Public key
    • Based on this initial exchange, browser and the website then initiate the 'SSL handshake'. The SSL handshake involves the generation of shared secrets to establish a uniquely secure connection between yourself and the website. This involves following steps:
      • The browser then uses the public key, to encrypt a random symmetric encryption key and sends it to the server with the encrypted URL required as well as other encrypted http data.
      • The web server decrypts the symmetric encryption key using its private key and uses the symmetric key to decrypt the URL and http data.
    • SSL used asymmetric Public key Infrastructure (PKI). This used two keys: Public keys to encrypt and Private key to decrypt. Anything encrypted with the public key can only be decrypted by the private key and vice-versa.
    • In Symmetric Public Key Infrastructure, same key is used for encryption and decryption.

Monday, December 8, 2014

Design pattern Quick Guide

Observer Pattern :  Behavioral pattern

Define a one-to-many dependency between objects so that when one object changes state, all its dependents are notified and updated automatically.

Real Life example:
  • RSS feeds.  When I want to get updates from a particular feed, I add it to my feed reader. Any time that the RSS feed has an update, it will appear in my reader automatically.  This is the Observer pattern in action, a publisher/subscriber relationship with one source having many subscribers 
  • The whole concept of listeners is based on this pattern.
Adapter Pattern : Structural pattern


Convert the interface of a class into another interface clients expect. Adapter lets classes work together that couldn't otherwise because of incompatible interfaces.

Real Life Example:
  • The best example for the adapter pattern is based around AC power adapters.
Downside:
Adapter pattern is just a fix for a badly designed system, which didn't consider all possibilties. While this is a fair point, it is an important part of a pluggable architecture.  It can also add a level of complexity to your code, making debugging more difficult.

The Facade pattern  : Structural pattern

Provide a unified interface to a set of interfaces in a subsystem. Façade defines a higher-level interface that makes the subsystem easier to use.

Real Life Example:
  • Concept behind facade is to simplify an interface, service oriented architectures make use of the facade pattern. For example, in web services, one web service might provide access to a number of smaller services that have been hidden from the caller by the facade.
Downside:
 By introducing the Facade into your code, you will be hardwiring subsystems into the Facade. This is fine if the subsystem never changes, but if it does, your Facade could be broken. Therefore, developers working on the subsystem should be made aware of any Facade around their code.

Factory method pattern : Creational pattern

Define an interface for creating an object, but let the subclasses decide which class to instantiate. The Factory method lets a class defer instantiation to subclasses

Real Life Example:
  • The idea behind the Factory Method pattern is that it allows for the case where a client doesn't know what concrete classes it will be required to create at run-time, but just wants to get a class that will do the job. The FactoryMethod builds on the concept of a simple Factory, but lets the subclasses decide which implementation of the concrete class to use.  You'll see factories used in logging frameworks, and in a lot of scenarios where the client doesn't need to know about the concrete implementations. It's a good approach to encapsulation.
Abstract Factory Pattern : Creational pattern
Provides an interface for creating families of related or dependent objects without specifying their concrete classes.

Strategy Pattern : Behavioral pattern
Defines a set of encapsulated algorithms that can be swapped to carry out a specific behaviour

Real Life Example:
  • Compression algorithm
Visitor Pattern : Behavioral pattern
Allows for one or more operation to be applied to a set of objects at runtime, decoupling the operations from the object structure.

Mongo sharding example


### To Start config server instance (config server port: 27019)
mongod.exe --configsvr --dbpath ../../configdb --port 27019

### To start mongos instance (router) (router port: 27017)
mongos.exe --configdb localhost:27019

###Adding a shard
First start mongoDb instance
mongod.exe --config ../../mongo.cfg
mongod.exe --config ../../mongo1.cfg

Config file content:
##store data here
dbpath=C:\poc\mdb\data

##all output go here
logpath=C:\poc\mdb\log\mongo.log

##log read and write operations
##diaglog=3

###Auth detail admin/admin
auth=false


##Port binding
port=27020


###Open a new instance and add above mongodb instance in cluster:
mongo.exe --host localhost --port 27017 
once mongos console connected:
 sh.addShard("localhost:27020")
 sh.addShard("localhost:27021") 

sh.enableSharding("ankur")
use config
db.databases.find()

###Insert bulk data
for (var i = 1; i <= 1000; i++) db.test_collection.insert( { x : i } )

###To know the shard status
sh.status()
 
###To shard a collection
 sh.shardCollection("ankur.records",{"zip":1, "name":1})
 
###Data insertion
{
db.records.insert( { zip : 60111 , name: "ankur"})
db.records.insert( { zip : 60111 , name: "goel"})
db.records.insert( { zip : 50111 , name: "ankur"})
db.records.insert( { zip : 50111 , name: "goel"})
db.records.insert( { zip : 70111 , name: "dr"})
} 

###To know chunk distribution
db.records.getShardDistribution()


Source: http://docs.mongodb.org/manual/reference/method/#sharding

A day with Maven

POM: Project Object Model

Maven has three life cycle:
  • Clean
  • default (build)
  • Site

Clean Life cycle has following phases:
  • Pre-Clean
  • Clean
  • Post-Clean
Build(or Default) life cycle has following phase(primarily used) sequence. (Build Life cycle has 23 phases):
  • prepare-resources
  • compile
  • package
  • Install
Site life-cycle has following phases:
  • pre-site
  • site
  • post-site
  • site-deploy

Syntax for life-cycle and phase:
mvn <life-cycle> : <phase>
ex: mvn clean:clean

Syntax for activating profile:
mvn <life-cycle>/<phase>   -P<profile>
ex: mvn clean -P phix


When a phase is called via Maven command, for example mvn compile, only phases upto and including that phase will execute.

In Maven terminology, a repository is a place i.e. directory where all the project jars, library jar, plugins or any other project specific artifacts are stored and can be used by Maven easily.

Maven repository are of three types
Maven is actually a plugin execution framework where every task is actually done by plugins.

A plugin generally provides a set of goals and which can be executed using following syntax:
mvn <plugin-name>:<goal-name>

Maven provided two types of plugin:
  • Build plugins (They execute during the build and should be configured in the <build/> element of pom.xml)
  • Reporting plugins (They execute during the site generation and they should be configured in the <reporting/> element of the pom.xml)

Dependency Management:
The dependency management section is a mechanism for centralizing dependency information. When you have a set of projects that inherits a common parent it's possible to put all information about the dependency in the common POM and have simpler references to the artifacts in the child POMs.

For parent:
<dependencyManagement>
   <dependencies>
     <dependency>
       <groupId>test</groupId>
       <artifactId>a</artifactId>
       <version>1.2</version>
     </dependency>
   </dependencies> 
</dependencyManagement>


For child: If below code is not copied in child then no dependency will be included in child project from parent. Need to define all required dependency in child project.
  <dependencies>
    <dependency>
      <groupId>group-a</groupId>
      <artifactId>artifact-a</artifactId>
    </dependency>
 </dependencies>