Problem We have multiple Spark jobs running during ETL phase of our application. Some of them are really large and complicated jobs which takes hours to complete. One of such job, which uses GraphX & Pregal libraries, started failing after running for sometime. We use Spark History server to debug any jobs after they are… Read More Missing Spark History logs
We have faced a situation where we needed to create Json representation for a dynamic set of fields & their values which represented by a Map[String, String]. Example: Scala class case class Person(name: String, age: String, customFields: Map[String,String]) Default Json representation of above class will be: { “name”: “anil”, “age”: “30”, “customFields”: { “field1”: “value1”,… Read More Dynamic fields to Json in Scala
Spark website provides three options for using a custom log4j configuration for logging. Here I am explaining about the 3rd option
Passing a function to another function as argument in Scala… Read More Higher-order Functions in Scala
One of the many cool features Scala provides over Java is the ability to provide default values for parameters. Such parameters become optional and the callers can invoke the method without providing a value for those parameters. This is useful in cases wherever you overload methods in Java – like constructor chaining, releasing a newer… Read More Optional Parameters in Scala
Learn how to create a MySQL instance in AWS RDS. RDS stands for Relational Database Service which is the DBaaS solution from AWS. It enables you to create database instances in the cloud with just few clicks. Such instances can be made publicly accessible or private so that it can be accessed only by other applications… Read More Creating AWS RDS MySQL instance in 5min
If you are exploring Golang (aka Go) you must have faced some difficulty in understanding the method concepts, especially if you are coming from a Java background which is more statically typed. It gets more complicated when you get into methods with pointers. Tour of Go gives step by step guide with example programs and an environment to edit… Read More Golang functions & methods simplified
Back in 2008, at the LinuxWorld and Next Generation Data Center Expo, Stateless Computing was suggested as the future in Cloud. A stateless application does not rely on the session state of any kind to process a client request. For such an application you should be able to say that, for any request “this request… Read More What is better than State-less?
Quick introduction to AWS Elastic Load Balancer (ELB) and explaining how to set-up an ELB with two web server EC2 instances. The video also shows ELB in action with one EC2 instance failure. Please watch and subscribe to my YouTube channel.
In this video I am explaining how you can launch an EC2 instance in AWS. EC2 stands for Elastic Compute cloud and its the term used by Amazon Web Service (AWS) for their virtual machines. Please watch and subscribe to my YouTube channel.
Anil G Kurian 