Missing Spark History logs

Problem We have multiple Spark jobs running during ETL phase of our application. Some of them are really large and complicated jobs which takes hours to complete. One of such job, which uses GraphX & Pregal libraries, started failing after running for sometime. We use Spark History server to debug any jobs after they are… Read More Missing Spark History logs

Setting up web-server, mail server & DB in AWS

Recently an emerging SMB from cookware manufacturing sector approached me to consult about some of the IT challenges they were facing. They have a simple website designed in WordPress and hosted in BigRock & a new website is under development. A limited number of corporate email accounts are also created in the same domain. They also have an ERP… Read More Setting up web-server, mail server & DB in AWS