Big Data Powered Job Portal
- The Client needed a job search portal, which helps the user to search for any job available in the United States and also get the employer’s details and insight information over the past 3 years.
- The website needs to provide all the latest job opportunities. The jobs can be sorted or also filtered according to user’s requirements. The site also provides insight information of approximately 70,000 companies. The latest news feeds of every company are also updated on the website. The site also displays the sentiment(positive/negative) for each company. The user also gets a survey regarding their experience with their current or previous employer. The user can also take surveys on their preferred jobs and interests. Based on the interest score, the jobs preferred by the user are marked as recommended.
- The payment methods were also needed to be implemented on the website, by which user can subscribe to higher user level to unlock more filters and get a better search experience. The user can also subscribe to guides and career advice provided on the website.
- Aggregate and store data of every company(approx 70,000) and their feeds.
- To analyse over a million records of political donations provided by each company in different years.
- To update the news feed table in HBase with the latest feed for every company stored in the Database.
- To calculate the sentiment for every company stored in the DataBase.
- Used Hbase to store all the aggregated company data and their feeds.
- Implemented the data analysis workflows using Knime.
- Used Apache Flume to take the feed URL and collects the news feed periodically and updates the Hbase table.
- The sentiment is calculated from the feed articles related with each company.
- All the solutions worked as expected to provide the user with a good job search experience.
- The user gets all the latest feeds about the company.
- Users can take surveys on their preferred job. This will also help in highlighting the jobs recommended by the user.
- Java 7
- Apache Flume
- NLP and Sentiment Analysis