Web Crawler Developer

Who We Are

We are a small deep-tech startup with specialized capacities in behavior change solutions, offering a proven combination of behavioral psychology, data analytics, and digital communications. Newton focuses effectively on creating science-based solutions that tend to embrace social purpose and enhance the quality of people’s lives by creating a variety of products that use cutting-edge machine learning and data science methods to model, segment, and create the products.

Job Summary

As a Web Crawler, your role is to develop scripts to extract data from multiple websites and maintain the data pipelines. Develop a deep understanding of our vast data sources on the web and know exactly how, when, and which data to scrap, parse and store this data.

You will be leveraging existing frameworks and processes for scraping and ingesting web content. You will also develop frameworks for automating and maintaining constant flow of data from multiple sources.

You will be responsible for maintaining web crawlers and frameworks.

Requirements

2+ years for experience in building crawler/web-scrapping applications.Knowledge of web scrapping libraries and frameworks i.e.(Scrapy, Selenium, Beautiful Soup)Experienced analyzing HTML and CSS code to identify and extract data from websites Coding experience in Python, IDE’s, APIs and Multithreading.Algorithmic skills. For example, we develop algorithms to detect and remove the surplus "clutter" (boilerplate, templates) around the main textual content of a web page.Experience working with databases.Proficient knowledge of Python language with hands-on experience with Database integrations.Experience with data parsing, data mining, data analytics, etc.Data visualization experience is preferred.

Salary: $50,000 - $65,000 plus bonus and equity.

Key Skills

• Backend
• Frontend
• Python