Data architecture and GUI in the real estate industry

A common problem of businesses in all kinds of industries is the integration of data from different sources into a central database. In one of our recent projects we built an entire data architecture from the scratch for a real estate company. The data architecture included an ETL-pipleline extracting and processing data from APIs and data entry masks, a central database plus a web front-end that allows users to manage the data pipeline and interact with the database - a simple but effective architecture.

The company extracts real estate data from various sources, mainly APIs, which had to be integrated into the data pipeline. In the next step, the data pipeline processes the raw data from the API and other sources and transforms the raw data into a high quality dataset, catering the requirements and preferences of the company. Finally, an automated data quality management system checks the dataset, cleans it if necessary and stores the data in the database. The pipeline and the respective algorithms for processing the data were written in Python.

Another component of the data architecture is a graphical user interface that allows its users to monitor the data pipeline and interact with the database, including searching, filtering, editing, adding, deleting, exporting or analyzing the data. To implement the GUI we used JavaScript and React, which are optimal for dynamic web interfaces. For deployment we used Django in the backend and hosted the system on a remote Linux server.