YFinance Portal & Pipeline

Full Description

This project involves leveraging Yahoo Finance's open-source API to gather and analyze stock data for multiple companies over the past ten years. Using Django, a Python-based web framework, the data is imported into a database and visualized. The project focuses on setting up a data flow from the yfinance API to a relational database, allowing for analysis and visualization of stock trends. Key features include data extraction via Jupyter notebooks, storage in Parquet formats, and management commands for data handling.

The project centers around the extraction, storage, and analysis of stock data using Yahoo Finance's (yfinance) API. The initial objective was to see if stock data could be gathered efficiently and cost-effectively due to its open-source nature, making it an excellent resource for data-rich projects. After working in FinTech, I recognize the potential for valuable insights without incurring data costs.

Data Extraction and Storage:

The data extraction process began with identifying relevant company symbols (e.g., TSLA for Tesla). A Jupyter notebook was used to pull historical data, with each company's symbol attached to the API call. This data was saved in Parquet files, offering flexibility and efficiency in handling large datasets.

The project adopted Django for its robust framework capabilities, allowing for the creation of relational models, views, and templates. Django's model structure was defined to form the data store, ensuring proper relational integrity and facilitating easy data manipulation. The data obtained from yfinance was then imported into the Django application, creating a comprehensive database of stock information.

Application Functionality:

Django's management commands played a crucial role in automating data processes. Custom commands were developed to streamline data importation and management. These commands could be executed via terminal, enabling batch processing of large datasets efficiently. The application employed try-except blocks to maintain data uniqueness, ensuring the integrity of over 500,000 records by batching them into manageable sizes before saving.

The Django application’s admin interface allowed superusers to interact with and manage the data effectively. Users could view stock metrics, such as Open vs. Close prices, and the data was aggregated by company for the latest dates. This facilitated a deeper understanding of stock performance and trends.

Data Visualization:

Data visualization was a core aspect of the project. Using the Matplotlib package, the project rendered visual representations of stock data. Line graphs illustrated stock prices over time for specific symbols, such as TSLA. Additionally, the project highlighted the top 6 and bottom 6 companies based on daily average Close prices over the past 30 days, providing clear insights into market movements.

Database and Deployment:

While the initial setup used SQLite for simplicity, the project recognized PostgreSQL as a more robust alternative for production environments. Django’s settings allowed easy modification of database specifications, supporting the potential for multiple databases and tenant-level access.

In summary, the project successfully integrates yfinance data with Django, creating a powerful tool for stock data analysis and visualization. The structured approach to data extraction, storage, and processing, combined with effective visualization techniques, demonstrates a comprehensive solution for financial data analysis.