diff --git a/docs/databases-and-storage.md b/docs/databases-and-storage.md new file mode 100644 index 0000000..a4356be --- /dev/null +++ b/docs/databases-and-storage.md @@ -0,0 +1,193 @@ +# ARGUS Storage Research + +## Intro + +It was previously a research what ARGUS should store and which database/storage approach fits the project. +Now it's a simple documentation about how the storage and domain logic looks like and should develop in the next sprints. + +ARGUS is moving from live API requests and in-memory analytics toward real data workflows. +The first storage decision should support local market analytics, SQL practice and future dashboard features without adding unnecessary infrastructure too early. + +--- + +## First Storage Approach + +DuckDB should be the first storage technology for ARGUS. + +Reason: + +* ARGUS currently needs local analytical storage, not a full server database +* DuckDB fits historical time-series analysis well +* it supports SQL-based analytics without requiring a database server +* it works well with Python and notebook-based exploration +* it keeps the first storage implementation manageable +* it can later be replaced or complemented by PostgreSQL if ARGUS becomes more product-like + +The first storage implementation should focus on: + +* historical market data +* cleaned OHLCV-ready price data +* source information +* instruments that ARGUS can analyze + +PostgreSQL and SQLGate become more relevant later. + +For the first DuckDB phase, the goal is to build a clean local analytics workflow. + +--- + +## Developer Interaction Workflow + +ARGUS should use a practical developer workflow for DuckDB. + +The goal is to make the database easy to inspect, explore and validate before logic is moved into production code. + +### Notebook Exploration + +Notebooks should be the main exploration layer. + +They are useful for: + +* opening the DuckDB database +* testing SQL queries +* validating imported data +* comparing SQL results with pandas calculations +* exploring metric logic +* documenting research assumptions + +This workflow is especially useful before turning queries into reusable project code. + +Notebook exploration should be preferred over a GUI database tool in the first phase. + +### DuckDB CLI + +The DuckDB CLI should be used for quick database inspection. + +It is useful for: + +* checking available tables +* running small SQL queries +* validating stored records +* debugging the local database file + +The CLI is not the main research environment, but it is useful as a fast inspection tool. + +--- + +## First Data Model Direction + +The first data model should support FX data now and broader market data later. + +ARGUS should not use a narrow `date | value` table as the main market-data model. + +That would work for simple exchange rates, but it would become limiting once ARGUS adds stocks, ETFs, indices or broader market APIs. + +The first model focuses on two primary metadata entities, supported by standard operational schemas and communication interfaces: + +```text +DataSource +Instrument +``` + +### DataSource + +Stores where data came from. + +Current operational fields: + +```text +name +provider_kind +requires_api_key (defaults to False) +``` + +### Instrument + +Stores what ARGUS can analyze. +Current operational fields: + +```text +symbol +name +asset_class +currency (optional) +exchange (optional) +base_currency (optional) +quote_currency (optional) +``` + +### Internal Operational Models + +To support decoupled runtime communication and execution pipelines, ARGUS utilizes specialized Python dataclasses. These structures orchestrate active data flows between data-fetching clients and services. + +**MarketDataRequest** +Encapsulates parameters passed to fetching engines and downstream query handlers. + +* `source`: The targeted `DataSource` instance +* `instrument`: The targeted `Instrument` instance +* `timeframe`: Granularity of requested bars (like `"1d"`, `"1h"`). +* `start`: Temporal lower bound (`datetime.date`) +* `end`: Temporal upper bound (`datetime.date`) + +**MarketDataResponse** +Encapsulates runtime data payloads, structural enforcement, and pipeline feedback. + +* `source`: The originating `DataSource` instance +* `instrument`: The corresponding `Instrument` instance +* `bars`: A `pandas.DataFrame` holding the time-series payload +* `message`: Pipeline feedback or error context (defaults to `""`) + +### The Price Bar Schema (Data Uniformity) + +Crucially, **Price Bars are not treated as a standalone domain metadata model or database entity.** Instead, the Price Bar specification acts purely as an internal **uniformity schema** to normalize incoming time-series data across disparate third-party APIs. + +All responses must be mapped into this standard schema layout within the `bars` DataFrame. + +#### Provider Mapping Layer + +Third-party structures are converted to this unified standard upon ingest. For example, `yfinance` fields map to the uniform schema via `YFINANCE_PRICE_BAR_MAPPING`: + +* `Date` $\rightarrow$ `timestamp` +* `Open` $\rightarrow$ `open` +* `High` $\rightarrow$ `high` +* `Low` $\rightarrow$ `low` +* `Close` $\rightarrow$ `close` +* `Adj Close` $\rightarrow$ `adjusted_close` +* `Volume` $\rightarrow$ `volume` + +--- + +## Future Direction + +Later sprints can expand the storage layer step by step. + +Possible later additions: + +| Future Area | Possible Additions | +| --- | --- | +| Better source mapping | source-specific symbols, provider metadata | +| Watchlists | user-selected instruments | +| Reports | generated report metadata and history | +| Macro data | FRED indicators and observations | +| Paper trading | simulated orders, positions and portfolio history | +| Server architecture | PostgreSQL | +| SQL tooling | SQLGate with PostgreSQL | +| Cloud direction | managed PostgreSQL or cloud storage | + +SQLGate should be kept for a later PostgreSQL phase. + +It becomes useful when ARGUS moves toward: + +* server-based storage +* stronger database management +* richer metadata +* more stable application state +* user-facing features +* report history +* cloud-ready architecture + +Additional metadata such as documentation links, terms links or provider governance fields can also become useful later. + +For the first DuckDB phase, these details should stay in research documentation instead of the database schema. + +--- diff --git a/pyproject.toml b/pyproject.toml index e2dc784..a8ac7b1 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -9,6 +9,7 @@ dependencies = [ "numpy", "matplotlib", "yfinance", + "duckdb", ] [project.optional-dependencies] diff --git a/src/argus/analytics/charts/trend_chart.py b/src/argus/analytics/charts/trend_chart.py index 1293361..db5dacb 100644 --- a/src/argus/analytics/charts/trend_chart.py +++ b/src/argus/analytics/charts/trend_chart.py @@ -1,8 +1,8 @@ import matplotlib.pyplot as plt -from argus.services.timeseries_service import prepare_trend_analysis +import pandas as pd -def create_trendchart(curr_symbol: str, start: str, end: str, interval: str): +def create_trendchart(df: pd.DataFrame, min_max_rates: dict): """ Create a trend chart for exchange-rate analysis. @@ -30,10 +30,6 @@ def create_trendchart(curr_symbol: str, start: str, end: str, interval: str): Minimum and maximum exchange-rate values are marked with scatter points and annotations. """ - result = prepare_trend_analysis(curr_symbol, start, end, interval) - if result is None: - return None - df, min_max_rates = result min_date = min_max_rates["min_date"][0] min_rate = min_max_rates["min_rate"][0] max_date = min_max_rates["max_date"][0] diff --git a/src/argus/clients/yfinance_client.py b/src/argus/clients/yfinance_client.py index de86041..468b87b 100644 --- a/src/argus/clients/yfinance_client.py +++ b/src/argus/clients/yfinance_client.py @@ -1,43 +1,48 @@ import yfinance as yf -import logging - - -def get_timeseries(curr_symbol, start, end, interval): - """ - Fetch historical exchange-rate time series data from Yahoo Finance. - - Args: - curr_symbol (str): Currency symbol used by Yahoo Finance, for example - "EURUSD=X". - start (str): Start date of the requested time range in YYYY-MM-DD format. - end (str): End date of the requested time range in YYYY-MM-DD format. - interval (str): Data interval supported by Yahoo Finance, for example - "1d", "1h", or "15m". - - Returns: - pandas.DataFrame | None: A DataFrame containing the columns ``date`` and - ``rate`` if data was successfully fetched. Returns ``None`` if the - request fails, returns no data, or an exception occurs. - """ +import pandas as pd +from argus.domain.internal_models import ( + MarketDataRequest, + PRICE_BAR_COLUMNS, + YFINANCE_PRICE_BAR_MAPPING, +) + + +def get_timeseries(request: MarketDataRequest) -> pd.DataFrame: + start = str(request.start) + end = str(request.end) + timeframe = request.timeframe + curr_pair = ( + f"{request.instrument.base_currency}{request.instrument.quote_currency}=X" + ) + try: - yf_logger = logging.getLogger("yfinance") - yf_logger.disabled = True - data = yf.download( - tickers=curr_symbol, + raw_resp = yf.download( + tickers=curr_pair, start=start, end=end, - interval=interval, + interval=timeframe, multi_level_index=False, progress=False, ) - yf_logger.disabled = False - if data is None: - return None - if data.empty: - return None - data = data.reset_index() - data = data[["Date", "Close"]] - data = data.rename(columns={"Date": "date", "Close": "rate"}) - return data except Exception: - return None + raise ConnectionError("Network error or connection timeout") + + if raw_resp is None: + raise ConnectionError("Yahoo Finance API returned an invalid response") + + if ( + raw_resp.empty + or "Close" not in raw_resp.columns + or raw_resp["Close"].dropna().empty + ): + raise ValueError("Quote not found or no data available for symbol") + + resp = normalize_yfinance_bars(raw_resp) + return resp + + +def normalize_yfinance_bars(raw_df: pd.DataFrame) -> pd.DataFrame: + df = raw_df.copy() + df = df.reset_index() + df = df.rename(columns=YFINANCE_PRICE_BAR_MAPPING) + return df[list(PRICE_BAR_COLUMNS)] diff --git a/src/argus/domain/internal_models.py b/src/argus/domain/internal_models.py new file mode 100644 index 0000000..0d64796 --- /dev/null +++ b/src/argus/domain/internal_models.py @@ -0,0 +1,71 @@ +from dataclasses import dataclass +from datetime import date +import pandas as pd + + +@dataclass +class DataSource: + name: str + provider_kind: str + requires_api_key: bool = False + + +@dataclass +class Instrument: + symbol: str + name: str + asset_class: str + currency: str | None = None + exchange: str | None = None + base_currency: str | None = None + quote_currency: str | None = None + + +PRICE_BAR_COLUMNS = ( + "timestamp", + "open", + "high", + "low", + "close", + "adjusted_close", + "volume", +) + +YFINANCE_PRICE_BAR_MAPPING = { + "Date": "timestamp", + "Open": "open", + "High": "high", + "Low": "low", + "Close": "close", + "Adj Close": "adjusted_close", + "Volume": "volume", +} + + +@dataclass +class MarketDataRequest: + source: DataSource + instrument: Instrument + timeframe: str + start: date + end: date + + +@dataclass +class MarketDataResponse: + source: DataSource + instrument: Instrument + bars: pd.DataFrame + message: str = "" + + def __post_init__(self) -> None: + if not isinstance(self.bars, pd.DataFrame): + raise TypeError("bars must be a pandas DataFrame") + if self.message == "": + missing_cols = [ + col for col in PRICE_BAR_COLUMNS if col not in self.bars.columns + ] + if missing_cols: + raise ValueError( + f"Missing required columns in bars DataFrame: {missing_cols}" + ) diff --git a/src/argus/gui/app.py b/src/argus/gui/app.py index 9a19523..02f4c1f 100644 --- a/src/argus/gui/app.py +++ b/src/argus/gui/app.py @@ -1,6 +1,7 @@ import tkinter as tk +import pandas as pd from matplotlib.backends.backend_tkagg import FigureCanvasTkAgg -from argus.analytics.charts.trend_chart import create_trendchart +from argus.services.market_data_service import prepare_trend_analysis from argus.services.calculator_service import calc, check_op from argus.services.conversion_service import convert, check_currency from argus.domain.validation import parse_amount @@ -76,11 +77,6 @@ def show_trend() -> None: global trend_canvas global trend_chart_widget - curr_symbol = "EURUSD=X" - start = "2024-01-01" - end = "2025-01-01" - interval = "1d" - calc_frame.pack_forget() conv_frame.pack_forget() menu_frame.pack_forget() @@ -90,7 +86,8 @@ def show_trend() -> None: content.pack(side="top", fill=tk.BOTH, expand=True) if trend_canvas is None: - fig = create_trendchart(curr_symbol, start, end, interval) + df = pd.DataFrame() + fig = prepare_trend_analysis(df) if fig is None: return None fig.set_size_inches(7, 4) diff --git a/src/argus/main.py b/src/argus/main.py index 7130dbf..105de61 100644 --- a/src/argus/main.py +++ b/src/argus/main.py @@ -1,11 +1,14 @@ from argus.gui.app import app +from argus.storage.database import initialize_database -def main() -> None: +def main(db) -> None: """ The main function that starts the application. """ + initialize_database(db) app() -main() +db = "" +main(db) diff --git a/src/argus/services/market_data_service.py b/src/argus/services/market_data_service.py new file mode 100644 index 0000000..9328141 --- /dev/null +++ b/src/argus/services/market_data_service.py @@ -0,0 +1,47 @@ +from argus.domain.internal_models import MarketDataRequest, MarketDataResponse +from argus.clients.yfinance_client import get_timeseries +from argus.storage.database import read_price_bars, insert_price_bar +import pandas as pd + + +def get_market_data( + db: str, + request: MarketDataRequest, +) -> MarketDataResponse: + """ + Get a time series either from local stroage or client with first-storage-workflow + + Args: + curr_symbol (str): Currency symbol used by Yahoo Finance, for example + "EURUSD=X". + start (str): Start date of the requested time range in YYYY-MM-DD format. + end (str): End date of the requested time range in YYYY-MM-DD format. + intervall (str): Data interval supported by Yahoo Finance, for example + "1d", "1h", or "15m". + + Returns: + pd.DataFrame | None: A + DataFrame with dates and rates. Returns + ``None`` if no time-series data could be fetched. + """ + bars = read_price_bars(db, request) + if not (bars.empty): + db_response = MarketDataResponse( + source=request.source, instrument=request.instrument, bars=bars, message="" + ) + return db_response + + try: + bars = get_timeseries(request) + api_response = MarketDataResponse( + source=request.source, instrument=request.instrument, bars=bars + ) + insert_price_bar(db, api_response) + return api_response + except (ConnectionError, ValueError) as e: + return MarketDataResponse( + source=request.source, + instrument=request.instrument, + bars=pd.DataFrame(), + message=str(e), + ) diff --git a/src/argus/services/timeseries_service.py b/src/argus/services/timeseries_service.py deleted file mode 100644 index b6251bb..0000000 --- a/src/argus/services/timeseries_service.py +++ /dev/null @@ -1,41 +0,0 @@ -import pandas as pd -from argus.clients.yfinance_client import get_timeseries -from argus.analytics.metrics.trend_metrics import ( - add_rolling_average, - add_daily_percentage_change, - get_min_max_rates, -) - - -def prepare_trend_analysis( - curr_symbol: str, start: str, end: str, intervall: str -) -> tuple[pd.DataFrame, dict] | None: - """ - Prepare time-series data for trend analysis. - - Fetches historical exchange-rate data for the given currency symbol and - enriches it with daily percentage changes and a rolling average. It also - calculates the minimum and maximum exchange rates for the resulting time - series. - - Args: - curr_symbol (str): Currency symbol used by Yahoo Finance, for example - "EURUSD=X". - start (str): Start date of the requested time range in YYYY-MM-DD format. - end (str): End date of the requested time range in YYYY-MM-DD format. - intervall (str): Data interval supported by Yahoo Finance, for example - "1d", "1h", or "15m". - - Returns: - tuple[pd.DataFrame, dict] | None: A tuple containing the prepared - DataFrame and a dictionary with minimum and maximum rates. Returns - ``None`` if no time-series data could be fetched. - """ - - df = get_timeseries(curr_symbol, start, end, intervall) - if df is None: - return None - df = add_daily_percentage_change(df) - df = add_rolling_average(df) - min_max_rates = get_min_max_rates(df) - return df, min_max_rates diff --git a/src/argus/services/trend_analysis_service.py b/src/argus/services/trend_analysis_service.py new file mode 100644 index 0000000..a8428ae --- /dev/null +++ b/src/argus/services/trend_analysis_service.py @@ -0,0 +1,37 @@ +from argus.analytics.metrics.trend_metrics import ( + add_rolling_average, + add_daily_percentage_change, + get_min_max_rates, +) +from matplotlib.figure import Figure +from argus.analytics.charts.trend_chart import create_trendchart +from argus.services.market_data_service import get_market_data +from argus.domain.internal_models import MarketDataRequest + + +def prepare_trend_analysis(db: str, request: MarketDataRequest) -> Figure | str: + """ + Prepare time-series data and generate a trend analysis chart. + + Enriches the historical exchange-rate DataFrame with daily percentage changes + and a rolling average, calculates the minimum and maximum rates, and uses + the result to build a trend visualization chart. + + Args: + df (pd.DataFrame): A DataFrame containing market data time-series. + + Returns: + plotly.graph_objects.Figure: A figure object representing the + generated trend chart. + """ + response = get_market_data(db, request) + + if response.message: + return response.message + + df = response.bars.copy() + df = add_daily_percentage_change(df) + df = add_rolling_average(df) + min_max_rates = get_min_max_rates(df) + fig = create_trendchart(df, min_max_rates) + return fig diff --git a/src/argus/storage/database.py b/src/argus/storage/database.py new file mode 100644 index 0000000..aa00c78 --- /dev/null +++ b/src/argus/storage/database.py @@ -0,0 +1,300 @@ +import duckdb +import pandas as pd +from argus.domain.internal_models import ( + DataSource, + Instrument, + MarketDataRequest, + MarketDataResponse, +) + + +def initialize_database(database_path: str) -> None: + """ + Initialize the DuckDB database schema. + + Creates the required sequences and tables for data sources, + instruments, and price bars. + + Args: + database_path (str): Path to the DuckDB database file. + + Returns: + None + """ + queries = [ + "CREATE SEQUENCE IF NOT EXISTS data_sources_id_seq;", + "CREATE SEQUENCE IF NOT EXISTS instruments_id_seq;", + "CREATE SEQUENCE IF NOT EXISTS price_bars_id_seq;", + """ + CREATE TABLE IF NOT EXISTS data_sources ( + id INTEGER PRIMARY KEY DEFAULT nextval('data_sources_id_seq'), + name TEXT NOT NULL UNIQUE, + provider_kind TEXT NOT NULL, + requires_api_key BOOLEAN NOT NULL + ); + """, + """ + CREATE TABLE IF NOT EXISTS instruments ( + id INTEGER PRIMARY KEY DEFAULT nextval('instruments_id_seq'), + symbol TEXT NOT NULL UNIQUE, + name TEXT NOT NULL, + asset_class TEXT NOT NULL, + currency TEXT, + exchange TEXT, + base_currency TEXT, + quote_currency TEXT + ); + """, + """ + CREATE TABLE IF NOT EXISTS price_bars ( + id INTEGER PRIMARY KEY DEFAULT nextval('price_bars_id_seq'), + source_id INTEGER NOT NULL, + instrument_id INTEGER NOT NULL, + timestamp DATE NOT NULL, + close DOUBLE NOT NULL, + open DOUBLE, + high DOUBLE, + low DOUBLE, + adjusted_close DOUBLE, + volume DOUBLE, + FOREIGN KEY (source_id) REFERENCES data_sources (id), + FOREIGN KEY (instrument_id) REFERENCES instruments (id), + UNIQUE (source_id, instrument_id, timestamp,close) + ); + """, + ] + + connection = duckdb.connect(database_path) + try: + for query in queries: + connection.execute(query) + finally: + connection.close() + + +def get_or_create_source(connection, source: DataSource) -> int: + """ + Get an existing data source ID or create a new data source. + + Searches for a data source by name. If it already exists, its ID is + returned. Otherwise, the data source is inserted and the new ID is + returned. + + Args: + connection: Active DuckDB connection. + source (DataSource): Data source model containing provider metadata. + + Returns: + int: Database ID of the existing or newly created data source. + + Raises: + ValueError: If the data source could not be inserted or found. + """ + insert_query = """ + INSERT INTO data_sources (name, provider_kind, requires_api_key) + VALUES (?,?,?) + ON CONFLICT DO NOTHING; + """ + search_query = """ + SELECT id FROM data_sources + WHERE name=? + """ + + result = connection.execute( + query=search_query, + parameters=[source.name], + ).fetchone() + if result is not None: + return result[0] + + connection.execute( + query=insert_query, + parameters=[source.name, source.provider_kind, source.requires_api_key], + ) + + result = connection.execute( + query=search_query, + parameters=[source.name], + ).fetchone() + + if result is None: + raise ValueError("Data source could not be inserted.") + + return result[0] + + +def get_or_create_instrument(connection, instrument: Instrument) -> int: + """ + Get an existing instrument ID or create a new instrument. + + Searches for an instrument by symbol. If it already exists, its ID is + returned. Otherwise, the instrument is inserted and the new ID is + returned. + + Args: + connection: Active DuckDB connection. + instrument (Instrument): Instrument model containing symbol and + asset metadata. + + Returns: + int: Database ID of the existing or newly created instrument. + + Raises: + ValueError: If the instrument could not be inserted or found. + """ + insert_query = """ + INSERT INTO instruments ( + symbol, + name, + asset_class, + currency, + exchange, + base_currency, + quote_currency) + VALUES (?,?,?,?,?,?,?) + ON CONFLICT DO NOTHING; + """ + search_query = """ + SELECT id FROM instruments + WHERE symbol=? + """ + + result = connection.execute( + query=search_query, + parameters=[instrument.symbol], + ).fetchone() + if result is not None: + return result[0] + + connection.execute( + query=insert_query, + parameters=[ + instrument.symbol, + instrument.name, + instrument.asset_class, + instrument.currency, + instrument.exchange, + instrument.base_currency, + instrument.quote_currency, + ], + ) + result = connection.execute( + query=search_query, + parameters=[instrument.symbol], + ).fetchone() + + if result is None: + raise ValueError("Instrument could not be inserted.") + + return result[0] + + +def insert_price_bar(db: str, marketdata: MarketDataResponse) -> None: + """ + Insert a price bar into the database. + + Ensures that the related data source and instrument exist, then inserts + the price bar into the ``price_bars`` table. Duplicate price bars are + ignored through the table's unique constraint. + + Args: + db (str): Path to the DuckDB database file. + price_bar (PriceBar): Price bar model containing source, + instrument, timestamp and market values. + + Returns: + None + """ + insert_query = """ + INSERT INTO price_bars ( + source_id, + instrument_id, + timestamp, + close, + open, + high, + low, + adjusted_close, + volume + ) + VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?) + ON CONFLICT DO NOTHING; + """ + connection = duckdb.connect(db) + try: + source_id = get_or_create_source(connection, marketdata.source) + instrument_id = get_or_create_instrument(connection, marketdata.instrument) + if marketdata.bars is None: + return None + for _, row in marketdata.bars.iterrows(): + connection.execute( + query=insert_query, + parameters=[ + source_id, + instrument_id, + row["timestamp"], + row["close"], + row["open"], + row["high"], + row["low"], + row["adjusted_close"], + row["volume"], + ], + ) + finally: + connection.close() + + +def read_price_bars(db: str, request: MarketDataRequest) -> pd.DataFrame: + """ + Read price bars for a source, instrument, and date range. + + Queries stored price bars joined with their data source and instrument + metadata. Results are ordered by timestamp and returned as a pandas + DataFrame. + + Args: + db (str): Path to the DuckDB database file. + source (DataSource): Data source used to filter the stored price bars. + instrument (Instrument): Instrument used to filter the stored price bars. + start_date (date): Inclusive start date of the requested time range. + end_date (date): Inclusive end date of the requested time range. + + Returns: + pd.DataFrame: DataFrame containing matching price bars and metadata. + """ + + search_query = """ + SELECT + data_sources.name AS source_name, + instruments.symbol AS instrument_symbol, + price_bars.timestamp, + price_bars.open, + price_bars.high, + price_bars.low, + price_bars.close, + price_bars.adjusted_close, + price_bars.volume + FROM price_bars + JOIN data_sources ON price_bars.source_id = data_sources.id + JOIN instruments ON price_bars.instrument_id = instruments.id + WHERE data_sources.name = ? + AND instruments.symbol = ? + AND price_bars.timestamp BETWEEN ? AND ? + ORDER BY price_bars.timestamp; + """ + + connection = duckdb.connect(db) + try: + result = connection.execute( + query=search_query, + parameters=[ + request.source.name, + request.instrument.symbol, + request.start, + request.end, + ], + ).df() + finally: + connection.close() + return result diff --git a/tests/test_internal_models.py b/tests/test_internal_models.py new file mode 100644 index 0000000..0fc7f0c --- /dev/null +++ b/tests/test_internal_models.py @@ -0,0 +1,66 @@ +import pytest +import pandas as pd +from datetime import date +from argus.domain.internal_models import DataSource, Instrument, MarketDataResponse + + +@pytest.fixture +def valid_source(): + return DataSource(name="Yahoo", provider_kind="yfinance_api") + + +@pytest.fixture +def valid_instrument(): + return Instrument(symbol="AAPL", name="Apple Inc.", asset_class="stock") + + +def test_market_data_response_accepts_valid_dataframe( + valid_source, valid_instrument +) -> None: + valid_bar = { + "timestamp": [date(2026, 1, 1)], + "open": [150.0], + "high": [155.0], + "low": [149.0], + "close": [153.5], + "adjusted_close": [153.5], + "volume": [1000000.0], + } + df = pd.DataFrame(valid_bar) + + resp = MarketDataResponse( + source=valid_source, instrument=valid_instrument, bars=df, message="" + ) + assert resp.bars.equals(df) + + +def test_market_data_response_raises_error_on_missing_columns( + valid_source, valid_instrument +) -> None: + incomplete_bar = { + "timestamp": [date(2026, 1, 1)], + "close": [153.5], + } + df = pd.DataFrame(incomplete_bar) + + with pytest.raises(ValueError) as exc_info: + MarketDataResponse( + source=valid_source, instrument=valid_instrument, bars=df, message="" + ) + + assert "Missing required columns" in str(exc_info.value) + + +def test_market_data_response_raises_error_if_not_a_dataframe( + valid_source, valid_instrument +) -> None: + invalid_input = "I'm just a string :D" + + with pytest.raises(TypeError) as exc_info: + MarketDataResponse( + source=valid_source, + instrument=valid_instrument, + bars=invalid_input, # type: ignore + ) + + assert "must be a pandas DataFrame" in str(exc_info.value) diff --git a/tests/test_market_data_series.py b/tests/test_market_data_series.py new file mode 100644 index 0000000..b7832a2 --- /dev/null +++ b/tests/test_market_data_series.py @@ -0,0 +1,138 @@ +import pytest +import pandas as pd +from datetime import date +from unittest.mock import Mock +from argus.domain.internal_models import ( + DataSource, + Instrument, + MarketDataRequest, + MarketDataResponse, +) +from argus.services.market_data_service import get_market_data + + +@pytest.fixture +def sample_source(): + return DataSource(name="Yahoo", provider_kind="yfinance_api") + + +@pytest.fixture +def sample_instrument(): + return Instrument(symbol="AAPL", name="Apple Inc.", asset_class="stock") + + +@pytest.fixture +def sample_response(sample_source, sample_instrument): + test_bar = { + "timestamp": date(2026, 1, 1), + "open": None, + "high": None, + "low": None, + "close": 1.89, + "adjusted_close": None, + "volume": None, + } + return MarketDataResponse( + source=sample_source, + instrument=sample_instrument, + bars=pd.DataFrame(test_bar, index=[0]), + ) + + +def test_get_market_data_storage_hit( + monkeypatch, sample_source, sample_instrument, sample_response +): + req = MarketDataRequest( + source=sample_source, + instrument=sample_instrument, + timeframe="1d", + start=date(2026, 1, 1), + end=date(2026, 1, 1), + ) + + monkeypatch.setattr( + "argus.services.market_data_service.read_price_bars", + lambda db, r: sample_response.bars, + ) + + mock_get_timeseries = Mock() + monkeypatch.setattr( + "argus.services.market_data_service.get_timeseries", mock_get_timeseries + ) + + res = get_market_data("mock_db_path", req) + + assert res is not None + assert not res.bars.empty + mock_get_timeseries.assert_not_called() + + +def test_get_market_data_storage_miss( + monkeypatch, sample_source, sample_instrument, sample_response +): + req = MarketDataRequest( + source=sample_source, + instrument=sample_instrument, + timeframe="1d", + start=date(2026, 1, 1), + end=date(2026, 1, 1), + ) + + monkeypatch.setattr( + "argus.services.market_data_service.read_price_bars", + lambda db, r: pd.DataFrame(), + ) + monkeypatch.setattr( + "argus.services.market_data_service.get_timeseries", + lambda r: sample_response.bars, + ) + + mock_insert = Mock() + monkeypatch.setattr( + "argus.services.market_data_service.insert_price_bar", mock_insert + ) + + res = get_market_data("mock_db_path", req) + + assert res is not None + mock_insert.assert_called_once() + + +def test_get_market_data_handles_client_exceptions( + monkeypatch, sample_source, sample_instrument +): + req = MarketDataRequest( + source=sample_source, + instrument=sample_instrument, + timeframe="1d", + start=date(2026, 1, 1), + end=date(2026, 1, 1), + ) + + monkeypatch.setattr( + "argus.services.market_data_service.read_price_bars", + lambda db, r: pd.DataFrame(), + ) + + def mock_value_error(request): + raise ValueError("Quote not found or no data available for symbol") + + monkeypatch.setattr( + "argus.services.market_data_service.get_timeseries", mock_value_error + ) + + res_val = get_market_data("mock_db_path", req) + assert res_val.bars.empty + assert res_val.message == "Quote not found or no data available for symbol" + + # Fall 2: ConnectionError testen (z.B. Internet weg) + def mock_connection_error(request): + raise ConnectionError("Network error or connection timeout") + + monkeypatch.setattr( + "argus.services.market_data_service.get_timeseries", mock_connection_error + ) + + res_conn = get_market_data("mock_db_path", req) + assert res_conn.bars.empty + assert res_conn.message == "Network error or connection timeout" diff --git a/tests/test_storage_database.py b/tests/test_storage_database.py new file mode 100644 index 0000000..9a25ac3 --- /dev/null +++ b/tests/test_storage_database.py @@ -0,0 +1,158 @@ +from datetime import date +import pytest +import duckdb +import pandas as pd +from argus.domain.internal_models import ( + DataSource, + Instrument, + MarketDataRequest, + MarketDataResponse, +) +from argus.storage.database import ( + initialize_database, + insert_price_bar, + read_price_bars, +) + + +@pytest.fixture +def sample_source(): + return DataSource( + name="Yahoo", provider_kind="yfinance_api", requires_api_key=False + ) + + +@pytest.fixture +def sample_instrument(): + return Instrument( + symbol="EUR/USD", + name="EUR - USD Rate", + asset_class="fx", + base_currency="EUR", + quote_currency="USD", + ) + + +@pytest.fixture +def sample_response(sample_source, sample_instrument): + test_bar = { + "timestamp": date(2026, 1, 1), + "open": None, + "high": None, + "low": None, + "close": 1.89, + "adjusted_close": None, + "volume": None, + } + return MarketDataResponse( + source=sample_source, + instrument=sample_instrument, + bars=pd.DataFrame(test_bar, index=[0]), + ) + + +@pytest.fixture +def db_path(tmp_path): + db = tmp_path / "test.duckdb" + initialize_database(db) + return db + + +def test_initialize_database_creates_required_tables(db_path): + connection = duckdb.connect(str(db_path)) + tables = connection.execute("SHOW TABLES;").fetchall() + connection.close() + table_names = {row[0] for row in tables} + + assert "data_sources" in table_names + assert "instruments" in table_names + assert "price_bars" in table_names + + +def test_data_is_inserted(db_path, sample_response) -> None: + insert_price_bar(db_path, sample_response) + + connection = duckdb.connect(str(db_path)) + + instrument_count = connection.execute( + "SELECT COUNT(*) FROM instruments;" + ).fetchone() + source_count = connection.execute("SELECT COUNT(*) FROM data_sources;").fetchone() + price_bar_count = connection.execute("SELECT COUNT(*) FROM price_bars;").fetchone() + + assert instrument_count is not None + assert source_count is not None + assert price_bar_count is not None + assert instrument_count[0] == 1 + assert source_count[0] == 1 + assert price_bar_count[0] == 1 + + +def test_fx_has_correct_format(db_path, sample_response) -> None: + insert_price_bar(db_path, sample_response) + + connection = duckdb.connect(str(db_path)) + try: + price_bar_fx = connection.execute("SELECT * FROM price_bars;").fetchone() + finally: + connection.close() + + assert price_bar_fx is not None + assert price_bar_fx[0] == 1 + assert price_bar_fx[1] == 1 + assert price_bar_fx[2] == 1 + assert price_bar_fx[3] == date(2026, 1, 1) + assert price_bar_fx[4] == 1.89 + assert price_bar_fx[5] is None + assert price_bar_fx[6] is None + assert price_bar_fx[7] is None + assert price_bar_fx[8] is None + assert price_bar_fx[9] is None + + +def test_duplicates_are_ignored(db_path, sample_response) -> None: + insert_price_bar(db_path, sample_response) + insert_price_bar(db_path, sample_response) # Erneuter Insert des Duplikats + + connection = duckdb.connect(str(db_path)) + try: + count = connection.execute("SELECT COUNT(*) FROM price_bars;").fetchone() + finally: + connection.close() + + assert count is not None + assert count[0] == 1 + + +def test_read_price_bars_returns_matching_data( + db_path, sample_source, sample_instrument, sample_response +) -> None: + req = MarketDataRequest( + source=sample_source, + instrument=sample_instrument, + timeframe="1d", + start=date(2026, 1, 1), + end=date(2026, 1, 1), + ) + insert_price_bar(db_path, sample_response) + + result = read_price_bars(db_path, req) + + assert len(result) == 1 + assert result.iloc[0]["close"] == 1.89 + + +def test_read_price_bars_returns_empty_dataframe_for_missing_range( + db_path, sample_source, sample_instrument +) -> None: + req = MarketDataRequest( + source=sample_source, + instrument=sample_instrument, + timeframe="1d", + start=date(2026, 1, 1), + end=date(2026, 1, 1), + ) + + result = read_price_bars(db_path, req) + + assert result.empty is True diff --git a/tests/test_timeseries_service.py b/tests/test_timeseries_service.py deleted file mode 100644 index 7dd3c9f..0000000 --- a/tests/test_timeseries_service.py +++ /dev/null @@ -1,35 +0,0 @@ -import pandas as pd -import pandas.testing as pdt -import numpy as np -from argus.services.timeseries_service import prepare_trend_analysis - - -def test_get_a_full_timeseries(): - test_curr = "EURUSD=X" - test_start = "2024-01-01" - test_end = "2024-01-04" - test_interval = "1d" - - expect_result = { - "date": ["2024-01-01", "2024-01-02", "2024-01-03"], - "rate": [1.1055831909179688, 1.1038745641708374, 1.0941756963729858], - "daily_pct_change": [np.nan, -0.1545452898675692, -0.8786204622023064], - "roll_avg": [1.1055831909179688, 1.104728877544403, 1.101211150487264], - } - expect_dict = { - "min_date": ["2024-01-03 00:00:00"], - "min_rate": [1.0941756963729858], - "max_date": ["2024-01-01 00:00:00"], - "max_rate": [1.1055831909179688], - } - result = prepare_trend_analysis(test_curr, test_start, test_end, test_interval) - if result is None: - return False - result_df, result_dict = result - result_df["date"] = result_df["date"].astype("str") - result_dict["min_date"] = [str(result_dict["min_date"][0])] - result_dict["max_date"] = [str(result_dict["max_date"][0])] - expect_df = pd.DataFrame(expect_result) - - pdt.assert_frame_equal(result_df, expect_df) - assert result_dict == expect_dict diff --git a/tests/test_trend_analysis_service.py b/tests/test_trend_analysis_service.py new file mode 100644 index 0000000..84f8ccd --- /dev/null +++ b/tests/test_trend_analysis_service.py @@ -0,0 +1,95 @@ +""" +import pytest +import pandas as pd +from datetime import date +from matplotlib.figure import Figure +from argus.domain.internal_models import ( + DataSource, + Instrument, + MarketDataRequest, + MarketDataResponse, +) +from argus.services.trend_analysis_service import prepare_trend_analysis +from argus.services.market_data_service import get_market_data +from argus.storage.database import initialize_database + + +@pytest.fixture +def sample_source(): + return DataSource( + name="Yahoo", provider_kind="yfinance_api", requires_api_key=False + ) + + +@pytest.fixture +def sample_instrument(): + return Instrument( + symbol="EUR/USD", + name="EUR - USD Rate", + asset_class="fx", + base_currency="EUR", + quote_currency="USD", + ) + + +@pytest.fixture +def sample_response(sample_source, sample_instrument): + test_bar = { + "timestamp": [date(2024, 1, 1), date(2024, 1, 2), date(2024, 1, 3)], + "open": [None, None, None], + "high": [None, None, None], + "low": [None, None, None], + "close": [1.10, 1.12, 1.11], + "adjusted_close": [None, None, None], + "volume": [None, None, None], + } + return MarketDataResponse( + source=sample_source, instrument=sample_instrument, bars=pd.DataFrame(test_bar) + ) + + +def test_prepare_trend_analysis_success( + sample_source, sample_instrument, sample_response, monkeypatch +): + req = MarketDataRequest( + source=sample_source, + instrument=sample_instrument, + timeframe="1d", + start=date(2024, 1, 1), + end=date(2024, 1, 4), + ) + + monkeypatch.setattr( + "argus.services.trend_analysis_service.get_market_data", + lambda db, r: sample_response, + ) + + res = prepare_trend_analysis("mock_db_path", req) + assert isinstance(res, Figure) + + +def test_prepare_trend_analysis_failure(sample_source, sample_instrument, monkeypatch): + req = MarketDataRequest( + source=sample_source, + instrument=sample_instrument, + timeframe="1d", + start=date(2024, 1, 1), + end=date(2024, 1, 4), + ) + + error_response = MarketDataResponse( + source=sample_source, + instrument=sample_instrument, + bars=pd.DataFrame(), + message="Quote not found or no data available for symbol", + ) + + monkeypatch.setattr( + "argus.services.trend_analysis_service.get_market_data", + lambda db, r: error_response, + ) + + res = prepare_trend_analysis("mock_db_path", req) + assert isinstance(res, str) + assert res == "Quote not found or no data available for symbol" +""" diff --git a/tests/test_yfinance_client.py b/tests/test_yfinance_client.py index faf15fc..419d999 100644 --- a/tests/test_yfinance_client.py +++ b/tests/test_yfinance_client.py @@ -1,80 +1,118 @@ from argus.clients.yfinance_client import get_timeseries +from argus.domain.internal_models import DataSource, Instrument, MarketDataRequest +from datetime import date +import pytest import pandas as pd import pandas.testing as pdt -def test_get_dataframe(monkeypatch): +@pytest.fixture +def sample_source(): + return DataSource( + name="Yahoo", provider_kind="yfinance_api", requires_api_key=False + ) + + +@pytest.fixture +def sample_instrument(): + return Instrument( + symbol="EUR/USD", + name="EUR - USD Rate", + asset_class="fx", + base_currency="EUR", + quote_currency="USD", + ) + + +def test_get_dataframe(monkeypatch, sample_source, sample_instrument): test_resp = pd.DataFrame( { + "Open": [None, None, None], + "High": [None, None, None], + "Low": [None, None, None], "Close": [1.105583, 1.103875, 1.094176], + "Adj Close": [None, None, None], + "Volume": [None, None, None], }, index=pd.to_datetime(["2024-01-01", "2024-01-02", "2024-01-03"]), ) test_resp.index.name = "Date" - test_curr = "EURUSD=X" - test_start = "2024-01-01" - test_end = "2024-01-04" - test_interval = "1d" + req = MarketDataRequest( + source=sample_source, + instrument=sample_instrument, + timeframe="1d", + start=date(2024, 1, 1), + end=date(2024, 1, 4), + ) def fake_yfinance_download(*args, **kwargs): return test_resp monkeypatch.setattr("yfinance.download", fake_yfinance_download) - - result = get_timeseries(test_curr, test_start, test_end, test_interval) + resp = get_timeseries(req) expected = pd.DataFrame( { - "date": pd.to_datetime(["2024-01-01", "2024-01-02", "2024-01-03"]), - "rate": [1.105583, 1.103875, 1.094176], + "timestamp": pd.to_datetime(["2024-01-01", "2024-01-02", "2024-01-03"]), + "open": [None, None, None], + "high": [None, None, None], + "low": [None, None, None], + "close": [1.105583, 1.103875, 1.094176], + "adjusted_close": [None, None, None], + "volume": [None, None, None], } ) - assert result is not None - pdt.assert_frame_equal(result, expected) + assert resp is not None + pdt.assert_frame_equal(resp, expected) -def test_get_none(monkeypatch): - test_curr = "EURUSD=X" - test_start = "2024-01-01" - test_end = "2024-01-04" - test_interval = "1d" - - def fake_yfinance_download(*args, **kwargs): - return None - monkeypatch.setattr("yfinance.download", fake_yfinance_download) +def test_client_network_error(monkeypatch, sample_source, sample_instrument): + req = MarketDataRequest( + source=sample_source, + instrument=sample_instrument, + timeframe="1d", + start=date(2024, 1, 1), + end=date(2024, 1, 4), + ) - result = get_timeseries(test_curr, test_start, test_end, test_interval) - assert result is None + def mock_crash(*args, **kwargs): + raise Exception() + monkeypatch.setattr("yfinance.download", mock_crash) -def test_get_empty_frame(monkeypatch): - test_curr = "EURUSD=X" - test_start = "2024-01-01" - test_end = "2024-01-01" - test_interval = "1d" + with pytest.raises(ConnectionError, match="Network error or connection timeout"): + get_timeseries(req) - def fake_yfinance_download(*args, **kwargs): - return pd.DataFrame() - monkeypatch.setattr("yfinance.download", fake_yfinance_download) +def test_client_invalid_response(monkeypatch, sample_source, sample_instrument): + req = MarketDataRequest( + source=sample_source, + instrument=sample_instrument, + timeframe="1d", + start=date(2024, 1, 1), + end=date(2024, 1, 4), + ) - result = get_timeseries(test_curr, test_start, test_end, test_interval) - assert result is None + monkeypatch.setattr("yfinance.download", lambda *args, **kwargs: None) + with pytest.raises( + ConnectionError, match="Yahoo Finance API returned an invalid response" + ): + get_timeseries(req) -def test_error_raise(monkeypatch): - test_curr = "EURUSD=X" - # start date is inclusiv and end date is exclusiv - the range 2024-01-01-2024-01-01 is not possible - test_start = "2024-01-04" - test_end = "2024-01-02" - test_interval = "1d" - def fake_yfinance_download( - tickers=test_curr, start=test_start, end=test_end, interval=test_interval - ): - return Exception("fake yfinance error") +def test_client_quote_not_found(monkeypatch, sample_source, sample_instrument): + req = MarketDataRequest( + source=sample_source, + instrument=sample_instrument, + timeframe="1d", + start=date(2024, 1, 1), + end=date(2024, 1, 4), + ) - monkeypatch.setattr("yfinance.download", fake_yfinance_download) + monkeypatch.setattr("yfinance.download", lambda *args, **kwargs: pd.DataFrame()) - result = get_timeseries(test_curr, test_start, test_end, test_interval) - assert result is None + with pytest.raises( + ValueError, match="Quote not found or no data available for symbol" + ): + get_timeseries(req)