Develop

docs/databases-and-storage.md

@@ -0,0 +1,193 @@
+# ARGUS Storage Research
+
+## Intro
+
+It was previously a research what ARGUS should store and which database/storage approach fits the project.
+Now it's a simple documentation about how the storage and domain logic looks like and should develop in the next sprints.
+
+ARGUS is moving from live API requests and in-memory analytics toward real data workflows.  
+The first storage decision should support local market analytics, SQL practice and future dashboard features without adding unnecessary infrastructure too early.
+
+---
+
+## First Storage Approach
+
+DuckDB should be the first storage technology for ARGUS.
+
+Reason:
+
+* ARGUS currently needs local analytical storage, not a full server database
+* DuckDB fits historical time-series analysis well
+* it supports SQL-based analytics without requiring a database server
+* it works well with Python and notebook-based exploration
+* it keeps the first storage implementation manageable
+* it can later be replaced or complemented by PostgreSQL if ARGUS becomes more product-like
+
+The first storage implementation should focus on:
+
+* historical market data
+* cleaned OHLCV-ready price data
+* source information
+* instruments that ARGUS can analyze
+
+PostgreSQL and SQLGate become more relevant later.
+
+For the first DuckDB phase, the goal is to build a clean local analytics workflow.
+
+---
+
+## Developer Interaction Workflow
+
+ARGUS should use a practical developer workflow for DuckDB.
+
+The goal is to make the database easy to inspect, explore and validate before logic is moved into production code.
+
+### Notebook Exploration
+
+Notebooks should be the main exploration layer.
+
+They are useful for:
+
+* opening the DuckDB database
+* testing SQL queries
+* validating imported data
+* comparing SQL results with pandas calculations
+* exploring metric logic
+* documenting research assumptions
+
+This workflow is especially useful before turning queries into reusable project code.
+
+Notebook exploration should be preferred over a GUI database tool in the first phase.
+
+### DuckDB CLI
+
+The DuckDB CLI should be used for quick database inspection.
+
+It is useful for:
+
+* checking available tables
+* running small SQL queries
+* validating stored records
+* debugging the local database file
+
+The CLI is not the main research environment, but it is useful as a fast inspection tool.
+
+---
+
+## First Data Model Direction
+
+The first data model should support FX data now and broader market data later.
+
+ARGUS should not use a narrow `date | value` table as the main market-data model.
+
+That would work for simple exchange rates, but it would become limiting once ARGUS adds stocks, ETFs, indices or broader market APIs.
+
+The first model focuses on two primary metadata entities, supported by standard operational schemas and communication interfaces:
+
+```text
+DataSource
+Instrument
+```
+
+### DataSource
+
+Stores where data came from.
+
+Current operational fields:
+
+```text
+name
+provider_kind
+requires_api_key (defaults to False)
+```
+
+### Instrument
+
+Stores what ARGUS can analyze.
+Current operational fields:
+
+```text
+symbol
+name
+asset_class
+currency (optional)
+exchange (optional)
+base_currency (optional)
+quote_currency (optional)
+```
+
+### Internal Operational Models
+
+To support decoupled runtime communication and execution pipelines, ARGUS utilizes specialized Python dataclasses. These structures orchestrate active data flows between data-fetching clients and services.
+
+**MarketDataRequest**
+Encapsulates parameters passed to fetching engines and downstream query handlers.
+
+* `source`: The targeted `DataSource` instance
+* `instrument`: The targeted `Instrument` instance
+* `timeframe`: Granularity of requested bars (like `"1d"`, `"1h"`).
+* `start`: Temporal lower bound (`datetime.date`)
+* `end`: Temporal upper bound (`datetime.date`)
+
+**MarketDataResponse**
+Encapsulates runtime data payloads, structural enforcement, and pipeline feedback.
+
+* `source`: The originating `DataSource` instance
+* `instrument`: The corresponding `Instrument` instance
+* `bars`: A `pandas.DataFrame` holding the time-series payload
+* `message`: Pipeline feedback or error context (defaults to `""`)
+
+### The Price Bar Schema (Data Uniformity)
+
+Crucially, **Price Bars are not treated as a standalone domain metadata model or database entity.** Instead, the Price Bar specification acts purely as an internal **uniformity schema** to normalize incoming time-series data across disparate third-party APIs.
+
+All responses must be mapped into this standard schema layout within the `bars` DataFrame.
+
+#### Provider Mapping Layer
+
+Third-party structures are converted to this unified standard upon ingest. For example, `yfinance` fields map to the uniform schema via `YFINANCE_PRICE_BAR_MAPPING`:
+
+* `Date` $\rightarrow$ `timestamp`
+* `Open` $\rightarrow$ `open`
+* `High` $\rightarrow$ `high`
+* `Low` $\rightarrow$ `low`
+* `Close` $\rightarrow$ `close`
+* `Adj Close` $\rightarrow$ `adjusted_close`
+* `Volume` $\rightarrow$ `volume`
+
+---
+
+## Future Direction
+
+Later sprints can expand the storage layer step by step.
+
+Possible later additions:
+
+| Future Area | Possible Additions |
+| --- | --- |
+| Better source mapping | source-specific symbols, provider metadata |
+| Watchlists | user-selected instruments |
+| Reports | generated report metadata and history |
+| Macro data | FRED indicators and observations |
+| Paper trading | simulated orders, positions and portfolio history |
+| Server architecture | PostgreSQL |
+| SQL tooling | SQLGate with PostgreSQL |
+| Cloud direction | managed PostgreSQL or cloud storage |
+
+SQLGate should be kept for a later PostgreSQL phase.
+
+It becomes useful when ARGUS moves toward:
+
+* server-based storage
+* stronger database management
+* richer metadata
+* more stable application state
+* user-facing features
+* report history
+* cloud-ready architecture
+
+Additional metadata such as documentation links, terms links or provider governance fields can also become useful later.
+
+For the first DuckDB phase, these details should stay in research documentation instead of the database schema.
+
+---

pyproject.toml

@@ -9,6 +9,7 @@ dependencies = [
     "numpy",
     "matplotlib",
     "yfinance",
+    "duckdb",
 ]
 
 [project.optional-dependencies]

src/argus/analytics/charts/trend_chart.py

@@ -1,8 +1,8 @@
 import matplotlib.pyplot as plt
-from argus.services.timeseries_service import prepare_trend_analysis
+import pandas as pd
 
 
-def create_trendchart(curr_symbol: str, start: str, end: str, interval: str):
+def create_trendchart(df: pd.DataFrame, min_max_rates: dict):
     """
     Create a trend chart for exchange-rate analysis.
 
@@ -30,10 +30,6 @@ def create_trendchart(curr_symbol: str, start: str, end: str, interval: str):
         Minimum and maximum exchange-rate values are marked with scatter
         points and annotations.
     """
-    result = prepare_trend_analysis(curr_symbol, start, end, interval)
-    if result is None:
-        return None
-    df, min_max_rates = result
     min_date = min_max_rates["min_date"][0]
     min_rate = min_max_rates["min_rate"][0]
     max_date = min_max_rates["max_date"][0]

src/argus/clients/yfinance_client.py

@@ -1,43 +1,48 @@
 import yfinance as yf
-import logging
-
-
-def get_timeseries(curr_symbol, start, end, interval):
-    """
-    Fetch historical exchange-rate time series data from Yahoo Finance.
-
-    Args:
-        curr_symbol (str): Currency symbol used by Yahoo Finance, for example
-            "EURUSD=X".
-        start (str): Start date of the requested time range in YYYY-MM-DD format.
-        end (str): End date of the requested time range in YYYY-MM-DD format.
-        interval (str): Data interval supported by Yahoo Finance, for example
-            "1d", "1h", or "15m".
-
-    Returns:
-        pandas.DataFrame | None: A DataFrame containing the columns ``date`` and
-        ``rate`` if data was successfully fetched. Returns ``None`` if the
-        request fails, returns no data, or an exception occurs.
-    """
+import pandas as pd
+from argus.domain.internal_models import (
+    MarketDataRequest,
+    PRICE_BAR_COLUMNS,
+    YFINANCE_PRICE_BAR_MAPPING,
+)
+
+
+def get_timeseries(request: MarketDataRequest) -> pd.DataFrame:
+    start = str(request.start)
+    end = str(request.end)
+    timeframe = request.timeframe
+    curr_pair = (
+        f"{request.instrument.base_currency}{request.instrument.quote_currency}=X"
+    )
+
     try:
-        yf_logger = logging.getLogger("yfinance")
-        yf_logger.disabled = True
-        data = yf.download(
-            tickers=curr_symbol,
+        raw_resp = yf.download(
+            tickers=curr_pair,
             start=start,
             end=end,
-            interval=interval,
+            interval=timeframe,
             multi_level_index=False,
             progress=False,
         )
-        yf_logger.disabled = False
-        if data is None:
-            return None
-        if data.empty:
-            return None
-        data = data.reset_index()
-        data = data[["Date", "Close"]]
-        data = data.rename(columns={"Date": "date", "Close": "rate"})
-        return data
     except Exception:
-        return None
+        raise ConnectionError("Network error or connection timeout")
+
+    if raw_resp is None:
+        raise ConnectionError("Yahoo Finance API returned an invalid response")
+
+    if (
+        raw_resp.empty
+        or "Close" not in raw_resp.columns
+        or raw_resp["Close"].dropna().empty
+    ):
+        raise ValueError("Quote not found or no data available for symbol")
+
+    resp = normalize_yfinance_bars(raw_resp)
+    return resp
+
+
+def normalize_yfinance_bars(raw_df: pd.DataFrame) -> pd.DataFrame:
+    df = raw_df.copy()
+    df = df.reset_index()
+    df = df.rename(columns=YFINANCE_PRICE_BAR_MAPPING)
+    return df[list(PRICE_BAR_COLUMNS)]

src/argus/domain/internal_models.py

@@ -0,0 +1,71 @@
+from dataclasses import dataclass
+from datetime import date
+import pandas as pd
+
+
+@dataclass
+class DataSource:
+    name: str
+    provider_kind: str
+    requires_api_key: bool = False
+
+
+@dataclass
+class Instrument:
+    symbol: str
+    name: str
+    asset_class: str
+    currency: str | None = None
+    exchange: str | None = None
+    base_currency: str | None = None
+    quote_currency: str | None = None
+
+
+PRICE_BAR_COLUMNS = (
+    "timestamp",
+    "open",
+    "high",
+    "low",
+    "close",
+    "adjusted_close",
+    "volume",
+)
+
+YFINANCE_PRICE_BAR_MAPPING = {
+    "Date": "timestamp",
+    "Open": "open",
+    "High": "high",
+    "Low": "low",
+    "Close": "close",
+    "Adj Close": "adjusted_close",
+    "Volume": "volume",
+}
+
+
+@dataclass
+class MarketDataRequest:
+    source: DataSource
+    instrument: Instrument
+    timeframe: str
+    start: date
+    end: date
+
+
+@dataclass
+class MarketDataResponse:
+    source: DataSource
+    instrument: Instrument
+    bars: pd.DataFrame
+    message: str = ""
+
+    def __post_init__(self) -> None:
+        if not isinstance(self.bars, pd.DataFrame):
+            raise TypeError("bars must be a pandas DataFrame")
+        if self.message == "":
+            missing_cols = [
+                col for col in PRICE_BAR_COLUMNS if col not in self.bars.columns
+            ]
+            if missing_cols:
+                raise ValueError(
+                    f"Missing required columns in bars DataFrame: {missing_cols}"
+                )