# Data Sync Guide

# Data Sync Guide

Learn how to sync data from sports data providers into Sports Stack.

## Overview

Sports Stack integrates with multiple data providers (SportRadar, FTN, API Sports, etc.) to fetch and normalize sports data. This guide covers how to configure and trigger data synchronization.

## How Data Sync Works

```mermaid
flowchart LR
    A[CMS Sync Request] --> B[Oban Job Queue]
    B --> C[Provider API Call]
    C --> D[Feed File Created]
    D --> E[RabbitMQ Message]
    E --> F[Conversion Pipeline]
    F --> G[Database Storage]
```

### Sync Flow

1. **Trigger**: User clicks "Sync now" in CMS or automated cron job
2. **Job Creation**: Oban job created for each route/league combination
3. **API Call**: Provider makes HTTP request to external API
4. **Feed File**: Raw response saved as FeedFileEntity
5. **Message Queue**: Message published to RabbitMQ
6. **Conversion**: Data converted to Common Model format
7. **Storage**: Entities saved to database

## Setting Up Data Providers

### Step 1: Configure Provider

1. Navigate to **Integrations → Data Providers** in CMS
2. Select a provider (e.g., SportRadar, FTN)
3. Click **"Configure"**
4. Enter API credentials:
   * API Key
   * Base URL (if different from default)
   * Additional configuration options

### Step 2: Enable Provider

1. Toggle **"Active"** switch to enable provider
2. Select **leagues** you want to sync
3. Configure **fetch settings** (which entities to sync)
4. Save configuration

### Step 3: Test Sync

1. Navigate to **Sync Data** tab
2. Select a route (e.g., "Teams", "Events")
3. Choose a league (e.g., NFL, NBA)
4. Click **"Sync now"**
5. Monitor progress in **Recent API Calls**

## Sync Methods

### Manual Sync (CMS UI)

**Use Case**: One-time syncs, testing, troubleshooting

**Steps**:

1. Go to **Integrations → Data Providers → \[Provider]**
2. Click **Sync Data** tab
3. Select route and league
4. Click **"Sync now"** button
5. Wait for completion (check Recent API Calls)

**Example URLs**:

```
http://localhost:4001/integrations/data_providers/sportradar?tab=Sync+Data&league=nfl
http://localhost:4001/integrations/data_providers/sportradar?tab=Sync+Data&league=nba
```

### Quick Sync (Entity Pages)

**Use Case**: Sync data for specific entities (teams, players, events)

**Steps**:

1. Navigate to entity detail page (e.g., Team, Event)
2. Find **Quick Actions** section
3. Select provider and action (e.g., "Sync Team Players")
4. Click sync button
5. Monitor in Recent API Calls

**Available Actions**:

* **Teams**: Sync players, sync events
* **Events**: Sync statistics, sync player stats
* **Players**: Sync team roster

### Automated Sync (Cron Jobs)

**Use Case**: Regular scheduled syncs

**Configuration**:

1. Set up cron schedule in provider configuration
2. Configure fetch settings (which entities/leagues)
3. Provider automatically syncs on schedule

**Example**:

```json
{
  "cron": "0 */6 * * *",  // Every 6 hours
  "fetch": {
    "nfl": {
      "core": ["events", "teams"],
      "stats": ["event_scoped_stats"]
    }
  }
}
```

## Sync Routes

### Common Routes

| Route                 | Description              | Entities Created            |
| --------------------- | ------------------------ | --------------------------- |
| **Teams**             | Fetch team rosters       | `team`                      |
| **Players**           | Fetch player information | `player`                    |
| **Events**            | Fetch game schedules     | `event`                     |
| **Seasons**           | Fetch season information | `season`                    |
| **League Hierarchy**  | Fetch league structure   | `team`, `league`, divisions |
| **Statistics**        | Fetch game statistics    | `event_scoped_stats`        |
| **Player Statistics** | Fetch player stats       | `season_scoped_stats`       |

### Route Categories

Routes are organized by category in the CMS:

* **Baseline Reference**: Core entities (teams, players, leagues)
* **Data**: General data sync routes
* **Advanced Stats**: Statistical data and projections
* **Custom**: Provider-specific routes

## Sync Configuration

### Fetch Settings

Configure which entities to sync:

```json
{
  "fetch": {
    "nfl": {
      "core": ["teams", "players", "events"],
      "stats": ["event_scoped_stats", "season_scoped_stats"]
    },
    "nba": {
      "core": ["teams", "players"]
    }
  }
}
```

### Route Parameters

Some routes require additional parameters:

**Players Route**:

* `team_id`: Required - Team ID to fetch players for

**Events Route**:

* `season_year`: Optional - Season year (defaults to current)
* `week`: Optional - Week number

**Statistics Route**:

* `event_id`: Required - Event ID to fetch stats for

## Monitoring Syncs

### Recent API Calls

View sync status in **Integrations → Recent API Calls**:

* **Status**: Success (200) or error codes
* **Timestamp**: When sync occurred
* **Trace ID**: Link to detailed trace logs
* **Response**: Preview of response data

### Trace Logs

Each sync creates a trace with:

* API request/response details
* Conversion processing logs
* Error messages (if any)
* Processing time

**Access**: Click trace ID in Recent API Calls

## Common Sync Scenarios

### Initial League Setup

1. **Sync League Hierarchy**:
   * Creates league structure
   * Creates teams with divisions/conferences
   * Sets up team colors and metadata

2. **Sync Seasons**:
   * Creates season records
   * Links seasons to leagues

3. **Sync Teams** (if needed):
   * Updates team rosters
   * Creates/updates players

4. **Sync Events**:
   * Creates game schedules
   * Links events to seasons

### Daily Updates

1. **Sync Events** (scheduled):
   * Updates game schedules
   * Updates event statuses
   * Creates new events

2. **Sync Statistics** (after games):
   * Fetches game statistics
   * Updates player stats
   * Updates team stats

### Troubleshooting Syncs

1. **Check Provider Status**:
   * Verify provider is active
   * Check API credentials are valid
   * Verify network connectivity

2. **Review Trace Logs**:
   * Check for API errors
   * Review conversion logs
   * Identify processing issues

3. **Validate Results**:
   * Check entities were created
   * Verify data completeness
   * Compare with provider data

## Best Practices

### 1. Sync Order Matters

Sync in this order for best results:

1. **League Hierarchy** → Creates league structure
2. **Seasons** → Creates season records
3. **Teams** → Creates/updates teams
4. **Players** → Creates/updates players
5. **Events** → Creates game schedules
6. **Statistics** → Fetches game stats

### 2. Use Appropriate Routes

* **Baseline Reference**: Use for initial setup
* **Data**: Use for regular updates
* **Advanced Stats**: Use for statistical analysis

### 3. Monitor Sync Frequency

* **Reference Data**: Sync daily or weekly
* **Event Data**: Sync multiple times per day
* **Live Data**: Sync every 30-60 seconds during games
* **Statistics**: Sync after games complete

### 4. Handle Errors Gracefully

* Check Recent API Calls for failures
* Review trace logs for details
* Retry failed syncs manually
* Contact support for persistent issues

## Troubleshooting

### Sync Not Starting

1. **Check Provider Configuration**:
   * Verify provider is active
   * Check API credentials
   * Verify league is selected

2. **Check Oban Queue**:
   * Verify Oban is running
   * Check queue for stuck jobs
   * Review Oban dashboard

### Sync Completing But No Data

1. **Check Conversion Logs**:
   * Review trace logs
   * Check for conversion errors
   * Verify processor matching

2. **Validate Data**:
   * Check entities in Explore
   * Verify tenant filtering
   * Check entity filters

### Rate Limiting

1. **Reduce Sync Frequency**:
   * Increase time between syncs
   * Sync fewer entities at once
   * Use cron scheduling

2. **Contact Support**:
   * Request rate limit increase
   * Optimize sync patterns

## Related Documentation

* [Testing Guide](./testing-guide.md) - How to test syncs
* [API Reference](../api-reference/) - API endpoint documentation
* [Webhooks Guide](./webhooks-guide.md) - Real-time notifications

## Support

For sync issues:

1. Check **Recent API Calls** for errors
2. Review **Trace Logs** for details
3. Check **Provider Configuration** status
4. Contact support with trace IDs