Data Sync Guide

Learn how to sync data from sports data providers into Sports Stack.

Overview

Sports Stack integrates with multiple data providers (SportRadar, FTN, API Sports, etc.) to fetch and normalize sports data. This guide covers how to configure and trigger data synchronization.

How Data Sync Works

flowchart LR
    A[CMS Sync Request] --> B[Oban Job Queue]
    B --> C[Provider API Call]
    C --> D[Feed File Created]
    D --> E[RabbitMQ Message]
    E --> F[Conversion Pipeline]
    F --> G[Database Storage]

Sync Flow

1. Trigger: User clicks "Sync now" in CMS or automated cron job
2. Job Creation: Oban job created for each route/league combination
3. API Call: Provider makes HTTP request to external API
4. Feed File: Raw response saved as FeedFileEntity
5. Message Queue: Message published to RabbitMQ
6. Conversion: Data converted to Common Model format
7. Storage: Entities saved to database

Setting Up Data Providers

Step 1: Configure Provider

1. Navigate to Integrations → Data Providers in CMS
2. Select a provider (e.g., SportRadar, FTN)
3. Click "Configure"
4. Enter API credentials:
   • API Key
   • Base URL (if different from default)
   • Additional configuration options

Step 2: Enable Provider

1. Toggle "Active" switch to enable provider
2. Select leagues you want to sync
3. Configure fetch settings (which entities to sync)
4. Save configuration

Step 3: Test Sync

1. Navigate to Sync Data tab
2. Select a route (e.g., "Teams", "Events")
3. Choose a league (e.g., NFL, NBA)
4. Click "Sync now"
5. Monitor progress in Recent API Calls

Sync Methods

Manual Sync (CMS UI)

Use Case: One-time syncs, testing, troubleshooting

Steps:

1. Go to Integrations → Data Providers → [Provider]
2. Click Sync Data tab
3. Select route and league
4. Click "Sync now" button
5. Wait for completion (check Recent API Calls)

Example URLs:

http://localhost:4001/integrations/data_providers/sportradar?tab=Sync+Data&league=nfl
http://localhost:4001/integrations/data_providers/sportradar?tab=Sync+Data&league=nba

Quick Sync (Entity Pages)

Use Case: Sync data for specific entities (teams, players, events)

Steps:

1. Navigate to entity detail page (e.g., Team, Event)
2. Find Quick Actions section
3. Select provider and action (e.g., "Sync Team Players")
4. Click sync button
5. Monitor in Recent API Calls

Available Actions:

• Teams: Sync players, sync events
• Events: Sync statistics, sync player stats
• Players: Sync team roster

Automated Sync (Cron Jobs)

Use Case: Regular scheduled syncs

Configuration:

1. Set up cron schedule in provider configuration
2. Configure fetch settings (which entities/leagues)
3. Provider automatically syncs on schedule

Example:

{
  "cron": "0 */6 * * *",  // Every 6 hours
  "fetch": {
    "nfl": {
      "core": ["events", "teams"],
      "stats": ["event_scoped_stats"]
    }
  }
}

Sync Routes

Common Routes

Route Categories

Routes are organized by category in the CMS:

• Baseline Reference: Core entities (teams, players, leagues)
• Data: General data sync routes
• Advanced Stats: Statistical data and projections
• Custom: Provider-specific routes

Sync Configuration

Fetch Settings

Configure which entities to sync:

{
  "fetch": {
    "nfl": {
      "core": ["teams", "players", "events"],
      "stats": ["event_scoped_stats", "season_scoped_stats"]
    },
    "nba": {
      "core": ["teams", "players"]
    }
  }
}

Route Parameters

Some routes require additional parameters:

Players Route:

• team_id: Required - Team ID to fetch players for

Events Route:

• season_year: Optional - Season year (defaults to current)
• week: Optional - Week number

Statistics Route:

• event_id: Required - Event ID to fetch stats for

Monitoring Syncs

Recent API Calls

View sync status in Integrations → Recent API Calls:

• Status: Success (200) or error codes
• Timestamp: When sync occurred
• Trace ID: Link to detailed trace logs
• Response: Preview of response data

Trace Logs

Each sync creates a trace with:

• API request/response details
• Conversion processing logs
• Error messages (if any)
• Processing time

Access: Click trace ID in Recent API Calls

Common Sync Scenarios

Initial League Setup

1. Sync League Hierarchy:

   • Creates league structure
   • Creates teams with divisions/conferences
   • Sets up team colors and metadata

2. Sync Seasons:

   • Creates season records
   • Links seasons to leagues

3. Sync Teams (if needed):

   • Updates team rosters
   • Creates/updates players

4. Sync Events:

   • Creates game schedules
   • Links events to seasons

Daily Updates

1. Sync Events (scheduled):

   • Updates game schedules
   • Updates event statuses
   • Creates new events

2. Sync Statistics (after games):

   • Fetches game statistics
   • Updates player stats
   • Updates team stats

Troubleshooting Syncs

1. Check Provider Status:

   • Verify provider is active
   • Check API credentials are valid
   • Verify network connectivity

2. Review Trace Logs:

   • Check for API errors
   • Review conversion logs
   • Identify processing issues

3. Validate Results:

   • Check entities were created
   • Verify data completeness
   • Compare with provider data

Best Practices

1. Sync Order Matters

Sync in this order for best results:

1. League Hierarchy → Creates league structure
2. Seasons → Creates season records
3. Teams → Creates/updates teams
4. Players → Creates/updates players
5. Events → Creates game schedules
6. Statistics → Fetches game stats

2. Use Appropriate Routes

• Baseline Reference: Use for initial setup
• Data: Use for regular updates
• Advanced Stats: Use for statistical analysis

3. Monitor Sync Frequency

• Reference Data: Sync daily or weekly
• Event Data: Sync multiple times per day
• Live Data: Sync every 30-60 seconds during games
• Statistics: Sync after games complete

4. Handle Errors Gracefully

• Check Recent API Calls for failures
• Review trace logs for details
• Retry failed syncs manually
• Contact support for persistent issues

Troubleshooting

Sync Not Starting

1. Check Provider Configuration:

   • Verify provider is active
   • Check API credentials
   • Verify league is selected

2. Check Oban Queue:

   • Verify Oban is running
   • Check queue for stuck jobs
   • Review Oban dashboard

Sync Completing But No Data

1. Check Conversion Logs:

   • Review trace logs
   • Check for conversion errors
   • Verify processor matching

2. Validate Data:

   • Check entities in Explore
   • Verify tenant filtering
   • Check entity filters

Rate Limiting

1. Reduce Sync Frequency:

   • Increase time between syncs
   • Sync fewer entities at once
   • Use cron scheduling

2. Contact Support:

   • Request rate limit increase
   • Optimize sync patterns

Related Documentation

• Testing Guide - How to test syncs
• API Reference - API endpoint documentation
• Webhooks Guide - Real-time notifications

Support

For sync issues:

1. Check Recent API Calls for errors
2. Review Trace Logs for details
3. Check Provider Configuration status
4. Contact support with trace IDs