spotify-voice-assistant/TEST_CASES.md

Voice Command Test Cases

Test suite for validating Spotify Voice Assistant behavior with natural language queries.

How to Use This

1. Say each command to your voice assistant
2. Note the actual response and whether it worked correctly
3. Use results to identify areas for improvement

Test Categories

1. Basic Artist Queries

Test 1.1: Simple artist request

• Command: "Play Coldplay"
• Expected: Searches for artist "Coldplay", plays on default speaker
• Result:
• Pass/Fail:

Test 1.2: Artist with speaker specified

• Command: "Play Coldplay on the kitchen speaker"
• Expected: Searches for artist "Coldplay", plays on kitchen speaker
• Result:
• Pass/Fail:

Test 1.3: Natural language artist request

• Command: "I'm in the mood for some Coldplay"
• Expected: Searches for artist "Coldplay", plays on default speaker
• Result:
• Pass/Fail:

Test 1.4: Artist with contextual description

• Command: "Play that British band with Chris Martin"
• Expected: LLM interprets as Coldplay, searches and plays
• Result:
• Pass/Fail:

---

2. Album Queries

Test 2.1: Specific album

• Command: "Play the album Parachutes"
• Expected: Searches for album "Parachutes", plays on default speaker
• Result:
• Pass/Fail:

Test 2.2: Album with artist

• Command: "Play Parachutes by Coldplay"
• Expected: Searches for album "Parachutes", plays correct album
• Result:
• Pass/Fail:

Test 2.3: Natural language album request

• Command: "Can you play the Parachutes album?"
• Expected: Searches for album "Parachutes", plays on default speaker
• Result:
• Pass/Fail:

Test 2.4: Album with speaker

• Command: "Play the album Parachutes on the gaming room speaker"
• Expected: Searches for album "Parachutes", plays on gaming room speaker
• Result:
• Pass/Fail:

Test 2.5: Album exact match

• Command: "Play Parachutes"
• Expected: Searches albums with limit=10, returns exact match for "Parachutes" (not "Parachutes Deluxe Edition")
• Result:
• Pass/Fail:

---

3. Track Queries

Test 3.1: Specific track

• Command: "Play Yellow"
• Expected: Searches for track "Yellow", plays on default speaker
• Result:
• Pass/Fail:

Test 3.2: Track with artist

• Command: "Play Yellow by Coldplay"
• Expected: Searches for track "Yellow", plays correct track
• Result:
• Pass/Fail:

Test 3.3: Track with speaker

• Command: "Play the song Yellow on the office speaker"
• Expected: Searches for track "Yellow", plays on office speaker
• Result:
• Pass/Fail:

Test 3.4: Natural language track request

• Command: "Put on that song Yellow"
• Expected: Searches for track "Yellow", plays on default speaker
• Result:
• Pass/Fail:

Test 3.5: Track exact match

• Command: "Play Yellow"
• Expected: Searches tracks with limit=10, returns exact match for "Yellow" (prioritizes exact name over similar songs)
• Result:
• Pass/Fail:

---

4. Ambiguous Track Names (Edge Cases)

Test 4.1: Common track name (no artist)

• Command: "Play Hurt"
• Expected: Searches for track "Hurt", returns most popular (likely Johnny Cash or Nine Inch Nails)
• Result:
• Pass/Fail:

Test 4.2: Common track name with artist

• Command: "Play Hurt by Nine Inch Nails"
• Expected: Searches for track "Hurt", LLM context helps return correct version
• Result:
• Pass/Fail:

Test 4.3: Common track name with different artist

• Command: "Play Hurt by Johnny Cash"
• Expected: Searches for track "Hurt", LLM context helps return Cash cover version
• Result:
• Pass/Fail:

---

5. Exact Match Testing (All Content Types)

Test 5.1: Exact artist name

• Command: "Play Coldplay"
• Expected: Returns exact match for artist "Coldplay" (not similar/recommended artists)
• Result:
• Pass/Fail:

Test 5.2: Similar artist name

• Command: "Play Cold War Kids"
• Expected: Returns exact match for "Cold War Kids" (not "Coldplay" despite similarity)
• Result:
• Pass/Fail:

Test 5.3: Artist with common name

• Command: "Play The Band"
• Expected: Returns exact match for artist "The Band" (not generic band recommendations)
• Result:
• Pass/Fail:

Test 5.4: Album exact match verification

• Command: "Play A Rush of Blood to the Head"
• Expected: Returns exact match for album name, not "Deluxe" or "Remastered" versions
• Result:
• Pass/Fail:

Test 5.5: Track exact match verification

• Command: "Play Fix You"
• Expected: Returns exact match for track "Fix You" from top 10 results
• Result:
• Pass/Fail:

---

6. Playlist Queries

Test 6.1: User playlist - exact match

• Command: "Play my workout playlist"
• Expected: Removes "playlist" from query, searches user's playlists for exact match "workout"
• Result:
• Pass/Fail:

Test 6.2: User playlist - partial match

• Command: "Play my running music"
• Expected: Checks user playlists first, finds partial match like "Running Music 2024"
• Result:
• Pass/Fail:

Test 6.3: User playlist - simple name

• Command: "Play vibes"
• Expected: Checks user playlists first for "Vibes", falls back to public playlists if not found
• Result:
• Pass/Fail:

Test 6.4: Public playlist fallback - exact match

• Command: "Play Today's Top Hits"
• Expected: Removes "playlist" if present, checks user playlists first, then finds in public playlists with exact match
• Result:
• Pass/Fail:

Test 6.5: Public playlist fallback - common name

• Command: "Play Chill Vibes"
• Expected: Checks user playlists first, then searches public playlists with exact match preference
• Result:
• Pass/Fail:

Test 6.6: Playlist not found anywhere

• Command: "Play xyz123 playlist"
• Expected: Searches user playlists, then public playlists, returns error if not found
• Result:
• Pass/Fail:

---

7. Queue Management

Test 7.1: Queue track

• Command: "Queue Wish You Were Here"
• Expected: Searches for track "Wish You Were Here", adds to queue without interrupting current playback
• Result:
• Pass/Fail:

Test 7.2: Add to queue with artist

• Command: "Add Yellow by Coldplay to the queue"
• Expected: Searches for track "Yellow", adds to queue
• Result:
• Pass/Fail:

Test 7.3: Queue album

• Command: "Add the album Dark Side of the Moon to queue"
• Expected: Searches for album "Dark Side of the Moon", adds to queue
• Result:
• Pass/Fail:

Test 7.4: Queue with speaker specified

• Command: "Queue some Pink Floyd on the kitchen speaker"
• Expected: Searches for artist "Pink Floyd", adds to queue on kitchen speaker
• Result:
• Pass/Fail:

Test 7.5: Natural language queue

• Command: "Queue up some Pink Floyd for later"
• Expected: Searches for artist "Pink Floyd", adds to queue
• Result:
• Pass/Fail:

---

8. Playback Control

Test 7.1: Pause

• Command: "Pause the music"
• Expected: Pauses current playback on active speaker
• Result:
• Pass/Fail:

Test 7.2: Resume

• Command: "Resume"
• Expected: Resumes playback on active speaker
• Result:
• Pass/Fail:

Test 7.3: Next track

• Command: "Skip to the next track"
• Expected: Advances to next track
• Result:
• Pass/Fail:

Test 7.4: Previous track

• Command: "Go back to the previous song"
• Expected: Returns to previous track
• Result:
• Pass/Fail:

Test 7.5: Volume up (natural language)

• Command: "Make it louder"
• Expected: Increases volume (LLM interprets as volume_set with higher level)
• Result:
• Pass/Fail:

Test 7.6: Volume specific

• Command: "Set volume to 50 percent"
• Expected: Sets volume to 50%
• Result:
• Pass/Fail:

Test 8.7: Stop

• Command: "Stop the music"
• Expected: Stops playback
• Result:
• Pass/Fail:

Test 8.8: Shuffle on

• Command: "Shuffle on"
• Expected: Enables shuffle mode
• Result:
• Pass/Fail:

Test 8.9: Shuffle off

• Command: "Turn shuffle off"
• Expected: Disables shuffle mode
• Result:
• Pass/Fail:

Test 8.10: Natural language shuffle

• Command: "Shuffle this"
• Expected: Enables shuffle mode
• Result:
• Pass/Fail:

---

9. Multi-Speaker Scenarios

Test 9.1: Different speaker each time

• Commands:
  • "Play Coldplay on the kitchen speaker"
  • "Play Radiohead on the gaming room speaker"
  • "Play Muse on the office speaker"
• Expected: Each plays on the specified speaker
• Result:
• Pass/Fail:

Test 9.2: No speaker specified (uses default)

• Command: "Play Coldplay"
• Expected: Plays on default speaker from system prompt
• Result:
• Pass/Fail:

---

10. Error Handling

Test 10.1: Non-existent artist

• Command: "Play XYZ123NotARealBand"
• Expected: Returns no results or best guess, handles gracefully
• Result:
• Pass/Fail:

Test 10.2: Typo in artist name

• Command: "Play Cold Play" (with space)
• Expected: Spotify's fuzzy search finds "Coldplay"
• Result:
• Pass/Fail:

Test 10.3: Invalid speaker

• Command: "Play Coldplay on the bedroom speaker" (if bedroom speaker doesn't exist)
• Expected: Error or fallback behavior
• Result:
• Pass/Fail:

---

11. Natural Language Variations

Test 11.1: Informal request

• Command: "Put on some Coldplay"
• Expected: Searches for artist "Coldplay", plays on default speaker
• Result:
• Pass/Fail:

Test 11.2: Question format

• Command: "Can you play Coldplay?"
• Expected: Searches for artist "Coldplay", plays on default speaker
• Result:
• Pass/Fail:

Test 11.3: Context-heavy request

• Command: "I want to listen to Coldplay right now"
• Expected: Searches for artist "Coldplay", plays on default speaker
• Result:
• Pass/Fail:

Test 11.4: Mood-based request

• Command: "Play something chill from Coldplay"
• Expected: Searches for artist "Coldplay", plays on default speaker
• Result:
• Pass/Fail:

---

Results Summary

Total Tests: 53 Passed: _____ Failed: _____ Pass Rate: _____%

Common Failure Patterns

(Document patterns that emerge from testing)

Recommendations for Improvement

(Based on test results, list specific enhancements needed)