S̶i̶n̶k̶ ̶o̶r̶ Swim
Updated 2 years ago
2.0 K
Virtual Reality Language Learning (updates at the bottom)

Cannonball! - Introduction

"S̶i̶n̶k̶ ̶o̶r̶ Swim" is an adventure virtual reality game where the player learns a target language through total immersion. You are Alex, a child who has crash landed on the beach of a foreign country that speaks a language you do not know. From the beginning of the game, you are introduced to a family that only speaks their target language so now that you are in this Sink or Swim situation, you must learn to swim by learning to communicate with them by hearing what they say, looking at their actions, and interacting with them. In no time at all you will find yourself communicating just swimmingly.


The players is Alex, a child that has crash landed on a beach in a foreign country where everyone speaks an unknown language and is now in deep water. Alex's first interaction upon showing up on the beach is a cheery little girl that immediately takes to Alex, becoming the first friend. Though this new friend speaks a language unknown to Alex, that does not stop her from expressing what she wants and bringing her new friend home. Alex is then introduced to the rest of the family and quickly gets integrated into their everyday life. Each day contains its own set of challenges requiring Alex to learn the new words and phrases to keep up with the events of everyday life. Over time, without even knowing, Alex begins to understand and be able to converse with the family and others around.

A Deep Dive into the Gameplay Mechanics

The player is capable of interacting with the environment and other characters in multiple ways. The full immersive experience includes not only listening and watching the other characters but taking part in the events as one would in real life. While there are multiple options for the player to do, VR allows for natural interactions that one would do in their daily life. Below is a list of the gameplay mechanics from basic to more advanced:
  • Movement - As with most games, the player is able to control their characters movements to go where they wish to go and see what they wish to see.
  • Manipulating Objects - The player is able to pick up, control, move, drop, throw and point at most objects in the game. While not all objects can have all of these done, most have at least some of the functions listed above.
  • Converse with characters - The player can progress the story through dialogue. All dialogue only takes place in the target language.
  • Inquiry about Objects - The player is capable of selecting objects simply by holding them or pointing at them and asking questions such as "What is that?" or "How do I say this?" followed by the response of a nearby character.
  • Requests to Characters - The player can also make requests of character such as "Please say that again" and characters will do as requested. This is of course limited to certain requests.
  • Progress Tracking - The player's progress is tracked so they can go back to revisit words or phrases that they have learned or share their progress with their friends.

How to keep your head above water - The Language Learning Process

Immersion. "S̶i̶n̶k̶ ̶o̶r̶ Swim", at its core, is a full immersion learning type game. What does full immersion mean? Do a search engine lookup for the best way to learn a new language fast and you'll find that the number one way to learn a language is to travel to a location that speaks that language and participate in the society completely, using ONLY their language. This Sink or Swim situation is called full immersion. The significance of this is that the learner must find ways of communicating and learning from the environment and people around, using ONLY the target language. That's what we do here. The starting language is irrelevant, all that matters is the target language because the game is "taught" (played) only in the target language. Most people don't have either the time or money to travel to another country to learn their language. "S̶i̶n̶k̶ ̶o̶r̶ Swim" provides the opportunity to experience being in the foreign land and speaking the language with some "locals". Every word, phrase, and grammatical structure is learned through context and repeated through other context causing the player to make connections relating the learned material to the actions or environment they see. By engaging the brain with multiple senses and giving them interactions that are both memorable and relatable, they are more likely to remember the material long term.
No Translation. One of the key elements to "S̶i̶n̶k̶ ̶o̶r̶ Swim" is the fact that there is no translation. We focus on learning the vocabulary and grammar intuitively as we do for majority of the vocabulary of our first language. Many language learning software use one language to teach another, resulting in the user relating the new word to the word in their original language. This is by far the easiest way to study but the hardest way to get lasting results. Learning a language by translating one language to another creates three problems:
  1. The first problem this creates is that the new terms are stored short term. What we remember most is whatever we can relate to our senses and emotions, the more senses something is tied to, the easier it is to remember. Translation relates a new word only to one sense and no emotion (besides possible frustration).
  2. The second problem is that it limits our understanding of the word in the other language. By relating the new word to the word in the original language we assume that these two words are equivalent, however, often times words in other languages will have a slightly different or deeper meaning than that of the original language. As soon as we translate the word we are possibly limiting our understanding of a word, giving us a rudimentary understanding of the language.
  3. The last important issue created by translation is that it makes our response time much longer. If we learn a new word in a language and translate it to our original language then the next time we hear that word, we will first translate it to the original language before understanding the meaning. If someone is learning Chinese and their first language is English, the response for hearing the word "Hello" in Chinese looks like this "你好"->"hello"->idea of greeting. Whereas if we learn without using translation it looks like this "你好“->idea of greeting, skipping one step in the thought process which makes achieving fluency much faster. This really starts to show when we look at the process of listen and respond: "你好吗?"->"How are you?" ->understanding of greeting and response ->"I am fine" ->"我很好" has 5 steps which shortens to 3 if we take out the intermediate step required for translation. This way we can have a smoother and more organic conversation.
Therefore, "S̶i̶n̶k̶ ̶o̶r̶ Swim" uses no translation in order to allow the player to get the most out of it, get a better understanding of what's learned, and recall what has been learned due of a deeper, efficient, and practical learning experience.
Natural Learning. The most boring part of studying a new language is flipping through countless note cards, trying to recall the definition, only to realize that last weeks easy word is long forgotten now. While repetition is a necessary part of the language learning process, it can sometimes make the player feel like they're in over their head. The natural way that we learn our first language is simply by hearing and seeing the word used in multiple scenarios and making connections in our brain to the different ways these were used to come up with a definitive method of using the word, phrase or even grammatical structure. Seeing and hearing the same words and phrases in different contexts allows the player to recognize patterns and quickly be able to understand the language, not memorize it. This is how the brain works and by working with the brain in the natural way it learns, we can learn in way that we are going to remember. The way we accomplish this is by initially using the word in an isolated and easy to understand situation, then repeating the word that was learned previously in a similar new context that is recognizable.
Spaced Repetition. There are many popular language learning applications today, Rosetta Stone, Berlitz, Pimsleur, Instant Immersion are just a few. One thing that they all have in common is that they all use some form of algorithm to decide when to repeat each word. The significance to this is that by "reminding" the brain of a word at certain increasing intervals, it will remember the word for longer periods of time until it is a word that is automatically recognized by the brain. This is a scientific process that was discovered in the early 1900's and has been researched and implemented in many learning applications. "S̶i̶n̶k̶ ̶o̶r̶ Swim" also implements spaced repetition in order to ensure that words are learned and remembered in the most efficient manner.
Interactivity. With the game being in Virtual Reality, it allows us to create a more immersive experience. Not only is everything the player see the game and everything you hear the language but they also have the capability of doing actions that we take for granted in our everyday life, asking questions like "How do I say this?" or "What is this called?". These are perfectly normal questions in a real life scenario that someone may ask in another country so we made sure to allow the player the opportunity to pick up or point at objects they wish to ask about, learning the language in the most normal and natural of methods.

Tools and Features used

The core of the game was built by us, dialogue system, NPC interaction (conversation handling between NPC and player), environment interaction (conversation handling player requesting information about the environment), progress tracking (keeping track of what words the player has learned either through the scripted story or through their own inquiries), level design, story writing, voice acting, and more. The following are the tools and features we used:
  • Unity 2018.2 - project creation and assembly
  • Daz3d - character creation
  • IBM Watson SDK - speech recognition, translation, smart conversation (watson conversation)
  • 3D EVERYTHING (multiple packs) - House interior models
  • NewtonVR - handling VR controls
Unity Features
  • Unity Windows keyword recognition
  • Timeline and Cinemachine - certain scripted sequences

Going Off The Deep End - What's Next?

Now that we have done this, there are several things we could do to add to the overall experience of the game. Currently, there are two language that can be learned, English and Chinese. The systems we created were designed to be used with any language, meaning we can easily extend the game to teach other languages simply by creating the voice recordings in the other languages. This is one way we would like to add to this game.
Another addition we would like to make is adding more "levels" (there are no levels but scenario) for intermediate and advanced learners. We currently only have a beginner level scenario implemented. We would like to add some more beginner level scenarios as well as some intermediate and advanced level ones.
The biggest addition we would like for the game is having an online mode, with different "servers" for speaking certain languages. We have some ideas for this that provide benefits to players that help others learn their language on their own servers as well as benefits to those who travel to other servers to practice their skills.

Don't be afraid, Let's dive right in to this ocean of possibility. Last one in is a rotten egg!

Don't forget to like the page if you want to see more. If you see a swimming pun opportunity I missed, leave it in the comments and I might add it!


January 21, 2019

The structure of our program allows for us to easily change the language to be learned so that virtually any language can be recognized, limited only by IBM Watson, our speech to text API. In order to add any language to our game we simply need to have the voice acting to do it, the rest handles itself. We have finished testing the English learning part of the game and that will be done once we finish the recording of the voice acting. After that, we will complete the Chinese (Mandarin) learning voices and test it.

January 31, 2019

The game is being uploaded to Enjoy!
Brandon Hansford
Ke Xu
Student - Student
Brandon Hansford
2 years ago
EldonHave you tested conversation where people have accents? How about in an environment where it may be noisy, would the NPC be confused on who it should listen to? I briefly worked in a VR company for a few months and also had a learning language game and these two issues were very difficult for us. By the way, will this be on the Oculus GO? I would like to try it out but I don't have a vive :(
Right now i'm only developing on the Vive but I don't intend to stop here. With the way it works it does need to be recognizable as the word or phrase they are trying to say, so if the accent is strong then it may not be recognized and the player may need to try again. I have tested this with some non natives who have foreign accents and they made it through fine. To your second question, noise is not much of an issue but if someone in the background says something that the game recognizes loud enough then it will react. The solution is to adjust the microphone settings. We have only had this be a problem when someone was intentionally trying to mess up the program.
2 years ago
VR Developer
Have you tested conversation where people have accents? How about in an environment where it may be noisy, would the NPC be confused on who it should listen to? I briefly worked in a VR company for a few months and also had a learning language game and these two issues were very difficult for us. By the way, will this be on the Oculus GO? I would like to try it out but I don't have a vive :(
Cal C
2 years ago
Has very detailed setting and models. Great concept to learning a new language! Never would guess IBM Watson could be used for this!
S Richter
2 years ago
Very impressive concept and idea. Looking forward to hearing and seeing more. Chinese language learner myself and this would certainly be something I would want to try.
Julio Ortega
2 years ago
Good Idea. No better way to learn a language than in the context of immersive explorations and games.