Unity Machine Learning With Big Data
Published 2 years ago
Machine Learning with Big Data technologies.
This project is designed upon my passion in learning three concepts (Machine Learning, Gaming, Big Data). It took me three days 10 hours each day to fulfill the my passion. And yes I did it ! This project is built as follows:
  • Big Data Technologies:
  1. Apache Hadoop: Those who are familiar with Hadoop, knows that Hadoop can store enormous amount of data, thanks to its HDFS architecture.
  2. Apache Kafka: Kafka is a powerful messaging queue that can handle millions of data per second.
  3. Apache Storm: Used for processing data in near real time using its bolts.
  4. Apache Zookeeper: For managing the single node clusters I created (Hadoop, Kafka, Storm).
  5. Overview: Starting from definitions, I used Kafka to listen to a topic called "locationlearning" and this will queue the incoming data listening to this topic. I used Storm spout and 2 bolts, a bolt to read from this queue and send the results to the next bolt, that bolt in turn will store them in file locally as log, finally a third bolt will store them on Hadoop (HDFS). All this process done under the management of Zookeeper.
  • Machine Learning:
  1. Python/Django: A good backend-server to handle REST APIs I created.
  2. SKLearn: I used SVM(Supp(ort)/(er) vector machine) to build my training set and predict a given location checking if it is bad or good.
  3. Anaconda: I used it because of its built in python packages.
  4. Overview: I used SVM just to test my first project, I also covered it with a UI to view the data on a graph in real time. I started investigating about reinforcement learning and I found it is the best option for learning agents. The APIs I built using Django used to handle the communication between the game and the server(Django backend).
  • Gaming: My love and passion towards gaming is endless, like an infinite loop. Every time I see a game I start decoding it in my brain. I see the code and not the game. I see games like League of Legends, so I designed a simple prototype of it, same for MarioParty etc... The problem in me is that I hate to work alone on a project so most of m projects are incomplete.
  1. Unity: Of course I used Unity as a front-end as it is number one for gaming, so I can test my model.
Finally, my passion towards these topics as I said is big-passion like big-data :D. I want to try your new ML Unity, but I won't use it because I love to create my own AI algorithms ;).
Hassan Shaitou
M2 Researcher - Educator