Education on a Map

Ankit Sanghvi -
It's like "a decentralized insti fundae session right ?"
~ Abhiram Shenoi

The journey for the CourseMap platform began with a simple idea -> What if courses in college are actually like nodes on a graph that you can study the interactions between ?

It has been a crazy journey of building this product from ground-zero to support a 1700 student base for now, with a vision to go much higher & further than that

The idea

It all started the day I got rejected from a Computer Networking course on a minor complication of course allotment protocols in place. It just hit me that organized information of courses in their true nature - as being linked entities was much required owing to 2 reasons :

  1. The massive plethora of options in courses that students can explore & pursue in college. Also, the fact that how each course interacts & compares with others in terms of prerequisite matching, difficulty, slot clashes
  2. The distributed nature of skillsets that most people acquire from multiple sources, that leads to the creation of their roadmap as a collection of steps taken over years in college.

I read "The Mom Test" as my reference (because Eric said so......) & interviewed around 40 people in total, mainly from my college

Hence, was born the idea for a platform that would treat each course as a node in a graph to begin with, and build on top of this presumption to eventually proceed towards better networks between students, professors & projects.

Getting the Data

It was a problem I said I would solve by providing structured information - the only flaw in the plan - I didn't even have unstructured information to begin with😆

I remember that the first approach I had was interviewing people, asking them to share their course details, then scribbling that in a notebook, going on academic.iitm.ac.in, & actually then manually relating courses to one another in my own notion db

What seemed like a stupid idea, kept me going strong to setup a backend, come up with a simple way to interact with Notion's API & support the entire architecture just on the presumption that more interviews would mean more courses being actually manually added in the Notion DB, so I could bootstrap the MVP with a simple 150 courses to begin with.

So with that, I set out with a 1.5 month product journey, starting in February, with a launch date set for 8th March. I lost hope by 15th February itself. Part of the reason is the lack of availability of actual course data that I could work upon. So my journey then on was pretty stagnated with lagging deadlines, failed implementations & just an air of de-motivation around.

Maintaining motivation is 2/3rds of the game
~ Michael Seibel, (YCombinator, Twitch)

I started off with a simple implementation using Notion as my database. The early days actually involved me just calling up my users for interviews, asking for their course lists, noting it down on paper, using the institute website to discover relations & finally storing it all, as structured as I could, by copy-pasting stuff manually in my Notion DB.

This went on for full of February & March. I was getting nowhere.

When I returned to campus, I started to use the opportunity of being in the network to web-scrape the entire website. The only barrier - I don't know how to do web-scraping XD

That's how after a fortnight of being demoralized, web-scraping only the names of 2 entries by mimicking button clicks, was I able to finally arrive at the decision that this could not scale.

I was 22 days away from launch day, at this point, with just barely enough to go on. By now I was even starting to experience loss of faith but the only thing that I strongly agreed was the urgency of the problem statement. It really used to keep me up at night. Then......eureka !

One day, as I was randomly exploring the Chrome Dev Tools while trying a web-scraping script, I happened to observe the Network calls the institute website was making. Luckily, I had been working on Khatabook's website performance optimization & was just learning to observe each network call. Some basic digging actually led me to the call that IIT Madras was using for its own PHP backend.

Then what ? a simple script to mimic the network calls for each course number (there were 2964 !) that would extract the HTML returned, parse it into structured objects & store it in my Mongo. Lucky me, the system did not have CORS or a rate limiter on the API so a simple script was all I needed to feed all the data into my Mongo.

50 consecutive query calls, 5 hours coding, 15 minutes IITM server downtime & 5 database backups later - the database was powerful enough to start powering the app 🚀

This was the day I saw a potential in CourseMap as a product being actually used by the students. Now the hiccup was building an ecosystem of services on top of this data right before the course registration for the next semester happened.

This was the closest this project got to an illegal action :)

The Designing

The UI had to atleast qualify certain base functionalities that I had in mind so that the MVP would atleast have a navigable graph, sorted department wise. It had to support some basic search functionality & it also had to convey the details of each course atleast in a readable manner.
Thus, I used a basic React Ant Design Library to offload some part of the design system for the frontend.

The final layer

I had already passed the the deadline by 6 days

It was 24th April. I was just now 2 more days that would finalize the ultimatum of whether CourseMap was a dream or reality. I felt a kind of final-swan-song like feeling for the entire project & that's when the best work happened

Straight 20 hours coding. Barely any breaks. 1 all nighter & relentless trials later, I had a working MVP that satisfied:

  1. Rendering the graph
  2. Drawing prerequisite<->course relation paths
  3. Search
  4. A lame Firebase authentication
  5. Analytics

Speaking of engineering decisions, I went with

  1. react-graph-vis & ant-design wrapped in NextJS for the frontend - The reason was a great community in the web & predefined UI components, saved me a tonne of time, enabling me to build actual features instead of reinventing the wheel with custom JS & CSS. NextJS will later come in handy as a good strategy for PWA & SEO features.
  2. NestJS for a microservice backend - NestJS feels like Django equivalent for JS. The modular, app-based way of building backend really opened doors for multiple mini-apps actually nested in my main backend. This will also help me for migrations, ORMs & further scaling my application with more & more services
  3. Algolia for search - Industry managed "search as a service" so I wouldn't have to manage "textsearch" myself
  4. Mixpanel for analytics - Since it's seamless to integrate & when coupled with the kind of insights it enables you to derive from your actions with Reports, Funnels, Flows - it's really kind of a one-stop-shop for analytics & studying the behavior of users using the app
  5. Firebase for authentication - Cause I had done it before. No other reason :)

With that, I was ready to go live on 26th April, 7AM. However, I just couldn't seem to figure out the deployment due to increased browser insistence on HTTPS becoming the protocol for communication.

What followed is 1 last dance of trials with Dockerizing backend using networks, using AWS -> ELB=> ECS => Fargate to run my own containers, even trying something with AWS Lambda but then I was just not going anywhere.

One thing I was sure of but, this thing would not fail due to lack of trying from my end

Launch Day

Fun thing is, the closer I felt to completing this, the worse & more frustrating the problems got.

I consulted the engineers at a company (Khatabook) I was interning with, for their opinion on this - I got what was quite simply my last lead - a barebone implementation of TLS certificate using NGINX as a reverse proxy

I just couldn't seem to wrap my head around this concept. I left the lab 2AM, disheartened but still driven enough to regroup on pizza slices & give this 1 last shot. I was out cold on the floor by 2:45am.

The next morning dawned at 7am, with yet again rigorous attempts at the last thing that I had faith in. And then the certificate worked.......and then the TLS succeeded......then the DNS worked, and just like that, thae app was brathed into life. I remember smiling that entire day, just with feelings peace & fulfilment.

But flashback to 10:30am, when the app went live. I used college email & just spam advertising on all college WhatsApp groups I'd been a part of & communities I'd developed, right for this moment. By 11:15am, the app had crashed due to overloaded. I couldn't even SSH into the damn EC2 😂

This was a big problem, but one I liked & dreamt of having. Within the next 15 minutes, I set up 15 running Docker containers of the application, load balanced with sticky sessions, by my NGINX reverse proxy & ready to give 3sec latencies to requests. As I finally pulled back from the continuous typing at 2pm, I'd never felt so content.

The nodes had come together & the graph was truly quite beautiful.

By evening, Mixpanel had blown up with inspiring analytics, people were coming up to me & thanking me for the impact the tool had in course planning - few even reached out with feature requests, suggestions & I got even good response on my engagement channels. If anything, I'm more excited to build the next set of features I've set out for now

As I shut my eyes that night, laying on the cold floor of Alaknanda Hostel (Room 365), looking at the scribblings of CourseMap UI & architecture on the whiteboards (stolen from the trash :) ) of my room, I grinned at what a relentless & rollercoaster journey it had been. And I was so ready for round 2.

The power of cheap creation, mass distribution - that I most like about software

CollegeMap a.k.a CourseMap 2.0

The next goal is to make share the journeys of past students, as seamlessly as possible, with everyone by treating each PoR, Course, Semester as nodes that are part of a bigger graph.

I've started exploring the concept of Ontology & Graph Databases now. It is exciting, it is scary & the possibilities are once again, keeping me awake at night.

Whether or not CollegeMap will succeed or not, is uncertain, but it's going to be different building this time. In fact, every time building will feel different now. The thing is, once you create something from ground up, more than the final product, you resonate with the process of always getting back up after each fall & after a point, the blows too start to feel like checkpoints to the next level. I smile a lot more now, having realized that I have found what I love doing in my life. And if that doesn't fill the soul with a fire, I don't really understand what can....

Hell, I can't already wait for the next chapter 🚀