Proposal
Overview and The Need
Waartaa is an open source communication tool for teams and groups. It is built on top of IRC. Currently, Waartaa is an IRC client as a service and it supports centralized logging, 24x7 idling, notifications and unique identity to a user on IRC across multiple clients/devices, and also a rich UI for awesome user experience. Waartaa is built on top of web technologies like Meteor (a Nodejs framework), MongoDB for data storage and twitter bootstrap for UI design and responsiveness. It runs seamlessly across various platforms with the same look and feel.
Any relevant experience you have
I have got decent experience working with Javascript technologies both on client and server side. During my internship, I worked on this project called CodePlayer which required good understanding of Javascript. I was able to complete the project in a short span.
Deliverables and Implementaion
Sub-tasks that I will be completing during GSoC period are:-
- Build a central hub for searching/reading channel logs for Open Source communities and projects.
- Use the log search framework to empower nifty addon features like: permalinks in mail notifications, app notifications, bookmarks, etc.
- Video/audio conferencing facility on top of HTML5 and JS technologies.
- Admin console panel
Build a central hub for searching/reading channel logs for Open Source communities and projects
Centralized logging is one of the important components of Waartaa. It removes the pain to sync logs across multiple IRC clients and devices. Logs gathered can be used to build the following features:-
- A separate standalone log browsing and search page accessible to the public. This will serve as a global directory of chat logs. There'll also be an option to join a server channel directly from the search page. Basic search interface will provide filtering by nick, full text and datetime ranges.
- Create permalinks for each chat log in the search interface. The permalink will show a page of chat logs containing the particular chat log, which will be highlighted. The permalinks need to be generated during handling incoming chat message so that mail notifications can contain the permalink.
- Notifications will also appear inside the chat interface (similar to Facebook notifications). The notifications will point to the permalink for the chat logs.
- A bookmark could be implemented using the same permalink concept. We just need to map user to permalinks in some collection, lets say:
Bookmarks {
server_name: String,
channel_name: String,
message: String,
type: String,
permalink: String,
user: String (username),
user_id: String (user._id)
}
where in the user can search/filter logs in page. These bookmarks saved will be shown on user's dashboard page.
Implementation details
After talking to the mentors, Ratnadeep and Sayan told me they have already explored different technologies and ElasticSearch seems best fit for the purpose. Currently, the logs are stored in MongoDB and the plan is to move them to ElasticSearch.
Waartaa-ElasticSearch integration
I plan to push new logs from Waartaa to ElasticSearch in an async way so that the app does not incur any slow down. If some issues come up with this approach during load testing this Waartaa-ElasticSearch integration, which I presume to be because of network IO setup latency, I will implement a task queue to throttle network connections to ES at a given point of time and send log data in chunks. This will allow us to send more data in a single network transaction. That said, my focus is to use as less dependencies as possible and I will add a dependency only if there's no alternate way.
Video/audio conferencing facility on top of HTML5 and JS technologies
To not over burden the servers, its best to setup a peer-to-peer connection for audio/video conferencing.
In this regard, WebRTC offers Browser-Agnostic solution. Anyone with a web browser and a microphone can make calls to anyone else with a web browser and a microphone. If one or both parties has some sort of video camera, the call can also involve video. Furthermore, Javascript APIs involved in enabling peer-to-peer communication are simple enough to use and with only 5-6 lines of Javascript code a WebRTC client can be created. However, only limitation it has is that it requires up-to date graphic cards.
It will be kind of an experimental feature considering the limitations of WebRTC. We’ll keep improving along with WebRTC.
Goal of audio/video communication integration with Waartaa
The mentors want basic audio/video communication feature integrated with Waartaa. This will serve as a proof of concept that audio/video chat can work well alongside IRC under the same hood to give better communication and collaboration experience. I and the mentors believe that it’s quite possible to get a very basic audio/video communication powered by webRTC integrated with Waartaa in the GSoC tenure for a start. However, this feature will need a lot of additional research and effort to make it production ready. I along with the mentors will continue to work on improving this feature to make it work in production and handle descent scale as soon as possible.
WebRTC Challenges and possible solutions
1. Setting up the Signaling process
WebRTC involves peer-to-peer connection but it still needs a way, a medium (e.g., a server) for clients to exchange metadata to initiate and coordinate communication.
The term I am here putting emphasis on is called Signalling. Signalling refers to the process of coordinating communication. In order for a WebRTC application to set up a 'call', its clients need to exchange information: Session control messages used to open or close communication.
- Error messages.
- Media metadata such as codecs and codec settings, bandwidth and media types.
- Key data, used to establish secure connections.
- Network data, such as a host's IP address and port as seen by the outside world.
There are three possible ways to set-up the Signalling process:
- Building your own Signalling server: After exploring, I found out the main challenge with this method is realtime back and forth communication between server and client. But this problem is very common and real world solution already exists. In our case, we could use socket.io and node. Socket.io is best suited for this job because of it’s in built concept of rooms and also, waartaa is fully implemented in node(javascript). Check out this demo code: [1]
- Using a third party Signaling server: This is similar to above method except we don’t have to deal with writing signaling server code, handle maintenance and scaling problems involved with servers. Some third party services that uses Socket.io and integrates webRTC with them are webRTC.io, easyRTC, Signalmaster etc.
- Serverless Signaling: This is the simplest method one could use to build this service. It’s only requirement is that the clients should be in touch with each other via some instant messaging service(which in our case will be itself) to send meta data(SDP). SDP will be auto generated and then client A just have to send these details to client B. See this demo: [2]
Each of the above solution has its pros and cons. The choice of method will depend on how much the method can scale with less issues. Also, each of the above methods can be implemented in GSoC timeframe because for each method open source code already exists and I saw the code size is not too big.
One more thing, I have some experience working with realtime technologies. I even implemented a javascript library which does kind of a same work as socket.io does but it was not as robust as socket.io. Here is what I built: [3]
2. Dealing with NATs and Firewalls
Even after the metadata signaling is done with or without intermediate server, still WebRTC requires peer-to-peer endpoints. In real world, most of the devices live behind firewall, NATs and proxies.
This problem can be overcomed by using an ICE framework. To enable this ICE server URLs must be passed to RTCPeerConnection, as described below:
var configuration = { 'iceServers': [{ 'url': 'stun:stun.example.org' }] };
pc = new RTCPeerConnection(configuration);
This will obviously require using third party servers called STUN and TURN. A STUN server is used to get an external network address and TURN servers are used to relay traffic if direct (peer to peer) connection fails.
I haven’t done much research on whether the use of external servers(STUN and TURN) can be totally avoided but my initial research about the topic says no.
Admin console panel
Admin console panel will give useful insights into the data to admin(s) like:
- Information on users online
- Information on active users (weekly, monthly etc.)
- Show node-irc client status: servers/channels listening to for a user, and allow admins to control the node-irc client (connect/disconnect/disable/enable) if needed.
- A django-admin like panel to perform CRUD operations on data collections.
A rough timeline for your progress
Average time per week = 40-50 hours
Before 30 April:
- Exam period.
1 May – 18 May:
- Learning and researching more about elastic search.
- Learn meteor and study project code.
- Discuss with mentors about elastic search project implementation details.
- Setup development environment and get a head start
19 May – 3 Jun:
- Implement storing logs for search backend in ElasticSearch instead of MongoDB.
- Storing permalinks of each chat logs.
- Write a script to transfer any previous logs from MongoDB to ElasticSearch, if required.
4 Jun – 11 Jun:
- Create a search interface for channel chat logs.
12 Jun -30 Jun:
- Implement basic search/filter.
- Implement bookmarking feature, permalinks in mail notifications.
- Search bar inside the chat app for searching through logs in page.
23 June(Mid Term deliverable):
- Fully functional global search
1 July – 25 July:
- Discuss and finalize with mentors the final Signaling process for webRTC to implement.
- Implement a working architecture of video/audio conferencing using WebRTC.
25 July – 10 Aug:
- Work on admin console
- Implement a basic interface to perform CRUD operations on collections by admins
- Stats dashboard to show data like active clients count, active channel/server listeners, etc.
- Panel to connect, disconnect, disable and enable user IRC clients.
11 Aug - 18 Aug:
- Bug fixes and polishing developed features.
- Finalizing code and feature documentation (done simultaneously as part of the development process).
Any other details you feel we should consider
I have been in constant touch with mentors, Ratnadeep and Sayan for past few weeks. I have got good idea about project goals and it's implementation steps. Also, I don't have any other engagements during GSoC period.
Have you communicated with a potential mentor? If so, who?
Yes, Ratnadeep Debnath and Sayan Chowdhury