Mumbai Monsoon – Calling All Developers Now!

The rains in Mumbai last week have once again showed why BMC and all its citizens have miles to go before we can truly call this a well managed city. This blog post is not an attempt to finger point but a sincere appeal targeting all like-minded people to join together in creating a series of applications that truly utilize digital services in delivering contextual and timely information about our Monsoon.

mumbai-rains1

Source : Getty Images

This is a long blog post, that goes into various areas. At the end of it all, I hope we developers all come together to do something about it. Let the story begin …

Monsoon App Downpour – Challenge Post

First a bit of history about why I am so interested in this. Last year, the online challenge site, ChallengePost organized a series of contests that invited developers to build out solutions to address key urban problems. This challenge was in collaboration with Ford Motors, where you were encouraged to integrate OpenXC standards into your applications.

Let us keep the technicalities aside for a moment. The key is to take a look at the clear requirement that was put in the challenge. I encourage you to read the first 4-5 paragraphs of this page. Sounds familiar to us Mumbaikars, doesn’t it ? The contest was on from mid-July to mid-October.

There were several submissions and a few winners. There were notable judges too as the home page indicates. I was one of the winners of the contest and did go to some extent to provide high tide, traffic, heavy rainfall information to users of my Web and Android application that I submitted in that contest. I am pretty certain not many folks of heard of my application and the other applications that were placed higher than me in the winners category and which I do believe were better and address key information that Mumbaikars would need.

Note : If you hit my application url, you will get a 404 and this is because the Web Services were brought down and I did not have a choice but to disable the apps. 

The key thing that I want to point out is that such contests / hackathons should clearly have been organized with the blessings of our local government bodies and that organizations/individuals should come and participate more in such contests. You may not be able to code but you do have great ideas that could help us. More on that later.

What made my application possible? The MCGM Monsoon Web site and its Web Services. One may complain about them but to be fair to them they had RESTful Web Services that I was able to poke into and integrate into my application. They gave me information on Tides, Rainfall across 30-40 weather stations that they have setup across the city, traffic alerts and more.

What is coming next is a clear reminder why my hopes were crushed via unwittingly by the development team that builds out the MCGM Web Services.

Breakdown of MCGM Monsoon Web Services page

I submitted my application in October 2014 and the results gave me the confidence to build out more features and publicize the application to a wide variety of users.

And guess what happens next? The Web Services are no longer available. Game over ! In one sweeping shot, my applications and intent to take this application to the next level were gone! It is easy to put the blame on BMC but years of working with APIs have given me the common sense that there were no agreements in place. BMC did not make any promise to continue hosting this service.

I thought this was a temporary thing and that maybe they would bring up the Web Services in and around monsoon time. I was even harboring thoughts that there would be richer information in the APIs that I could make use of.

But that did not happen and I took down my website to avoid my own embarrassment. June 19th brought all these memories back again and made me start digging into what is going on the MCGM Monsoon Website. Are they not interested in providing information, data, services that developers like us could use to bring our applications that could help all of us.

It led me down an interesting path, each of which I will share. Let’s move ahead.

Current BMC Disaster Manager Mobile Apps + Websites + Social Media Accounts

A few years back with much fanfare, BMC released a site called www.mumbaimonsoon.in. This site promised to do the following:

“Provide information on rainfall that collected information from 30-40 weather stations that were set across the city. This information was supposed to be updated every 15 minutes. In addition to that, BMC in coordination with traffic police, railways and airport authorities was to provide information about traffic diversions, train and flight disruptions”.

And if you have already clicked the www.mumbaimonsoon.in in a hurry (I don’t blame you!) — you were presented with a domain page that shows that the site is down. And what is shocking in this whole process:

  • BMC did not even bother to renew the domain. They just let it expire.
  • A Whois search on the domain shows the owner information as given below:mumbaimonsoonin-whois
  • The owner has put up the domain name for sale at $2000.

There cannot be an excuse for this. The site was announced in 2012 and in 2014-2015 – it was gone with the domain not being renewed too.

Let’s move on to the next level of apathy that is visible in its digital services team. Twitter was by far, a very useful service during these days to get information on what is going on. You would expect that they would be active on that and I reproduce below the following. Be your own judge!

disastermanagement

How many of you know about the Disaster Management Twitter account. Look at the following:

1) The profile is not updated and it still references the now unavailable mumbaimonsoon.in

2) The last Tweet was in October 2012.

3) I am not going to even debate why High Tide alert information is essential in the month of October, but that is another discussion.

I found the following Twitter account too, but I really want to believe that this is not the official account. Please tell me that.

bmccorp

So is everything lost in that case? Shall we pack up and go home? The answer is a clear No !

There is an updated website (more on this crazy renaming later and how it broke multiple applications including mine in the next section). But for now, there is a shiny website that provides this information. How long this lasts is anyone’s guess but let’s be confident that it does.

GDMA (Greater Mumbai Disaster Management Authority) Web site

This site was originally available at monsoon.mcgm.gov.in but that site is no longer available.

http://dm.mcgm.gov.in:9080/gmdma/

A screenshot is shown below:

gmdma1

Don’t even ask me about port 9080 and all that. Let’s leave that aside for now.

Key things to note:

  • On the right navigation menu, you will find 2 important links: Weather and Tide Info and Traffic Diversions.
  • Click on Traffic Diversions to get notified about any Rail / Road / Air issues. At the time of writing, I was greeted with this page:gmdma2
    There was some diversions for Road and the details are given above. Some progress definitely.
  • The Weather and Tide information is more exciting and which was there at the older website that they simply changed the name and broke us all. It gives the following page (http://dm.mcgm.gov.in:8080/RESTFulWS/newestindex.html)

gmdma3

The website is quite useful and you can see the rainfall across various time intervals in the area of your choice. This is the result I believe of various weather stations that collect rainfall information. You can see the high tide information too.

A few questions/points (that developers can relate to!):

  • Check the page name : newestindex.html — what the hell is that?
  • By the way, index.html page is also available. Try it, it does not work!
  • Is there really a team developing this ?
  • Any QA check ?
  • Anyone who approves sane development standards/processes?
  • Who developed this website ….anyone knows?
  • Is the site responsive aka works on various form factors (desktop , mobile browser)? You tell me!

Anyways, answers to these questions do not matter anymore.

What about mobile apps? Yes – there are mobile apps too!

Android App

Here is the link to the Play Store.

gmdma4

Interesting to see the App Id as : com.abel.mmv2 …hmmm? A news report here shows that the site is being developed by Abel Software and Consultants Pvt. Ltd, based out of Mulund, Mumbai. It is the same company that it says developed mumbaimonsoon.in. Sure – that makes complete sense now.

I have only one suggestion to them. At least come up with a proper application id for your Android application.

I have many other questions vis-a-vis who owns the code for this application? Where is it stored? Under whose account? Backups? …. never mind. Let’s move on. Well I am not done yet!

Take a look at the website again :

At the bottom you will find the following (I have put the image here):

gmdma5

A small but important thing. Take a look at the statement “Theme by Abelsoftcons”. This clearly shows or claims that the theme was developed by Abelsoftcons. Click and see where the link goes. I do not have anything else to say here. Atleast attribute it to the right owner guys! That is the least you can do!

iOS App

The link on the main web site : https://itunes.apple.com/us/app/mumbai-monsoon/id885404768?ls=1&mt=8 leads to nowhere. The App is not available. I tried searching it in both Indian and the US App Store and it came up short. More informed users could tell me if they are able to find the application. If not, this is a serious breach which both BMC and the Application Development Company responsible for this app needs to answer. 

Other platforms

Tough luck. Use the website ! And good luck using the website in your mobile browser.

MCGM Monsoon Web Services – Updated Documentation

This is a section for developers – so please bear with me.

The first thing that I did on discovering the new site was to check if the old web services that provided information on traffic diversions, rainfall data and high tide information existed in some form or the other.

This is how I went about figuring this one out (It does not take an expert to get to this). The steps were simple:

  1. Launch Chrome browser (other browsers have enough ammo too but I love Chrome Dev Tools!)
  2. Go to http://dm.mcgm.gov.in:8080/RESTFulWS/newestindex.html
  3. Go to Chrome Dev Tools (F12 … all hail the keyboard shortcuts!)
  4. A few JavaScript errors will greet you …. let’s pretend we are blind to those for our sanity!
  5. Click “Network” tab in Dev Tools and refresh the page again.
  6. See the Web Services calls that are getting made as given below:gmdma6
  7. We can debate forever the shocking number of calls needed. The developers need to consider aggregated calls, cache local data and more. But that is a discussion for another day. We need to have a starting point somewhere right …?

I had my previous private documentation on how these worked and I am reproducing some of that for you today, so that you can get started with the REST APIs that they have made available:

Get Tide Information

On any particular day (date) – you can get the high tide information via the following:

HTTP Method : GET

HTTP Endpoint:

http://dm.mcgm.gov.in:8080/RESTFulWS/rest/MumbaiMonsoonService/getsptidesdata

Go ahead and click it. It gives a nice JSON response as given below:

gmdma7

Few things to note here:

  • The Unit of Measure is metres (m).
  • hth stands for high tide height
  • lth stands for low tide height
  • htt stands for high tide time
  • ltt stands for low tide time

At what point this information is updated, I have no clue. I suggest you run a regular CRON Job and then update your application data accordingly.

Get Diversion Information

You can get the regular traffic diversions (Rail, Road and Air) via the following:

HTTP Method : GET

HTTP Endpoint: http://dm.mcgm.gov.in:8080/RESTFulWS/rest/MumbaiMonsoonService/getsptrafficdiversiondata

Go ahead and click it. It gives a nice JSON response as given below:

gmdma8

The response JSON is intuitive enough. I guess having webhooks that can update you in case of changes is a long shot for the development team here. You will need to poll this regularly for any changes. I suggest you providing this service to other apps – should be very useful.

Some other data points missing here is a time stamp for this or indicating the date/time when the problem has started.

Get Rainfall information

This one is a bit tricky but let me explain this.

First you need to know about various locations for which the data is available (there are several across the city for which the rainfall data/weather data is collected). These are locations and then there are sub-locations within that.

To get locations, you can execute the following call:

HTTP Method : GET

HTTP URL : http://dm.mcgm.gov.in:8080/RESTFulWS/rest/MumbaiMonsoonService/getsplocationsdata

You will get the following information (abbreviated screenshot is below):

gmdma11

The important thing to note is the locationidData field. This id field will be useful to retrieve the Weather information for that location id and also the Rainfall information for that location id.

You can also get an information on all sub-locations as given below:

HTTP Method : GET

HTTP URL: http://dm.mcgm.gov.in:8080/RESTFulWS/rest/MumbaiMonsoonService/getspsearchlocationdata

You will get the following information (abbreviated screenshot is below):

gmdma12

The sub-locations are definitely a lot more in number and seems incomplete in terms of the attribute values. But this is what you need to do, if you want to work with sub-locations and not the main locations.

1) Given a sub-location id i.e. idData field, you need to call the following method

HTTP Method : POST

HTTP URL : http://dm.mcgm.gov.in:8080/RESTFulWS/rest/MumbaiMonsoonService/getspsublocationlocationiddata

The POST FORM data is shown below:

{“locationidsubselected”:”67″}

where 67 is the sub location id (idData attribute value).

This will give you a response as follows:

gmdma13

There you go. You have the locationidData value i.e. 33. You can now call the next 2 methods i.e. Get Location Rainfall and Get Weather Data using the location id value that you now have.

Get Location Rainfall

You need the location id for this. Once you have that, execute the following call:

HTTP Method : POST

HTTP URL:

The POST FORM data is shown below:

{“locationidselected”:”25″}

For e.g. 25 is the locationid for the location that you want information on.

The data returned is as follows:

gmdma10

The data returned is in mm. And there are 6 time intervals in which it is returned. For e.g. 15min means rainfall in the last 15 minutes, 1 hour means rainfall in the last 1 hour. Neat !

This data itself could be used for interesting alerts where you could warn users that a particular location is witnessing very heavy rainfall.

Get Weather Data

You need the location id for this. Once you have that, execute the following call:

HTTP Method : POST

HTTP URL : http://dm.mcgm.gov.in:8080/RESTFulWS/rest/MumbaiMonsoonService/getspweatherdata

The POST FORM data is shown below:

{“locationidselected”:”25″}

For e.g. 25 is the locationid for the location that you want information on.

The data returned is as follows:

gmdma9

The temperature is in Celcius. The humidityData is in %. The wind Data is in kmph. The barData stands for pressure and is in Hpa.

There seem to be a lot more web service calls get through that could be interesting to your applications but I will leave that for another day. These basic calls should suffice here for you to make a lot of interesting applications.

MCGM Monsoon Web Services – Road Ahead and some recommendations

The current state of MCGM RESTful services that provide information is something that is far from optimal. In fact, I am bold enough to say that it is abysmal in its design/documentation and road ahead.

Here are my suggestions for whatever it is worth:

Base URL, standard Port Numbers and more

What sort of base URL is this? And port 9080. The API needs to standardize on that and not use these obscure naming conventions.

Introduce Versioning

Granted that your API will change as time passed on. New requirements/changes will come in which will requires that new data formats/capabilities be supported. What is likely to happen and which I fear is that conveniently the API team will change the API (Read that as break the API) and change their website to use the new API. The result is that all applications that are integrated with the API will break instantly, resulting in horrible user experiences and an immediate appearing of fly-by-night experts that will question what went wrong!

We must introduce versioning immediately in the API. And you cannot stop a version without adequate notice/information to its users to upgrade to a newer version.

Client Libraries

If you want to unleash the API on developers and allow them to iterate rapidly through their ideas for Monsoon Apps, you need to provide them client libraries in a variety of programming languages. Not REST APIs in their raw form. That is too much for most developers to even think about spending their time on.

Write client libraries that provide wrappers in popular programming languages like Java, Python, C# and more.

Documentation

Provide API documentation, sample code and a quick “Hello World” that can help demonstrate the information that the API provides in minutes. Any more than 10-15 minutes of the developer time in evaluating your API is as good as a dead API strategy.

API Quotas

You are providing a public API and there seems to be no information or protection vis-à-vis quotas. Perfect recipe for a disaster. Introduce Quotas that limit developers from playing around with your API. Document your API fair usage policy. Think about different API plans. You have to function and execute like an API provider if you want to be successful. There are companies providing API management and gateway solutions. Investigate into those to get a jumpstart.

There are a lot more recommendations for API Best Practices – but you get the gist of what I am trying to say here.

Call for Action

The above information should have been planned from day 1 in case the teams working on this were seriously interested. The technical project execution team should have the above deliverables ideally and a concerted effort should have been in place to make this documentation public and available to all.

Having said that, our life is like a train. Just forget the stations that have passed by and let’s look forward. You may have all the reasons to be sore with BMC, in fact I do. But screw all that and do your own bit. I have given you enough information now, use your offices and individual skills to take this to the next level.

Use the API information. Build out applications. Share it with everyone. Talk to me. Maybe we can do something together. Meetup groups could organize a hackathon. Maybe you could hold a hackathon in your own organization. The possibilities are endless and you are an equal partner.

I would like to hear more from you. Write to me at romin.irani@mindstormsoftware.com

IoT Projects Showcase

We had an event in our office (Xoriant) last week where any group of employees could come together and put up a stall. The stall could be a game, some food stuff or any other skill you want to show case. In a technical company, these events are usually a channel to utilize some of your other ideas, interests and talents …. and in most of the cases, people usually do that in non-technical areas.

IMG_20150529_122742

 

One team stood out in my opinion (of course I am biased towards technology… did you have any doubt?) with a stall that showcased multiple gadgets and technologies like Microcontrollers, Microcomputers, various Sensors (via the IoT Route) and other personal devices that are likely to lit up our homes (or are already doing so). The result was a smorgasbord of all of the above that received positive reviews.

Specifically when it came to Internet of Things, there were 2 mini projects that demonstrated the concept quite well in my opinion. This may look trivial to some of you but using multiple boards from different vendors, integrating with the cloud, setting it up while having a normal working job and doing this in under a day speaks volumes of the persistence but also how easy it is becoming to connect devices to the cloud and thereby produce actionable results.

In summary, when it came to IoT, the following 2 kits were used:

Now, lets get to the projects that were demonstrated:

Home Automation via littleBits

littleBits is a great kit that helps you put together electronic circuits in a lego-like fashion in no time. In my opinion, it is a great vehicle for anyone new to electronics to understand how the basics work and then combine them in an innovative way via a bit of your everyday craft material knowledge.

The project demonstrated the cloudBit module, which is able to receive and send commands to the littleBits Cloud and then in return drive the rest of the bits connected to it. It also provides neat integration with IFTTT.com, that can be used to trigger actions via a variety of other common internet tasks.

The project demonstrated how you could control the following things in your home via the littleBits Cloud module:

  • Fan
  • Light
  • Bell

To simulate the above stuff, littleBits modules for Fan, LED and Vibration were used. In my opinion that is good enough and in real life, you would use a relay to drive these things. The ON/OFF was done via the cloudBits Control Panel that littleBits provides. The API could have been used to create an app to do that but that is no big deal.

Here is the project in action. Once the cloudBit receives a “1”, it will drive 3 parallel circuits i.e. the Fan, the lights and the vibration motor that will ring a bell.

IMG_20150529_123035

Environment Monitoring with Tessel

Tessel.io is a more sophisticated micro-controller that allows one to program using JavaScript (yes, you read that correct in case you have never heard of it before!).

In this project, 2 Tessel sensor modules were used:

  • Ambient module : This provided the Light and Sound data values.
  • Climate module : This provided the Temperature and Humidity data values.

IMG_20150529_123217

The data was then transmitted to an excellent cloud service called ubidots. It provides you with an ability to define your variables (in this case 4 of them i.e. temperature, humidity, sound and light) and gives you an API to populate them from your client application. Tessel runs Javascript and all that was needed was a simple HTTP POST to put the values into ubidots.

ubidots is then able to visualize the variables via a variety of ready-made widgets like a time-series line chart, gauge and more. The User Interface that you see above is the Dashboard exposed by ubidots.

In Summary

There were several other gadgets/devices on display. They included Raspberry Pi, Android TV, Chromecast, NFC stuff and more. But the above were specific to IoT where the devices were in the connected cloud.

If you have any questions on the above, let me know. In case you would like to setup similar stuff and discuss ideas around this, drop a line. The possibilities are endless and the fun has just started.

Stay tuned for more on IoT.