What is whereis?


Earlier this year I created the whereis Spark Bot (whereis@sparkbot.io) to address a challenge I faced in the new Cisco North Sydney office (see - The Australian). While the space gives maximum flexibility in work style, and the technology allows for anywhere, anytime communication, sometimes you still need to find someone in the office. This could be for an ad-hoc discussion that would benefit from being face-to-face, meeting up before heading out to a customer meeting, or simply to have a coffee with a colleague. In an Activity Based Workspace (or a Cisco Connected Workspace), finding a colleague can be a challenge. whereis solves this by providing on-demand colleague location in Spark, leveraging Spark Webhooks and the Cisco Connected Mobile Experience (CMX) API.



In this blog, I'll explain how the bot works.


Architectural Overview


whereis consists of 3 python scripts:

- whereis.py which is the webhook receiver. It receives the POST from Spark, GETs the message, and then POSTs the queried username to the other scripts.

- see-em-x.py which takes the POSTed username, queries a list of CMX servers to find the user (based on 802.1X username), and then renders the location on the floor image.

- see-em-x-mse.py which has a similar function to see-em-x.py but queries the legacy MSE API.


The CMX API is the preferred method as it provides a list of all devices with the queried username e.g. for users with more than one device (isn't that all of us? ). The MSE API is used for buildings that aren't synced with CMX, and as a backup for CMX. It is limited in that it only returns the first device with that username.


The modular nature is mainly due to historical reasons which you will see in the code - it was initially designed for a web front-end so see-em-x.py also accepts input from an HTML form and responds with the data in HTML format.


The Initial Request


When a user enters a username (either just the username in a 1:1 "People" Spark conversation, or after mentioning the bot in a "Space"), the following happens:



When whereis.py receives the POST from the webhook, it requests the content of the message with the sendSparkGET function.


def sendSparkGET(url):
    request = urllib2.Request(url,
                            headers={"Accept" : "application/json",
    request.add_header("Authorization", "Bearer "+bearer)
    contents = urllib2.urlopen(request).read()
    return contents


def index(request):
    webhook = json.loads(request.body)
    print webhook['data']['id']
    result = sendSparkGET('https://api.ciscospark.com/v1/messages/{0}'.format(webhook['data']['id']))
    result = json.loads(result)


It then does some formatting (e.g. removing the "whereis" from the mention and changing everything to lowercase) as well as handling "help" input before sending the username to the other scripts.


The grequests module is used to send a POST with the username to both scripts.


import grequests as async


        payload = {'person':person, 'clientCurrent':'0', 'source':'spark', 'clientList':'[]'}
        headers = {'User-Agent': 'Mozilla/5.0'}

        async_dict = {}

        request = (async.post(url = u, headers=headers, data=payload) for u in cmx_script_urls)
        responses = async.map(request)


The see-em-x.py script stores the username as 'person' and sends a GET for '/api/location/v2/clients?username=' to every CMX server with all domain-username combinations e.g. 'matfowle', 'CISCO\matfowle', 'cisco\matfowle'.


        # Add each CMX URL with each username, domain prefix combination to the list.
        for u in urls:
          urlClientByUsernameWithDomain.extend((u+urlClientByUsername+p) for p in prefixes)

        # Dictionary where we will store the responses from CMX in a format of URL:JSON.
        async_dict = {}

        # Use the grequests module to send requests to all the CMX servers at the same time with all domain prefix and username combinations.
        request = (async.get(url = u+person, auth = HTTPBasicAuth(cmxUser,cmxPass),verify=False, timeout=2) for u in urlClientByUsernameWithDomain)
        responses = async.map(request, exception_handler=exception_handler)


A similar process is done in the see-em-x-mse.py script with some changes in the API URL.


The Initial Response

At the same time that the above was happening, whereis.py has communicated back to the room that the bot is looking for the user. This is to give the user some quick feedback that the request has been received.


If the use is found, the response from the CMX server is a list of devices found with the requested username. If the user is not found, an empty JSON response is returned by CMX.


The responses from every CMX server is stored in a dictionary in a 'url : json' format which is then looped through to see which CMX server has found the user.


        # Check that the returned responses are all valid and if so, put them in the list in the format of URL:JSON.
        for result in responses:
          if result:
            async_dict[result.url] = json.loads(result.content)

        # From the dictionary, find which server responded and set that as the CMX server so that we can do the image call.
        for k in async_dict:
          if async_dict[k]:
            for u in urls:
              if u in k:
                cmxAddr = u
            clientList = async_dict[k]
            clientList = async_dict[k]


From this response, the username, map hierarchy (Campus>Building>Floor), floor image name, floor dimensions, the x,y coordinates for the device, the SSID name and the status (associated/probing) will be used.

Get the Floor Image

Now that the floor image name is known, the see-em-x.py script will go and get the actual image file from the CMX server using '/api/config/v1/maps/imagesource/' and then cache it on the local disk. This means if we see this floor again in a request we don't have to waste bandwidth getting it every time from the CMX server.


    # Check if we have already got the floor image file stored locally.
    if os.path.isfile(image_path + client["mapInfo"]["image"]["imageName"]) == True:
      file = image_path + client["mapInfo"]["image"]["imageName"]
      fh = open(file, "rb")
      image = storeMemory(fh.read()).encode("base64").strip()

    # Get the image file of the floor the client is on from CMX, save it to a file, then load it in StingIO so we can work with it in memory.
    # We save it to a file first so that next time we don't have to pull an image from CMX. Speed things up and reduces load on CMX.
      image = cmxContent(cmxAddr+urlFloorImage + client["mapInfo"]["image"]["imageName"])
      file = image_path + client["mapInfo"]["image"]["imageName"]
      fh = open(file, "w+")
      fh = open(file, "rb")
      image = storeMemory(fh.read()).encode("base64").strip()



Return the Location


At this stage, see-em-x.py has all the information it needs to return the location, but it needs to render the location on the floor image. This is done using pyplot by creating a plot on top of the image and marking the x,y of the client with the scatter function. The first scatter line plots the x,y with a dot and the next 3 draw concentric circles around the point.


import matplotlib.pyplot as plt


implot = plt.imshow(convertedim, extent=[0, client["mapInfo"]["floorDimension"]["width"], 0, client["mapInfo"]["floorDimension"]["length"]], origin='lower', aspect=1)

    # Mark the client's coordinates that we received from CMX.
    # The first line will draw a dot at the x,y location and the second and third lines will draw circles around it.
    plt.scatter([str(client["mapCoordinate"]["x"])], [str(client["mapCoordinate"]["y"])], facecolor='r', edgecolor='r')
    plt.scatter([str(client["mapCoordinate"]["x"])], [str(client["mapCoordinate"]["y"])], s=1000, facecolors='none', edgecolor='r')
    plt.scatter([str(client["mapCoordinate"]["x"])], [str(client["mapCoordinate"]["y"])], s=2000, facecolors='none', edgecolor='r')
    plt.scatter([str(client["mapCoordinate"]["x"])], [str(client["mapCoordinate"]["y"])], s=3500, facecolors='none', edgecolor='r')


The plot is then formatted correctly using the gca function.


    # Currently the plot is the same size as the image, but the scale is off so we need to correct that.
    ax = plt.gca()

    # The plot starts 0,0 from the bottom left corner but CMX uses the top left.
    # So, we need to invert the y-axis and, to make it easier to read, move the x axis markings to the top (if you choose to show them).

    # Use this to decide whether you want to show or hide the axis markings.

    # Save our new image with the plot overlayed to memory. The dpi option here makes the image larger.
    plt.savefig(buff, format='png', dpi=500)


see-em-x.py now has everything it needs to send back to whereis.py.


The username, status, SSID, building name, floor name, final image location (which is written to disk with a UUID filename), number of devices for this user, and the JSON response from CMX are returned to whereis.py.


    # If the request came from Spark, save the image to a file so that the Spark Bot can send it as a file. It also sends some text about the user and their location.
    if source == "spark":
      file = image_path + str(uuid.uuid4()) +".png"
      fh = open(file, "w+")
      print "Content-type: application/json"
      response = {'text': client["userName"] + ' is '+ client["dot11Status"] +' to '+ client["ssId"] +' in '+ hierarchy[1] +' on level '+ hierarchy[2], 'image': file, 'clientCount': clientCount, 'clientList': clientList}
      print (json.JSONEncoder().encode(response))


Similar is done in see-em-x-mse.py but as mentioned, only one device is returned for the specified username.


whereis.py will receive responses from both scripts and put them in a dictionary. It will first check if see-em-x.py found the user (by checking if an image was sent), if not it will check if see-em-x-mse.py found the user. If it also doesn't find the user, the not found message is sent using sendSparkPOST(), otherwise it will send the found message and the image. It will also clean up the plotted image from the disk.


        for result in responses:
            if result:
                async_dict[result.url] = json.loads(result.content)

        if async_dict[cmx_script_url]['image'] == False:
            data = async_dict[cmx_script_url2]
            text = data['text']
            filepath = data['image']
            clientCount = int(data['clientCount'])
            clientCurrent = 0
            clientNext = clientCurrent
            filetype = 'image/png'
            print text
            if filepath == False:
                sendSparkPOST("https://api.ciscospark.com/v1/messages", {"roomId": webhook['data']['roomId'], "text": text})
                return "true"
            sendSparkPOST("https://api.ciscospark.com/v1/messages", {"roomId": webhook['data']['roomId'], "text": text, "files": ('location', open(filepath, 'rb'), filetype)})
            return "true"
        if async_dict[cmx_script_url2]['image'] is not False:


If see-em-x.py does find the user then it will use this data instead as it has support for multiple devices per user. However, this means that whereis.py needs to check the number of devices (clientCount). If it is greater then one, first, send the first device location to the Spark room.


        data = async_dict[cmx_script_url]
        text = data['text']
        filepath = data['image']
        clientCount = int(data['clientCount'])
        clientList = data['clientList']
        clientCurrent = 0
        clientNext = clientCurrent + 1
        filetype = 'image/png'
        print text

        if clientCount > 1:
            sendSparkPOST("https://api.ciscospark.com/v1/messages", {"roomId": webhook['data']['roomId'], "text": person +" has "+ str(clientCount) +" devices. I'll get their locations for you."})
            sendSparkPOST("https://api.ciscospark.com/v1/messages", {"roomId": webhook['data']['roomId'], "text": text, "files": ('location', open(filepath, 'rb'), filetype)})


Then, for every extra device with this username, POST the JSON received from CMX to see-em-x.py but set the currentClient to the next device in the list. By passing the CMX JSON between see-em-x.py and whereis.py and back again, we can skip see-em-x.py having to query CMX again.


            while clientNext < clientCount:
                payload = {'person':person, 'clientCurrent':str(clientNext), 'source':'spark', 'clientList':clientList}
                headers = {'User-Agent': 'Mozilla/5.0'}
                msg = requests.post(cmx_script_url, headers=headers, data=payload)
                data = json.loads(msg.content)
                text = data['text']
                filepath = data['image']
                filetype = 'image/png'
                sendSparkPOST("https://api.ciscospark.com/v1/messages", {"roomId": webhook['data']['roomId'], "text": text, "files": ('location', open(filepath, 'rb'), filetype)})
                clientNext += 1


And that's it!




What Next?

First, whereis@sparkbot.io will only work for Cisco employees but you can grab the code for whereis and adapt it for your environment here - GitHub - matfowle/whereis. Note, I am no developer so my code can probably use some work!

In the future I would like to add corporate directory support so that users don't have to remember usernames and can instead enter "Firstname Lastname". Until then, let me know what you think below in the comments. Also, be sure to check out the CMX API over at Cisco DevNet: CMX Mobility Services for yourself, and post below if you have created something leveraging the API!

Welcome to our new CMX Mobility Services website on Cisco DevNet!


We are proud to announce the release of our newly redesigned website.   We have improved the structure of our content, so you’ll get more from a quick read.

You can find us at https://developer.cisco.com/site/cmx-mobility-services/


In addition to the changed design and layout of the pages, new functions have been implemented in this version.


Design and Navigation

The design of the web pages and the structure of information have been changed to improve overview and usability.  The new design and colors now reflect the general Cisco DevNet image and ambiance.


Product Overview

We have created three new Learning Labs for CMX Mobility Services. Try them!


Access to Sample Code and Scripts
The presentation of the code repositories in this section has been improved with more information and an optimized navigation structure.


CMX Sandboxes

It is now possible to view the Sandbox topology and to read about CMX user interface, as a place to get started learning CMX REST APIs.


Blogs and Forum Posts

See what others are saying about CMX Mobility Services.  The blogs and other user resources are still available, and only a single click away.


We hope you will enjoy our new site. If you have questions, comments or suggestions please send them to Matthew Farrell at matfarre@cisco.com

With CMX Analytics, people often ask how long the data is kept for.


The answer is "it depends", but here is some more details.


First, in memory we keep the ACTIVE clients list, this is the list of clients that we are actively getting data from the WLC from until the following


The second duration of devices that are kept is the location history table.

PURGE TIME = Screen Shot 2016-10-05 at 4.44.27 PM.png

Job run at Midnight for all MAC address > Days configured in History Pruning Interval.

This is the BIG table that is purged daily.

The THIRD set of purging is for REPEAT visitors.  Each MAC address is classified as a NEW or REPEAT visitor.

To do that, we need to who has been here before, for that we maintain a list of MAC addresses and say where the device has been in the ZONE at a MONTHLY level

For example, we would keep that MAC address aa:bb:cc:dd:ee:ff has been in the venue in July, August and Sept and then in the month of Jan, we would delete the data from July.

For example, this query that show over 6534 devices, over 3384 are new and 3150 are devices which have been seen more then once.

Screen Shot 2016-10-05 at 4.48.33 PM.png

Finally, when we do have analytics data about a zone, we can keep the aggregated data for a total of up to 8 years back so you can look at year one vs year two.  CMX is not that old, but the data is still there.

The only limit here is based on harddrive space.


CMX Analytics Heatmaps

Posted by dsladden Oct 4, 2016

Author: Darryl Sladden, Senior TME Manager

Date:   October, 2016

Subject: CMX 10.2.x HEATMAPS


Cisco CMX Analytics has the capability to create an hourly heatmap of client activity for any floor as part of the analytics capabilities.







The heatmap is generated hourly and is in the format of a JPG image which is accessible via the GUI. As part of the GUI you can play the hourly generated heatmaps back to form an understanding of what has happened on the floor during the day.


The heatmap is generated by a dataset of devices that have had a locations calculated during the last hour.  It is important to note the following;


  • - With Standard location, a datapoint for location is generated when the client decides to send a PROBE packet out.  This is normally done when a client first enters a venue, and is done less frequently when a device is in a venue and is in a static location.  This would mean location such as lounges that have static devices may not have as many devices shown with standard location as would be shown with HyperLocation.  In addition, places where devices first encounter WIFI access points will be shown hotter.
  • - A new location is usually determined when a device has moved, as such for a large number of static devices, there will not be a lot of location calculated.
  • - With Hyperlocation, a datapoint is calculated much more frequently, approximately one time each 15 seconds.  These datapoints give a more realistic view in the heatmaps of where devices are.
  • - The data in a heatmap and the colors are based on a logarithmic scale where the specific color will not associate to a specific number.
  • - The primary uses cases for HEATMAP is as a comparison tool for two different events or two different days.  Given the same pattern of visitors the heat map would appear in the same way.





  1. 1) Compare different days of the weeks to see if different days have specific times or places then a benchmark day
  2. 2) Compare different events to see if the timing of the event resulted in different enterances uses or different location of devices





  1. 1) Determine and compare different areas of the venue to see where CONNECTED devices are more likely to be.
  2. 2) Determine the flow of devices in and out of a venue during an event.






HyperLocation refers to the uses of Datapackets and if available Angle of Arrival for the determination of a client location.  HyperLocation can be accomplished with either a standard AP in Enhanced Local Mode where it scans for other clients, it can be accomplished with a dedicated monitor mode radio such as the RM3010 or, in the most advanced method is accomplished with a dedicated HyperLocation Antenna Array and a RM3010L module.

Author: Darryl Sladden, Senior TME Manager

Date:   October, 2016

Subject: CMX 10.2.x ANALYTICS Device Counts


Change from 10.2.0 to 10.2.2




Reports with the selection of multiple Zones or multiple Floors in 10.2.0 DEDUPLICATE visits.  For example, when the same device visits ZONE 1 and ZONE 2.


With 10.2.0 it is counted as 1 VISIT in a report of ZONE 1 and ZONE 2.


With 10.2.2 we do NOT DEDUPLICATE visits when the same device visits ZONE 1 and ZONE 2.


With 10.2.2 it is counted as 2


This can result in the exact same report showing a much higher number of devices with CMX 10.2.2 vs CMX 10.2.0.


This issues occurs with a CAMPUS report if you have multiple top level CAMPUS, but is not the case if you have a single top level CAMPUS for campus level reporting.


If you select a report with more than one zone/floor/campus, you will have this issue.


The workaround is to TAG multiple CAMPUS/BUILDINGS/FLOORS/ZONE with the same TAG and report at the TAG level.


This change is NOT retroactive.




In 10.2.0 there were some errors in calculating location of some devices due to heat map failure.

This resulted in devices that had not been in the venue for a long time to not be counted.


With 10.2.0 the total number of devices with locations calculated is lower due to this error.


In 10.2.2 we have corrected and expanded the HEATMAPS (an internal construct that is used for calculating how far from an AP a client is), this results in more device having a valid location calculated.


With this change you will see more devices that are seen for a short period of time if the AP are in public spaces.  You will need to filter devices that are seen for a short period of time if you want this removed from your dataset.   This should be done by setting the DWELL TIME FILTER to 30 mins


With 10.2.2 device count in analytics will be higher due to this correction.


This change will persist in all future releases.


CHANGE IN EXIT NOTIFICATION (Does not impact counts or Dwell time)


In 10.2.0 we exit devices from zones when a device is not reported from WLC after 20 mins.


In 10.2.2 we exit devices from a zone when a device is not reported from WLC after 10 mins.


This change will impact the DETECT and LOCATION page device counts.





A small number of devices have unrecognized OUI and send locally administered (ie RANDOM) MAC addresses.  These should be filtered out by the location engine.  This filter remains constant in 10.2.0 and 10.2.2 and will remain the same in 10.2.3.


The OUI list of valid MAC addresses is updated as part of the ongoing system updates.


Please note, Apple devices, when they are not associated to WIFI do not send out a valid MAC addresses and are not included in PROBING device counts by default.




There can be a large number of short duration devices in the any CMX analytics dataset if the AP are outdoors and can hear devices that are very infrequent, such as a car driving by. The method to filter these short duration devices would be to use the DWELL TIME FILTER and set the minimum DWELL time for a report to 30 mins. 







The Filtering for RSSI signal strength is based on at least one AP reporting the signal at a value of which is higher than a preset value.    The default value in 10.2.0 was -128.  The default value in 10.2.2 was set to -85.  The user can configure this value to what is appropriate to the specific venues.  If the values where changed when the report was created, then you would have resulted in different values from 10.2.0 vs 10.2.2.  A recommended value for this setting in many WIFI networks is -75.




In Presence the calculation of a VISITORS in a specific ZONE is based on the device being present in a ZONE for 8 minutes in the last 20 mins.  The device count widget also shows the total number of detected visitors. The visitors have to be in a zone for a specific length of time (configured) and at a specific signal strength.





There is an error in the repeat visitor count in CMX 10.2.2.  They show as 100% repeats in many cases.  This is fixed in 10.2.3. 


There is no workaround to correct in CMX 10.2.2

This blog was generated from CDN blog

Created by: Seetharaman Renganathan on 13-09-2010 08:00:56 AM
Hey Developers , 

Need to add Cisco Unified Presence functionality to your Context-Aware Service Application? 

Here comes the best solution for you. ContextAware  Sample application  has now been extended to support Cisco Unified Presence functionality , check out the
ContextAware_CUP Integrated Sample application.

This  Sample Application

  1)Helps you understand the integration of 2 different technologies, the ContextAware Mobility and Cisco Unified Presence,

  2)Its a simple webclient application, which gets the single/multiple user location details along with the users presence information.

This sample Application documentations can be found under the Resources section of the Context-Aware Mobility server Tech Center.

So why are you waiting still, just download this web  application & deploy in any web server like tomcat have fun to  add CUP functionality to your Context-Aware Service Application .

This blog was generated from CDN blog

Created by: Seetharaman Renganathan on 13-09-2010 07:39:20 AM
Hi Fellow Developers,

Are you looking for samples to help you understand the use of Context Aware API¿s?

The Context Aware technology now provides a solution to view the SOAP messages and sample JAVA source code for these Context Aware APIs. To find out more, check the MSE API LookUp Sample application.
The goal of this sample application is to demonstrate the ease of use of MSE APIs and provide SOAP messages that are ready to use.

The documentation for the Context Aware API LookUp Sample Application can be found under the Resources section of the Context-Aware Mobility server Tech Center.

Download the Context Aware MSE API LookUp Sample Application and deploy it in your web server (preferably tomcat) and have fun testing the functionalities in your Context-Aware Service Enabled Wireless building.

This blog was generated from CDN blog

Created by: Seetharaman Renganathan on 21-06-2010 09:05:27 AM
Hi Fellow Developers,

Are you looking for a Cisco Context-Aware Service Sample Application to understand the use of Context Aware APIs in a real time scenario?

The Context Aware technology has created a solution to:

1. Track stolen assets like laptops, PDAs, smartphones, etc.
2. Monitor wired critical network assets, such as switches, routers, etc.

To find out more, check out the University Asset Tracker Sample Application.

The goal of this sample application is to demonstrate the following:

    *     Ease of integration with MSE.
    *     Monitor the Wireless/Wired Assets.
    *     Send notifications to external users or systems when stolen/missing assets are located.
    *     Send notifications  to external users or systems when wired assets are no longer detected on the network.

The University Asset Tracker is a simple web client application, which tracks down the stolen assets when they reappear on the university network.

This application is supported in two formats:

1.    CISCO_University_Asset_Tracker.exe for windows, it supports the following versions of windows.

    * Windows XP
    * Windows 7
    * Windows Vista

2.    CISCO_University_Asset_Tracker.bin for Linux, it supports Linux with UI enabled option.

The documentation for the University Asset Tracker Sample Application can be found under the MSE Context Aware platform Sample Application section of the Context-Aware Mobility Tech Center.

Download the University Asset Tracker Sample Application and deploy it in your web server (preferably tomcat) and to start testing the functionalities in Context-Aware Service.

This blog was generated from CDN blog

Created by: Prabhu Punniamurthy on 09-02-2009 10:25:33 AM
Hey Context-Aware API developers , are you confused or not sure how to start with Context-Aware Mobility APIs?

Here comes the best solution for you, Popular Integrated Development Environment(IDE) such as Eclipse and Microsoft Visual Studio has now been extended to support Context-Aware APIs .

These IDE Plug-ins will

  • Reduces the initial learning of Development Environment
  • Simplifies application integration
  • Accelerate partners adoption of Cisco Wired/Wireless technologies
  • Provides Features such as Templates , Code Snippets and InteliiSense /Starter Kits
  • Auto generation of codes
IDE Plug-ins and their documentations can be found under the Resources section of the Context-Aware Mobility Tech Center.

So why are you waiting still,  just download these pug-ins and have fun in playing with the Context-Aware APIs .


Filter Blog

By date:
By tag: