Early last week the personal diary app Path became the fulcrum of a massive discussion about how cavalier mobile apps are getting with harvesting your, presumably, personal information. Path was found by a developer to send the entire contents of its users Address Books, where, it was uncovered, it was being stored locally.
Predictably, when privacy issues are concerned, there was an outcry about how Path handled the data, and many decried it for being underhanded or even flat out lying about its procedures. But, as with most things, there is a bigger story here and it turns out that what Path was doing was far from out of the ordinary.
In fact, according to independent testing shared with The Next Web by developer Paul Haddad of Tapbots, many apps that are far more popular than Path are transmitting your data, and many of them have been doing it completely without your knowledge or consent until the Path story blew up, forcing them to immediately update their apps.
During testing, Haddad discovered that Foursquare, for instance, was uploading all of the email addresses and phone numbers in your address book with no warning and no explicit consent given. This is similar to the way that Path was working pre-update and the way that Instagram worked before its recent update.
You can see from the following list that transmitting your personal information is far from out of the ordinary.
Apps that do send data, with no warning
Foursquare stands alone here as an app that was, until an update issued on February 14th, sending personal data with no warning. This is similar to the previous behavior of Path that got it in so much hot water. Since the update, Foursqare now warns users before uploading data. Foursquare says that, while it was uploading the data, it was not storing it.
- Foursquare (Email, Phone Numbers no warning)
Apps that do send data, after warning
Apps in this category send copious amounts of data including email addresses, first and last names and phone numbers. Path appears to be sending the most information out of the apps that were tested.
- Path (Pretty much everything including mailing addresses, after warning)
- Instagram (Email, Phone Numbers, First, Last , after warning)
- Facebook (Email, Phone Numbers, First, Last, after warning)
- Twitter for iOS (Email, Phone Numbers, after warning)
- Voxer (Email, First, Last, Phone numbers, after warning)
Apps that appear not to send data
The apps in this category do not appear to send any information, though they all link against the Address Book framework in iOS. This means that they could theoretically grab the data to use locally, in a completely natural and normal way for apps that use contact info. There is a possibility that some interaction with the app could access the data and send it to the server, but no evidence of that was found in testing.
- Google+ (Nothing obvious, links against Address Book)
- Find My Friends (Nothing obvious, links against Address Book)
- Skype (Impossible to tell, links against Address Book)
- Yahoo! Messenger (Nothing obvious, links against Address Book)
- Quora (Nothing obvious, links against Address Book)
- Textfree (Nothing obvious , links against Address Book)
- AIM (Nothing obvious, links against Address Book)
It’s clear that the lambasting that Path took in the internet press put the fear of god into any app that transmitted user data without explicit permission. This is probably a good thing, but there are still questions that need to be asked.
Why is the data needed?
It stands to reason that any applications that try to connect you with your friends might have to have access to the Address Book. That’s all well and good. But there are simple methods that allow developers to anonymize the data by ‘hashing’ the data and saving a checksum, ditching the plain text of the data, which never gets transmitted or stored on the servers at all.
This is a much more secure process than the barely-there security of an HTTPS transmission. If you’d like more detail, developer Matt Gemmell has prepared a simple and informative guide to the process.
The important point here is that developers do not have to upload plain text data to their servers in order to offer these convenience features. They can upload hashed, and therefore anonymous, data instead. Then they can use that data to provide the features without ever having seen or stored the plain information.
The answer is likely not that these developers are evil or looking to harvest your data. Instead, it’s likely to be a simple matter of them not understanding that there are better ways to go about it. Developers are only human and many teams have a limited amount of resources
“I think its a mix of being the easiest way to handle finding “friends” and lack of knowledge of alternative solutions,” Haddad told us.
We spoke to Impending’s Phill Ryu briefly and he supported that conclusion, saying that “maybe sometimes there’s intent to use it later, but I honestly think in most cases it’s just a mentality of ‘that disaster won’t happen to us, and better us working on fixing bugs in the product or adding new features”
That doesn’t, however, absolve developers from their handling of people’s personal data. There is also the matter of why it wasn’t disclosed that the apps were gathering the data after all. This kind of lack of disclosure is one of those things that serves to show that there is a genuine disconnect between the way that developers think about data and the way that users do.
To a developer, any bit of data that they can use to make the app work better or more seamlessly is treated as a resource to be harvested. Not to use in any evil way, but perhaps not treated with the sensitivity that should be afforded people’s personal data.
This is likely why, even when the decision was made to have the apps transmit data back, it wasn’t even thought of to disclose it or warn the user beforehand. It’s an incredible lapse of trust from a user standpoint, but the developers of these apps likely never thought twice. This brings up another really good question, whose job is it to make sure that your data is protected, the individual developer, or Apple?
Is it Apple’s job to force developers into data disclosure?
Buried at the heart of this multilayered issue is the fact that there is nothing built into the Address Book framework that forces a developer to ask the user before accessing and transmitting the data in this way. The onus is completely on them to make sure that they say ‘hey, we’re going to grab your data and use it for this feature’.
In all honesty, action needs to be taken on both ends. Apple, as the company that sets policies for its App Store, has already stated in its terms that apps need to be forward about how they handle user data and never to do it without first alerting the user. Unfortunately, as we can see from the apps detailed below, this has not worked.
Apple clearly needs to change its policies in order to make it harder for developers, even incidentally, to upload your personal data, period. But the responsibility is also on developers to be more conscientious.
“I think it’s Apple’s responsibility to change their API to make access to Address Book something that must be explicitly allowed by each user,” Haddad told us, “I think its a developer’s responsibility to clearly lay out exactly what data is being sent and for what reasons.”
There’s no simple solution here. As developer Justin Williams notes, there are issues with simply adding another ‘warning’ box:
Tossing up another dialog asking for user confirmation doesn’t solve the problem users are faced with. It just puts a band-aid on it. At the core is a more fundamental problem in how iOS handles permissions and access to data. Basically, I have no idea what sort of permissions or access an app wants until I download it and launch it the first time. Moreover, I really don’t to see another dialog pop up in my face as I’m using an app.
He suggests that the App Store needs more transparency when it comes to permissions, allowing users to see what data the application will be using right up front. This is similar to the way that the Android Market and Windows Marketplace work on their respective platforms.
No easy answer
There is no ‘magic button’ to be had here. Yes, Apple will very likely have to begin enforcing access to the Address Book at the system level. But there are dangers that this will lead to ‘overload’, causing users to tap right by the warning and putting us in the same spot we are now.
Developers also need to be more conscious of privacy issues, and just how sensitive they are to most users. The disconnect between how users value personal data and how developers treat that data needs to be bridged.
Until that happens, we’ll keep talking about these kinds of privacy issues.
Now, the Data
Since we found the data interesting and it really makes an impact when you see it right there in your face, we decided to include Haddad’s testing methodology, if you’d like to reproduce it, along with the data intercepted during the test.
Note that all of the data you see here was transmitted directly over HTTPS, which is relatively secure, but the data is easily (as is demonstrated by the translations below) readable once on the server. This means that if can easily be used by anyone who gains access to the server and, of course, the companies themselves.
Haddad explains his testing methods:
All testing was done using the Charles web proxy. The device was restored to a blank state and two dummy contacts were added. The Charles SSL certificate was installed and accepted. And the proxy settings for iOS were set to point to a machine running Charles.
Each app was then launched and for most new accounts were created. For some the actions that caused Address Book data to be sent happened right away, for others I had to navigate through a few different screens. Then I searched through all the HTTPS requests to see if I could find any of the dummy address data.
The contacts used in the test:
[{"name":"John Doe","phones":["1 (234) 567-8910","(555) 555-5555"],"emails":["joe.doe@joe.com","joe.doe@work.com"],"record_id":0},{"name":"Jane Smith","phones":["(888) 888-8888","(222) 222-2222"],"emails":["jane@smith.com","jane@work.com"],"record_id":1}]
You can see the raw information passed to Facebook’s servers here.
Foursquare
As mentioned above, Foursquare updated its app earlier today to insert a warning before transmitting the data. Previous do this it was transmitting it with no warning, similar to the way that Path and Instagram had been working.
llAcc=65.000000&alt=203.481232&v=20120130&oauth_token=5U0YU34MEJPPOCK
O2OYSH1S4LGOVSA0B5R3P
Y4QSI5LVG0MM&requests=/users/search?llAcc=65.000000&phone=(888) 888-8888,(555)
555-5555,(222) 222-2222,1 (234) 567-
8910&alt=203.481232&v=20120130&email=jane@smith.com,
joe.doe@joe.com,jane@work.com,joe.doe@work.com&oauth_token=5U0YU34MEJPPOCKO2
OYSH1S4LGOVSA0B5R3PY4QSI5LVG0MM&altAcc=
11.956229&ll=33.035668,-97.091726&altAcc=11.956229&ll=33.035668,-97.091726
You can see the raw information passed to Foursquare’s servers here.
Instagram is another app which received an update after the Path issue, with a warning message now being delivered after you tap on ‘From my contact list’.
contacts=[{“email_addresses”:[“joe.doe@joe.com”,”joe.doe@work.com”],”phone_numbers”:[“1 (234) 567-8910″,”(555) 555-5555″],”first_name”:”John”,”last_name”:”Doe”},{“email_addresses”:[“jane@smith.com”,”jane@work.com”],”phone_numbers”:[“(888) 888-8888″,”(222) 222-2222″],”first_name”:”Jane”,”last_name”:”Smith”}]
You can see the raw information passed to Instagram’s servers here.
Path
<key>first_name</key>
<string>Jane</string>
<key>last_name</key>
<string>Smith</string>
<key>phone_numbers</key>
<array><string>(888) 888-8888</string>
<string>(222) 222-2222</string>
</array><key>street_addresses</key>
<array><dict><key>City</key>
<string>New York</string><key>Country</key>
<string>United States</string><key>CountryCode</key><string>us</string>
<key>State</key><string>NY</string><key>Street</key><string>1111 E. Wind St</string>
<key>ZIP</key><string>10000</string></dict></array>
You can see a raw dump of the data sent to Path’s servers here.
While Twitter does warn you before you give away your info, it has also admitted that it stores your data locally, on its severs for up to 18 months. That data can include “IP address, browser type, the referring domain, pages visited, your mobile carrier, device and application IDs, and search terms.”
email=joe.doe%40joe.com%2Cjoe.doe%40work.com%2Cjane%40smith.com%2Cjane%40work.com&map=true&phone=%2B12345678910%2C%2B15555555555%2C%2B18888888888%2C%2B12222222222&application_id=aPOST /1/users/lookup.json HTTP/1.1 Host: api.twitter.com User-Agent: Twitter-iPhone/4.0.1 iOS/5.0.1 Content-Length: 187 Accept: */* Authorization: OAuth oauth_nonce="473922CC-9888-4B86-8B13-54C5143D33E3", oauth_signature_method="HMAC-SHA1", oauth_timestamp="1329252602", oauth_consumer_key="WXZE9QillkIZpTANgLNT9g", oauth_token="118377126-bzFZK8iPhBtd9MdnWirpwPCwSADRLoaWwaea8GQ", oauth_signature="NyKdWSJWlnyaDSPTZEydKLUI%2FSI%3D", oauth_version="1.0" X-Twitter-Client-Version: 4.0.1 X-Twitter-Client: Twitter-iPhone X-Client-UUID: 30ddfa8c Accept-Language: en-us Accept-Encoding: gzip, deflate Content-Type: application/x-www-form-urlencoded Cookie: lang=en; _twitter_sess=BAh7CjoMY3NyZl9pZCIlMTcwMGUxZDU5NTg2OWVjNTg4MDY3N2ZhMjgyODQx%250AYTQ6D2NyZWF0ZWRfYXRsKwiSkp99NQE6B3VhMCIKZmxhc2hJQzonQWN0aW9u%250AQ29udHJvbGxlcjo6Rmxhc2g6OkZsYXNoSGFzaHsABjoKQHVzZWR7ADoHaWQi%250AJTkyNTVhNmUyMjI5ZTU3YjZhMWRmNTExYzdmNWQyNGRm--67dc797d4d913c651bc519aa24e91d501a5eecf3; guest_id=v1%3A132925250419628944; k=10.34.125.115.1329252504193246; twid=u%3D118377126%7Chw%2F2x7sznFxjpPLtArNg%2BlDqyU4%3D; t1=1 Connection: keep-alive Proxy-Connection: keep-alive email=joe.doe%40joe.com%2Cjoe.doe%40work.com%2Cjane%40smith.com%2Cjane%40work.com&map=true&phone=%2B12345678910%2C%2B15555555555%2C%2B18888888888%2C%2B12222222222&application_id=333903271
Voxer
POST /3/cs/update_addressbook?Rv_session_key=51eb5c4871af0868182b0cf82d924bbd&now=1329254317.017884&full=true HTTP/1.1
Host: prod-nr178.voxer.com:443
User-Agent: Voxer 2.3.10 rv:0684 (iPod touch; iPhone OS 5.0.1; en_US)
Content-Length: 393
Content-Type: application/json
Accept-Encoding: gzip
X-Voxer-JSON-Checksum: a98a48b219e7f278e28761b4a5a03a2b
Connection: keep-alive
Proxy-Connection: keep-alive[{“first”:”John”,”last”:”Doe”,”emails”:[{“l”:”work”,”v”:”joe.doe@work.com”},{“l”:”home”,”v”:”joe.doe@joe.com”}],”phones”:[{“l”:”mobile”,”v”:”1 (234) 567-8910″},{“l”:”iPhone”,”v”:”(555) 555-5555″}]},{“first”:”Jane”,”last”:”Smith”,”emails”:[{“l”:”work”,”v”:”jane@work.com”},{“l”:”home”,”v”:”jane@smith.com”}],”phones”:[{“l”:”mobile”,”v”:”(888) 888-8888″},{“l”:”iPhone”,”v”:”(222) 222-2222″}]}]
Voxer’s data was transmitted in readable form, so no decoding was needed.
Catch the latest Apple news right now at TNW Apple.
Get the TNW newsletter
Get the most important tech news in your inbox each week.