Documentation
NAV Navbar
  • Intelligent Tracking Prevention
  • Intelligent Tracking Prevention

    Introduction

    Apple's Intelligent Tracking Prevention (ITP) is a technology aimed at increasing user privacy by preventing unwanted tracking by websites and the various scripts installed on them. It is implemented in all Safari browsers: in desktop computers (Macintosh) but more importantly, in all iOS devices (iPhone and iPad). Contrary to previous versions, ITP's latest version, 2.1, brings some major restrictions to what operations JavaScript-based software can perform, especially regarding cookies. While previous versions mainly targeted online advertising systems, the newest version is much more agressive and impacts standard Web Analytics software such as Google Analytics in a very serious way. A/B testing tools are also impacted, maybe even more than analytics systems, as we'll discuss in this article.

    At the time of writing this article (June 2019), we believe Kameleoon is the only Conversion Optimization platform that automatically and transparently handles Apple's ITP. We think it's a strong competitive advantage that brings real value to our customers, and thus decided to write a documentation article dedicated to ITP, in the hope of raising awareness among web marketers on this important topic. We will focus on the following points:

    What Intelligent Tracking Prevention is and why it really matters for your website

    Cookies and ITP 2.1 Technical Restrictions

    Intelligent Tracking Prevention implements a lot of restrictions concerning the usage of cookies. It is thus important to evoke a few important characteristics of cookies here.

    Cookies are (very) small pieces of data that are stored in the browser. Very often, they represent unique, randomly generated identifiers. Every cookie is tied to a domain (such as www.example.com) and has an expiry date. Any time a server call is made to any domain linked to one or more cookies stored on your computer, the data contained in these cookies is sent to that web server, automatically and transparently. This means cookies are constantly being added to most web requests, with two immediate consequences: the weight of web requests increases, slowing your browsing speed; and your browser can transfer data to remote servers without your explicit consent, allowing you to be identified and tracked. As cookies are usually quite small, the first issue is generally not that important. However, the second is much more far-reaching and this is what advocates of user privacy are trying to fight.

    This is of course a delicate topic, since cookies are being used for very different purposes on the web. Many of them, due to the stateless nature of the HTTP protocol, are needed for basic website operations - such as e-commerce transactions - the web would not work without. With ITP, Apple's developers are trying to separate the "useful" cookies (those mandatory for the normal operation of a given web site) from the "bad" ones (those associated with advertisement, tracking and unwanted privacy breaches).

    Cookies can be set via two different mechanisms: either when a call is made to a web server (the cookie is then set via the addition of an HTTP header in the HTTP reply), or via JavaScript code. The associated cookie domain depends on the method used: for server cookies, it is the domain where the server call was made. For JavaScript cookies, the domain is that of the page where the JavaScript code is run (NOT the JS file host domain!). Third-party cookies refer to cookies sent to a domain different than the web page you're currently viewing. For instance, if you're browsing www.randomshop.com, but for whatever reason a call is made to tracking.ad-system.com, cookies associated with www.ad-system.com will be sent along the HTTP query. They will be considered third-party cookies within the context of your current browsing session on www.randomshop.com.

    Finally, it's worth noting that when a cookie expires, it is removed from your browser / computer entirely. However, as the expiry date was normally chosen by the entity setting / controlling the cookie, it could be set very far in the future (10 years or more), ensuring that it would basically never get deleted. As we'll see, things are changing with ITP.

    The two main cookie restrictions imposed by ITP 2.1 are the following:

    Since Web Analytics software does not generally rely on third-party cookies, the first restriction mainly concerns advertising systems. We will thus (mostly) focus on the second restriction in the rest of this article.

    Consequences of this limitation for web-based analytics & testing platforms

    Capping JavaScript based cookies to a 7 day expiry is very agressive. Almost all Web Analytics platforms (including Google Analytics, the most popular solution) are now JavaScript / Front-end based, and they all use a cookie to store the visitor's unique identifier. This identifier is randomly generated on the first pageview of the first visit of this user to your website. Later, on all subsequent pageviews and visits, it is reused to identify this particular user. This allows for instance Google Analytics to know if a visitor is a new visitor (new identifier / cookie is generated) or a returning one (a previously set cookie was found). Since cookies now have a lifetime of 7 days in Safari (overriding, of course, the default expiry date normally set by GA, which is obviously much longer) a visitor making a first visit on Monday, then returning on Tuesday of the following week will be considered as a totally new visitor. His new visit won't be linked to the previous one. As a result, your new visitor metric, as reported by Google Analytics, is completely false for any Safari trafic. Many, many other metrics are completely distorted as well (number of unique visitors, time to buy, etc). It's actually hard to trust any number produced by cookie-based analytics solutions anymore!

    For desktop websites, this could be considered a minor nuisance, since Apple's Safari desktop marketshare is relatively low (between 5% and 10% generally). We would still argue that it is a serious issue for accurate web analytics and data analysis. For mobile heavy websites, it is a totally different story: as the marketshare of iOS devices is closer to the 40%-50% range, almost half of your trafic can be affected! ITP 2.1 started being rolled out in February 2019, so the effect was not immediate, but it is to be expected that eventually all iOS devices will be using ITP 2.1 (or higher).

    The majority of testing and conversion optimization platforms also bundle web analytics capabilities. Of course, these suffer from the exact same issues we just described, which is problematic in itself. However for A/B experiments, an even more serious obstacle appears. Each time a visitor is bucketed into an experiment, a variation is selected for him. This variation must be remembered, since if the visitor comes back, he must see the same variation he saw previously. All front-end testing solutions store this information (association between experiment and variation) in a cookie. So, with ITP, a visitor coming back more than 7 days after initially triggering an experiment runs the risk of seeing a different variation. Under these conditions, A/B testing is useless and does not produce reliable results.

    For much more technical details, see this excellent (and very exhaustive) blog post on ITP and its implications for Web Analytics. This post also provides a list of solutions (or rather, workarounds) to the ITP challenge, which is what we'll also discuss in the next section.

    Available workarounds

    Cookies set by JavaScript, which were not 100% trustworthy even before ITP because users could manually wipe them out (as with any cookies), basically became totally unreliable with ITP. Several solutions immediately come to mind, but most of them are not really compatible with web-based analytics software.

    For instance, software vendors could try to set the required cookies on the server via an HTTP header, rather than via their JavaScript code. But to achieve that they would need access to the back-end code, which adds a lot of complexity. Most of these solutions are installed on the premise of an extremely simple setup (just a static JavaScript file to include in the HTML source code, or even simpler, deployed via a Tag Manager). If they required extensive efforts from the customer to be installed and maintained, these solutions would probably not be used at all. Thus, any workaround relying on HTTP headers (even via edge workers, as is mentionned in the previously linked post) is not really a viable option for analytics vendors - although it can still be an interesting approach for tech-savvy customers that want to "tweak" their installed tools themselves.

    There is however one solution that does not require cooperation / developments between customers and software vendors: the use of Local Storage instead of cookies.

    Local Storage to the rescue

    Local Storage, a standard Web technology supported by virtually all browsers in 2019, has some common characteristics with cookies. It's basically a data store in the browser, so it can handle data in the same way that cookies do. It can actually store much more data than cookies: the exact amount is browser dependent, but is usually around a few MBs (compared to a few KBs for cookies). Contrary to cookies, Local Storage content can be written only through JavaScript access, servers cannot store anything in this space. Local Storage data can only be retrieved via JavaScript as well; LS content is never sent out to remote HTTP servers, which makes it inherently more secure / private. Like cookies, LS data is tied to a domain, the domain of the web page where the JS code is executed. Once written, only code located in the exact same domain will be able to access this content (which is logical from a security point of view).

    Given all of its characteristics, Local Storage seems to be an ideal candidate to replace cookies. And ITP 2.1 does not place restrictions on Local Storage. Actually, several testing platforms already implement LS based storage instead of cookie based storage. However, there is a huge problem with Local Storage: it is limited to a single exact subdomain.

    New technology, but new limitations: back to where we started

    Unfortunately, naively implementing Local Storage support is far from being enough. This is because of the subdomain / protocol partitioning scheme used by Local Storage. Unlike cookies, where JavaScript code running on http://www.example.com can create a cookie associated with the domain *.example.com (which means that you can access this cookie from buy.example.com, or even buy.now.really.example.com), Local Storage is partitioned by exact subdomain AND protocol. If you write data on the LS on the subdomain http://www.example.com, you cannot access it later from a page athttp://buy.example.com, or even from https://www.example.com. Both protocol and domain have to match exactly.

    For this very reason, unless you're lucky enough to have a single site and a single subdomain throughout this entire site, you will still suffer from serious issues if you use a solution relying on an easy Local Storage implementation. These issues would manifest themselves in a way quite similar to those caused by ITP, although the root cause is totally different. As an example, let's imagine that most of your e-commerce website is hosted on https://www.randomshop.com, but the conversion funnel is on https://transaction.randomshop.com. If your visitor ID is persisted on the LS, your analytics solution will report two different visitors when a complete purchase occurs, one seen on www.randomshop.com and the other on transaction.randomshop.com. Data does not transition consistently from one domain to another, and such a solution would see the two as completely separate sessions, rather than a single visitor session.

    For A/B testing, this is - again - even worse. If you run an experiment modifying the navigation menu, with two variations, one user could very well be exposed to the first variation on the main site, but to the second one on the funnel! This is of course disastrous in terms of user experience, while also voiding the results of the experiment.

    So we went further and implemented a full cross-domain Local Storage solution, as even a small percentage of pages on a different subdomain on your website can mean unreliable and even false results for A/B experiments.

    How our engineering team implemented a cross-domain Local Storage for Kameleoon

    The genesis of this project

    We have to confess something here: we actually were lucky when we first saw the restrictions of ITP 2.1 and quickly understood that it would be mostly harmless for us. We were lucky, because our cross-domain Local Storage was already implemented and ready, and it had been so for a long time! We actually started the implementation of this project in 2014, and at the time, our reasons for doing so were of course completely unrelated to ITP. We just needed a large storage space on the browser, because we wanted to store all data gathered by Kameleoon on the front-end in order to run our predictive machine learning algorithms on that data (Yes, you read that correctly: we run neural network algorithms directly on the browser, in JavaScript. It works very well, but that's a topic for a different article). So of course, we investigated the Local Storage approach, which was correctly supported on most browsers even back in 2014. We quickly understood that it was ideal for what we wanted to achieve, but that the domain partitioning scheme would be a real problem that would make it unusable for most of our customers. So we decided to implement a cross-domain Local Storage.

    At the time, we estimated the amount of development time required by our JavaScript engine team to be around 3 months. We were completely wrong: it actually took about 3 years (!) to finish this work and have a reliable, production-ready cross-domain LS - even with a stellar team of JS experts. We clearly underestimated the amount of effort required, because the base idea behind cross-domain LS is not that complex (or just seems that way). This is for example (correctly) mentionned in Simo Ahava's blog. Instead of fetching data from the Local Storage directly via JavaScript code running in the context, you load an iFrame (hosted on a different domain) and send a data query to this iFrame via the postMessage() method. Then within the iFrame, with JavaScript code running on a different domain / context, you retrieve data from the LS associated and then again use the postMessage() method to send back the data to the "host" page / domain. To store data, the principle is exactly the same, you send a message to the iFrame containing the data to be written.

    This works well, because whatever the URL / domain of your host pages, you always load the same iFrame hosted on the same domain, and data is always read and written in this domain. Thus the full visitor journey, as seen by the data store, always occurs in a unique domain. In addition, it's worth noting that the iFrame file is cached on the browser (it's a static file), so this doesn't hurt performance too much.

    As is often the case with software development, the main difficulty of the implementation was not with the base mechanism, but rather with some unforeseen challenges.

    The 3 main challenges we faced

    Without going into too much details, because we want to keep some in-house implementation features secret, these are the three nightmares we encountered on our path to a reliable cross-domain Local Storage:

    1. Asynchronous I/O. This was clearly the most painful point. Since reading and writing data has to happen via postMessage(), all data reads and writes happen asynchronously. This creates a very difficult programming environment for our engine, especially in JavaScript where developers are not used to this kind of hindrance. All the code of our engine had to be rewritten within this new paradigm, and carefully checked.

    2. Multiple open tabs on the same site / customer. Associated with the first point, this makes for an even more unstable environment where data can be updated randomly and at any time. Indeed, several tabs can be reading and writing at the same time on the LS iFrame, and all of that happens asynchronously! In addition, debugging these kind of issues is extremely difficult, as it happens only on our customer's visitor's browsers, where it's impossible to add any development or logging tool.

    3. Performance issues. One of our main focuses at Kameleoon has always been performance: we consider our platform to be the best - regarding performance - among front-end optimization tools. Using an additional external iFrame added additional hurdles, and we had to implement several tweaks to continue supporting our Anti-Flicker technology within this context.

    From a timeline perspective, the initial base implementation was done in about 3 months as initially planned. However, it took an additional 18 months to refactor and reimplement our entire engine to fix the 3 issues mentionned above. Thus we deployed the first implementations - in beta - to selected customers almost two years after. Following this, it took about one more year to fix all remaining issues, as some were quite difficult to reproduce due to their volatile nature. We still remember fixing one particularly nasty bug after months of tracking and debugging! In the end, we consider that our implementation reached production maturity in 2017, 3 years after the first lines of code were initially written. Our first successful pilots in Predictive Targeting / A.I. Personalization, which relied heavily on this feature, were launched around this time as well.

    Current situation

    When the first versions of ITP came out, we actually had to perform some minor changes to our system. Strangely, external LS domains, in the same way as cookies, are actually treated as third-party context by all ITP versions. We initially hosted the common iFrame file on our own (Kameleoon) domain, not on the customer domain (mostly to facilitate setup), so the common LS data was considered to be third-party and was blocked by Safari. To fix this, we just changed the hosting of the iFrame so that it be hosted on the customer domain. Details about this transition are available here.

    So to enjoy ITP-free A/B testing and analytics on Kameleoon, make sure you use our Dual JavaScript & iFrame files implementation method. That's all there is to it, really!

    Two additional closing notes:

    Conclusion

    The Kameleoon team is extremely proud to provide our customers with this painless solution to the ITP 2.1 challenge. Thanks to it, our customers can continue to enjoy stable, reliable A/B testing on their mobile trafic, without having to spend precious time implementing custom hacks to mitigate ITP. As a software vendor, we strongly believe we should do the maximum to ease the lives of our customers and justify the price of our platform. This is a perfect example of this philosophy.

    Of course, ITP 2.1 is not the last move Apple will make in its war to safeguard user privacy on the Web. We will continue to monitor the evolution of Apple's ITP technology, as well as other initiatives from other browser vendors (Firefox has something similar to ITP, but not as extreme as of yet). Any breaking changes will be swiftly analyzed and solutions will be provided as soon as possible. We have a very sizable headstart over competing platforms on this topic right now (as pointed out, 2 to 3 years of development time), and we intend to keep it.