Intelligent Tracking Prevention
At the time of writing this article (June 2019), we believe Kameleoon is the only Conversion Optimization platform that automatically and transparently handles Apple's ITP. We think it's a strong competitive advantage that brings real value to our customers, and thus decided to write a documentation article dedicated to ITP, in the hope of raising awareness among web marketers on this important topic. We will focus on the following points:
- what exactly are the technical limitations imposed by ITP 2.1 and its consequences on the operation of Web Analytics and Conversion Optimization software;
- current available workarounds to bypass or limit the impact of ITP on your website's installed technologies;
- the details of Kameleoon's tracking implementation (how exactly did we solve the ITP challenge).
What Intelligent Tracking Prevention is and why it really matters for your website
Cookies and ITP 2.1 Technical Restrictions
Intelligent Tracking Prevention implements a lot of restrictions concerning the usage of cookies. It is thus important to evoke a few important characteristics of cookies here.
Cookies are (very) small pieces of data that are stored in the browser. Very often, they represent unique, randomly generated identifiers. Every cookie is tied to a domain (such as
www.example.com) and has an expiry date. Any time a server call is made to any domain linked to one or more cookies stored on your computer, the data contained in these cookies is sent to that web server, automatically and transparently. This means cookies are constantly being added to most web requests, with two immediate consequences: the weight of web requests increases, slowing your browsing speed; and your browser can transfer data to remote servers without your explicit consent, allowing you to be identified and tracked. As cookies are usually quite small, the first issue is generally not that important. However, the second is much more far-reaching and this is what advocates of user privacy are trying to fight.
This is of course a delicate topic, since cookies are being used for very different purposes on the web. Many of them, due to the stateless nature of the HTTP protocol, are needed for basic website operations - such as e-commerce transactions - the web would not work without. With ITP, Apple's developers are trying to separate the "useful" cookies (those mandatory for the normal operation of a given web site) from the "bad" ones (those associated with advertisement, tracking and unwanted privacy breaches).
Third-party cookies refer to cookies sent to a domain different than the web page you're currently viewing. For instance, if you're browsing
www.randomshop.com, but for whatever reason a call is made to
tracking.ad-system.com, cookies associated with
www.ad-system.com will be sent along the HTTP query. They will be considered third-party cookies within the context of your current browsing session on
Finally, it's worth noting that when a cookie expires, it is removed from your browser / computer entirely. However, as the expiry date was normally chosen by the entity setting / controlling the cookie, it could be set very far in the future (10 years or more), ensuring that it would basically never get deleted. As we'll see, things are changing with ITP.
The two main cookie restrictions imposed by ITP 2.1 are the following:
- In all ITP versions, third-party cookies suffer from a variety of restrictions. The exact details are complex and out of the scope of this article, but in short, third-party cookies are blocked by Safari 24 hours after they have been set. They are not technically deleted, but are no longer being sent to the third party server via HTTP requests. In our previous example, a visitor coming to
www.randomshop.comgets a cookie from
www.ad-system.comvia an HTTP call to that server during their first visit. If he comes back more than 1 day later, his next HTTP request to
www.ad-system.comwon't get the original cookie data as it will be blocked.
Since Web Analytics software does not generally rely on third-party cookies, the first restriction mainly concerns advertising systems. We will thus (mostly) focus on the second restriction in the rest of this article.
Consequences of this limitation for web-based analytics & testing platforms
For desktop websites, this could be considered a minor nuisance, since Apple's Safari desktop marketshare is relatively low (between 5% and 10% generally). We would still argue that it is a serious issue for accurate web analytics and data analysis. For mobile heavy websites, it is a totally different story: as the marketshare of iOS devices is closer to the 40%-50% range, almost half of your trafic can be affected! ITP 2.1 started being rolled out in February 2019, so the effect was not immediate, but it is to be expected that eventually all iOS devices will be using ITP 2.1 (or higher).
The majority of testing and conversion optimization platforms also bundle web analytics capabilities. Of course, these suffer from the exact same issues we just described, which is problematic in itself. However for A/B experiments, an even more serious obstacle appears. Each time a visitor is bucketed into an experiment, a variation is selected for him. This variation must be remembered, since if the visitor comes back, he must see the same variation he saw previously. All front-end testing solutions store this information (association between experiment and variation) in a cookie. So, with ITP, a visitor coming back more than 7 days after initially triggering an experiment runs the risk of seeing a different variation. Under these conditions, A/B testing is useless and does not produce reliable results.
For much more technical details, see this excellent (and very exhaustive) blog post on ITP and its implications for Web Analytics. This post also provides a list of solutions (or rather, workarounds) to the ITP challenge, which is what we'll also discuss in the next section.
There is however one solution that does not require cooperation / developments between customers and software vendors: the use of Local Storage instead of cookies.
Local Storage to the rescue
Given all of its characteristics, Local Storage seems to be an ideal candidate to replace cookies. And ITP 2.1 does not place restrictions on Local Storage. Actually, several testing platforms already implement LS based storage instead of cookie based storage. However, there is a huge problem with Local Storage: it is limited to a single exact subdomain.
New technology, but new limitations: back to where we started
http://www.example.com can create a cookie associated with the domain
*.example.com (which means that you can access this cookie from
buy.example.com, or even
buy.now.really.example.com), Local Storage is partitioned by exact subdomain AND protocol. If you write data on the LS on the subdomain
http://www.example.com, you cannot access it later from a page at
http://buy.example.com, or even from
https://www.example.com. Both protocol and domain have to match exactly.
For this very reason, unless you're lucky enough to have a single site and a single subdomain throughout this entire site, you will still suffer from serious issues if you use a solution relying on an easy Local Storage implementation. These issues would manifest themselves in a way quite similar to those caused by ITP, although the root cause is totally different. As an example, let's imagine that most of your e-commerce website is hosted on
https://www.randomshop.com, but the conversion funnel is on
https://transaction.randomshop.com. If your visitor ID is persisted on the LS, your analytics solution will report two different visitors when a complete purchase occurs, one seen on
www.randomshop.com and the other on
transaction.randomshop.com. Data does not transition consistently from one domain to another, and such a solution would see the two as completely separate sessions, rather than a single visitor session.
So we went further and implemented a full cross-domain Local Storage solution, as even a small percentage of pages on a different subdomain on your website can mean unreliable and even false results for A/B experiments.
How our engineering team implemented a cross-domain Local Storage for Kameleoon
The genesis of this project
postMessage() method to send back the data to the "host" page / domain. To store data, the principle is exactly the same, you send a message to the iFrame containing the data to be written.
This works well, because whatever the URL / domain of your host pages, you always load the same iFrame hosted on the same domain, and data is always read and written in this domain. Thus the full visitor journey, as seen by the data store, always occurs in a unique domain. In addition, it's worth noting that the iFrame file is cached on the browser (it's a static file), so this doesn't hurt performance too much.
As is often the case with software development, the main difficulty of the implementation was not with the base mechanism, but rather with some unforeseen challenges.
The 3 main challenges we faced
Without going into too much details, because we want to keep some in-house implementation features secret, these are the three nightmares we encountered on our path to a reliable cross-domain Local Storage:
Asynchronous I/O. This was clearly the most painful point. Since reading and writing data has to happen via
Multiple open tabs on the same site / customer. Associated with the first point, this makes for an even more unstable environment where data can be updated randomly and at any time. Indeed, several tabs can be reading and writing at the same time on the LS iFrame, and all of that happens asynchronously! In addition, debugging these kind of issues is extremely difficult, as it happens only on our customer's visitor's browsers, where it's impossible to add any development or logging tool.
Performance issues. One of our main focuses at Kameleoon has always been performance: we consider our platform to be the best - regarding performance - among front-end optimization tools. Using an additional external iFrame added additional hurdles, and we had to implement several tweaks to continue supporting our Anti-Flicker technology within this context.
From a timeline perspective, the initial base implementation was done in about 3 months as initially planned. However, it took an additional 18 months to refactor and reimplement our entire engine to fix the 3 issues mentionned above. Thus we deployed the first implementations - in beta - to selected customers almost two years after. Following this, it took about one more year to fix all remaining issues, as some were quite difficult to reproduce due to their volatile nature. We still remember fixing one particularly nasty bug after months of tracking and debugging! In the end, we consider that our implementation reached production maturity in 2017, 3 years after the first lines of code were initially written. Our first successful pilots in Predictive Targeting / A.I. Personalization, which relied heavily on this feature, were launched around this time as well.
When the first versions of ITP came out, we actually had to perform some minor changes to our system. Strangely, external LS domains, in the same way as cookies, are actually treated as third-party context by all ITP versions. We initially hosted the common iFrame file on our own (Kameleoon) domain, not on the customer domain (mostly to facilitate setup), so the common LS data was considered to be third-party and was blocked by Safari. To fix this, we just changed the hosting of the iFrame so that it be hosted on the customer domain. Details about this transition are available here.
Two additional closing notes:
Although we can guarantee that our analytics core is unaffected by ITP, and that all our reported trafic data is reliable, of course we cannot guarantee the same for third party web analytics platforms. Our integration bridges with partner tools continue to work in the same way as before. We recommend that customers discuss the situation with the vendors of their chosen solution.
Our server-side SDKs (and Mobile SDKs) are completely unaffected by ITP, as analytics tracking calls are made server to server, never on the front-end. If you use the Back-End / Front-End bridge, a cookie will be used to synchronize the kameleoonVisitorCode, but it will be set via the server (HTTP header), not via the front-end, and will be considered a first-party cookie, and thus not affected by ITP.
The Kameleoon team is extremely proud to provide our customers with this painless solution to the ITP 2.1 challenge. Thanks to it, our customers can continue to enjoy stable, reliable A/B testing on their mobile trafic, without having to spend precious time implementing custom hacks to mitigate ITP. As a software vendor, we strongly believe we should do the maximum to ease the lives of our customers and justify the price of our platform. This is a perfect example of this philosophy.
Of course, ITP 2.1 is not the last move Apple will make in its war to safeguard user privacy on the Web. We will continue to monitor the evolution of Apple's ITP technology, as well as other initiatives from other browser vendors (Firefox has something similar to ITP, but not as extreme as of yet). Any breaking changes will be swiftly analyzed and solutions will be provided as soon as possible. We have a very sizable headstart over competing platforms on this topic right now (as pointed out, 2 to 3 years of development time), and we intend to keep it.