Cross-device experimentation

A customer’s journey is rarely confined to a single device. From smartphones to tablets to desktop computers, users regularly switch between devices, leaving our users to wonder: How can we connect the dots?

That’s where cross-device experimentation comes in. While it might sound straightforward, bridging the gap between devices is a nuanced challenge, and no solution is perfect. But with the right tools—like reliable user IDs and thoughtful data practices—you can unlock insights into your visitors’ experiences across all touchpoints.

What Kameleoon's cross-device experimentation can do

• Ensure variation consistency: Logged-in users will always see the same variation in an experiment, no matter what device they use.
• Enable unified targeting: Consider all visits and actions from a single user, regardless of the device.
• Provide accurate reporting: Accurately count unique users across devices to better understand their behavior.

Critical points and practical insights

• Cross-device tracking relies heavily on identifying users. If a user stays anonymous across devices, merging their data is impossible. No system can bypass this limitation.
• A reliable user ID (like one tied to a login) is essential for linking data across devices. Without this, merging browsing histories or actions isn’t feasible.
• Actions taken in an anonymous state (for example, browsing without logging in) can’t be tied to an identified user until a login occurs. Once logged in, a user’s history merges—but only moving forward.
  • However, it's important to note that for Web Expermimentation, once the custom data is provided, Kameleoon can retrieve it for all subsequent visits on the same device, even if the user has not yet logged in. This retrieval happens because the data is stored in local storage, which does not occur in feature experimentation.
• While cross-device experimentation works for many use cases, it won’t fit every experiment. Cross-device experimentation depends on your tech stack, user customer journeys, and goals.

Understanding identity boundaries

In the context of cross-device experimentation, identity boundaries refer to how Kameleoon recognizes and handles users across different sessions, devices, and states of identification. These boundaries are crucial in determining whether Kameleoon can link user data across experiences.

Kameleoon primarily distinguishes between two types of user identities:

• Anonymous users: These are users who haven’t logged in and are identified by a temporary device-based identifier (visitorCode). Anonymous identifiers are unique to the device and browser and do not persist across devices.
• Identified users: These are users who log in, providing consistent and unique identifiers (like a user ID). This identifier is tied to the individual user, allowing Kameleoon to merge their actions and history across devices and sessions.

Activating cross-device history reconciliation

To use cross-device history reconciliation, you must create a new Kameleoon custom data entry and activate the Use this custom data as a unique identifier for cross-device matching option. When activated, Kameleoon treats this custom data as a unique identifier for your visitors, which will be used to map several Kameleoon visits to a unique user. You also need to set the custom data value with a unique identifier, such as a user ID, using one of the acquisition methods, like the SDK addData() method or the Activation API.

note

If the option is inactive for your custom data, it indicates that you may already have an active custom data set up as a unique identifier. This is because you can only have one custom data of this type per project.

caution

Incorrect configuration can impact cross-device history reconciliation and other Kameleoon operations, such as targeting and bucketing. It's important to avoid assigning a constant value, such as 0, to all visitors, as Kameleoon will interpret them as identical unique visitors.

Cross-device history reconciliation for variation allocation consistency

The goal of cross-device history reconciliation for variation allocation is to ensure that visitors assigned to an experiment consistently see the same variation, regardless of the device they use to access the website.

However, it's important to acknowledge that maintaining this consistency can be challenging in certain scenarios. For example, if a user visits the website on their initial device, logs in, and triggers an experiment, Kameleoon may assign them to variation V1. If the same user then visits the website on another device, triggers the same experiment before logging in, they might be assigned to variation V2. This inconsistency occurs because the system has not yet recognized the user across devices. It's important to note that we intentionally do not override the variation allocation during a single visit to avoid disrupting the user experience.

To mitigate this issue and ensure consistency, it's crucial for Kameleoon to promptly recognize visitors throughout their website journey. By identifying visitors early on when they access the website from different devices, we can ensure they are consistently assigned the same variation. This approach allows Kameleoon to provide a seamless and coherent testing experience.

Kameleoon Web Experimentation

The Kameleoon Web Experimentation solution primarily utilizes client-side storage mechanisms to manage variant allocation (bucketing). Users' variant allocation is initially stored in local storage when they participate in an experiment. This storage ensures that a user consistently sees the same variant on the same device, even if the customer updates the traffic allocation of the experiment.

note

Rebucketing only occurs if explicitly requested, either by disabling a variant previously assigned to a user or forcing the reallocation of the experiment's traffic.

If the same user logs in on a different device, Kameleoon retrieves the previously assigned variant information from our backend. If the same experiment is triggered after this step, Kameleoon assigns the user the same variant. This synchronization ensures that a variant is consistently applied across devices without reassigning a new variant through bucketing.

Cross-device experimentation for feature experimentation

Feature experimentation occurs in environments where data is stored in memory for the duration of a session (which defaults to 30 minutes, thus making it an unreliable storage for expirable variation allocation). SDKs dynamically resolve variations during runtime, introducing unique complexities for cross-device consistency.

Role of SDKs

SDKs are essential for feature experimentation as they:

• Dynamically resolve variant allocations based on user identity (visitorCode, which is randomly generated by the SDK, or your own user ID) when targeting occurs.
• Use identity resolution to retrieve previously collected data across different devices, ensuring consistent variant assignment as the user switches devices.

Challenges and considerations

• No local storage: Unlike Kameleoon Web Experimentation, variants are not stored persistently; they are recalculated during runtime based on user identity.
• Anonymous sessions: Ensuring consistency when users begin as anonymous and later log in can be complex.
• Cross-device identity resolution: Accurately attributing experiment variants across devices depends on robust identity resolution.

Examples of use cases

The following use cases represent the most common scenarios our customers encounter when implementing cross-device experimentation. These examples highlight typical challenges and solutions for ensuring consistent user experiences and accurate variant attribution across multiple devices.

Use case 1 (same device)

In the first session, an anonymous user is targeted by an experiment and then logs in. In the second session, the visitor is logged in and is targeted by the same experiment.

• Web experimentation: The variant remains consistent with the variant allocated during the initial session because Kameleoon retrieves the previously assigned variant from local storage.
• Feature experimentation: If you have associated the anonymous visitor ID with the user ID using custom data, the same variant will be returned when getVariation() is called with either the anonymous ID or the user ID. However, a new variant will be allocated if getVariation() is called when the user ID is used in the method, and no mapping exists between the user ID and the anonymous ID.

Use case 2 (same device)

A user logs in and is targeted by an experiment during a session. In a subsequent session, the user is logged out, becomes anonymous, and is targeted by the same experiment.

• Web experimentation: The variant remains consistent with the variant allocated during the initial session because Kameleoon retrieves the previously assigned variant from local storage, ensuring continuity across sessions.
• Feature experimentation:
  • If, during session 1, the user moved from an anonymous state (anonymous1) to an identified state (userId) and the anonymous ID was mapped to the user ID, in session 2, Kameleoon will reuse anonymous1 to bucket the user with the same variant.
  • However, if no anonymous state was managed in session 1 (by calling the getVisitorCode() method), a new visitorCode (anonymous2) will be generated for session 2, resulting in a new variant allocation.

Use case 3 (different devices)

An anonymous user logs in on one device and is targeted by an experiment during the same session. In a later session on a different device, the user starts anonymously, gets targeted by the same experiment, and logs in.

• Web experimentation: In this scenario, the user may receive two different variants because Kameleoon does not recognize the user on the second device when making a targeting decision. The variant allocation on the second device occurs before the user logs in, resulting in a lack of synchronization between devices.
• Feature experimentation: Like web experimentation, the user may receive two different variants because Kameleoon does not recognize the user on the second device when making a targeting decision. In this case, the variant allocation on the second device occurs before the user logs in, meaning there is no synchronization between devices.

Use case 4 (different devices)

An anonymous user logs in on the first device and is targeted by an experiment. Later, on a second device, the user logs in before being targeted by the same experiment.

• Web experimentation: The variant remains consistent across devices. Since the user logs in before the experiment is reevaluated, the system aligns the variant with the variant allocated on the first device.
• Feature experimentation: Kameleoon determines the variant allocation based on the mapping of the user ID to the most recent anonymous ID generated for the visitor.
  • On Device 1, an anonymous visitor (anonymous1) is evaluated for the experiment and receives a variant (var1). Upon login, Kameleoon creates a mapping between userId and anonymous1.
  • On Device 2, if the session begins directly with the userId (with no call to getVisitorCode() generating an anonymous ID), calling getRemoteVisitorData(userId) allows Kameleoon to retrieve the mapping for the most recent anonymous visitor. (anonymous1). Consequently, when you call getRemoteVisitorData() and getVariation() for the experiment, the user will receive the variant (var1) assigned to anonymous1. However, if the session starts in an anonymous state, a new variant may be allocated since bucketing will use the newly generated bucketing key.

Use case 5 (different devices)

An anonymous user is targeted by an experiment on one device and logs in. On a second device, the user logs in and is targeted by the same experiment.

• Web experimentation: The variant remains consistent across devices. Once the user logs in, the system can attribute the same variant to the user.
• Feature experimentation: The variant remains consistent only if the customer calls getRemoteVisitorData(). Without this call, the system may not associate the two sessions, potentially leading to different variant attribution across devices.

Cross-device history reconciliation for targeting

Kameleoon automatically downloads a list of visits from other devices and stores this information in the current device's storage when a new session begins and custom data is set with a value. However, if the custom data is empty, reconciliation won't be performed at the start of the session. Reconciliation is triggered only when the custom data is passed, and targeting is re-evaluated for all active experiments and personalizations. Typically, this occurs when a user logs in to your site and provides their user ID, email, or other unique identifier.

It's important to note that once the custom data is provided, Kameleoon can retrieve it for all subsequent visits on the same device, even if the user has not yet logged in. Reconciliation occurs at the start of each subsequent visit, ensuring consistent tracking and targeting capabilities.

tip

To maintain a coherent unified data history for a visitor, it's recommended to provide the custom data value as early as possible during their journey on your website. Kameleoon cannot retrieve this history until the custom data is set up.

Data affected by cross-device history reconciliation

The history reconciliation feature adds the list of all previous visits that are not yet known to the current device. This includes typical information such as

• Number of visits and page views
• Session duration
• Acquisition channels and referrers
• Custom data (note: custom data provided in other sessions with a visitor scope will be merged into the current session).

You can use targeting segments as usual, and they will function as expected. For example, a targeting condition based on the number of previous visits will accurately consider all visits, as will a condition based on the duration of the previous visit

note

A maximum of 25 visits can be available for cross-device reconciliation, with a 3-month retention policy applying to the retrievable data.

Cross-device history reconciliation for analytics

History reconciliation impacts only the visitor view in Kameleoon reporting; the visit view remains unaffected. In the visitor model, instead of grouping multiple sessions into a single visitor record using the standard Kameleoon visitorCode value (which is only possible for returning visitors on the same device), we merge them based on the custom data value. If the custom data is empty or not provided, the visitorCode value is used instead. This approach ensures that unique visitor metrics remain accurate, even if the user is not yet identified in your customer system (though this applies only to returning visitors on the same device).

As a result, the visitor view correctly displays metrics across all devices for identified users (or known customers). For example, a single customer with three sessions on two different devices is counted as a single visitor. Without history reconciliation, the same customer would be counted as two distinct visitors (one per device).

note

In some cases, a user may experience multiple variants of an experiment if there is a switch of bucketing ID, either in different sessions or the same session. This is how our reporting/analytics will consider such cases:

• Visit view In the visit view, Kameleoon counts each session separately for the variant the user interacted with.
  • Example:
    • If a user is exposed to Variant A during one session and Variant B during another session, the counts would be:
      • Variant A: 1 session
      • Variant B: 1 session
• Visitor view In the visitor view, Kameleoon groups data by unique visitor codes (or mapping ID), counting each visitor only once per variant.
• Example:
  • If a user is exposed to Variant A in one session and Variant B in another, the counts would be:
    • Variant A: 1 visitor
    • Variant B: 1 visitor
• Single session with multiple variants When a user interacts with multiple variants in one session, Kameleoon only counts the last variant they engaged with.

Using custom data for session merging

Using custom data as a unique identifier for session merging is only possible for sessions saved after Kameleoon has identified which custom data to use. Therefore, we highly recommend selecting your custom data during the initial installation and configuration of Kameleoon to avoid changing it later.

For instance, if you allow Kameleoon to run for one month without enabling cross-device history reconciliation, and then activate it in the second month, a single user who visited twice (once in each month) will be counted as two different visitors. This discrepancy occurs because session merging in the first month uses the visitorCode as the key, whereas in the second month, it uses the custom data with a different value, even if the user logged in on both visits using the same device.

Any modifications to the custom data setup responsible for history reconciliation will temporarily affect visitor-based metrics accuracy as the majority of visitors log in, depending on how often they return to the site and authenticate themselves.