Retrieving experiment results using the Automation API
The tutorial focused on code-based experiments explores the essential operations that can be carried out using the Automation API, including creating a new experiment, modifying an existing variation, associating specific goals with a targeted segment, and launching the experiment. Once launched, the experiment generates valuable results that provide critical insights for determining which variation performs best. This follow-up tutorial will delve into how to use the Automation API to request results of a given experiment and use the result parameters to determine that experiment's winning variation.
Requirements
access_token
To use the Automation API, you first need an access token.
Follow the instructions in this section to retrieve it programmatically.
Goal
This tutorial will use the following example experiment:
This experiment, called Product Page Redesign, includes two variations in addition to the original version: Redesign 1 (ID 828220) and Redesign 2 (ID 828221). While the experiment has several objectives, this tutorial will focus solely on the main goal, which is to track insurance subscriptions through Click Tracking.
This guide will teach you how to retrieve and interpret experiment results, and apply filters.
The tutorial can also be applied to Feature Flags Experiment; however, you must use the https://api.kameleoon.com/feature-flags/* endpoints instead of https://api.kameleoon.com/experiments/*:
The experimentId can be found in the Rollout Planner :
1. Retrieve the data code
Case 1: You want to see the results yourself
Endpoint:
To retrieve this experiment's results, you must first send a POST request to the Request experiment's result endpoint.
| Name | Type | Description |
|---|---|---|
experimentId | String | This is a mandatory field and must be included in the request’s header. |
goalId | Array | This example specifically focuses on the main goal's result. Therefore, you must only provide the ID for that particular goal in the request. |
referenceVariationId | Object | This ID is required as it will be used to compare the two variations against the original. |
sequentialTesting | String | This parameter is set to "true" because the experiment will employ this specific statistical method to determine the confidence interval. |
visitorData | String | It is set to “false” in this example to retrieve data based on each visit targeted by the experiment. If you need to retrieve data for each Visitor targeted by the experiment, you can set it to “true”. |
Example:
curl -L -X POST 'https://api.kameleoon.com/experiments/188308/results' \
-H 'Content-Type: application/json' \
-H 'Accept: */*' \
-H 'Authorization: Bearer eyJraWQiOiJLYW1lbGVvb24iLCJ0eXAiOiJKV1QiLCJhbGciOiJSUzI1NiJ9.eyJzdWIiOiIyOTgzMi1hYXRpcXVlK2F1dG9tYXRpb24ta2FtZWxlb29uLWNvbSIsImF1ZCI6IjI5ODMyLWFhdGlxdWUrYXV0b21hdGlvbi1rYW1lbGVvb24tY29tIiwiZG9tYWluIjoia2FtZWxlb29uIiwiaXNzIjoiaHR0cDpcL1wvYXBpLmthbWVsZW9vbi5jb21cL2FtXC9rYW1lbGVvb25cL29pZGMiLCJleHAiOjE3MzYyNzUyODcsImlhdCI6MTczNjI2ODA4NywianRpIjoiTHhsb1dMZDhXTGhlMEZLSVdXSnBSc0t3MXl2elloeUlsMzhLQzhJa29wOCIsInVzZXJuYW1lIjoiYWF0aXF1ZSthdXRvbWF0aW9uQGthbWVsZW9vbi5jb20ifQ.EqngEd1uA1W3_luq_Jb642Jjpu4UUgb3A-K06tGvFhYhkOYU51u88y9WyIcInTXb3ooT9nkkUasxpB_Qlr02DLA4TBwkNpKJOcoIsAbZfxNijxr3KZBRR-pUW86r2Z93dp8dB-yqunFeendJrsh3lM97nSQuBt0ukbRPQcHdaOK9HIUg3tPN0NObqXwag6O1jHJWAXaIz_y2nF8rNUJlQfIHlrzGocjIRXojpsPgd2DUe82mUqLHGAMts_c8Lq_3OYaLZsp9gcfpRQRWHyrBJRW37hP2leb4PwMh1h1yoA4Wv65xBvLVpzij0R_N6V8Dr7cjgZvida5c16M0SfLnAQ' \
--data-raw '{
"visitorData": false,
"sequentialTesting": true,
"referenceVariationId": "0",
"goalsIds": [
279599
]
}'
Case 2: You want to share the results with a user without Authorization
1. Retrieve a SharedToken
Endpoint:
To share the results of an experiment with someone without Kameleoon's API credential, a SharedToken must be fetched at the Share experiment results endpoint.
curl -L -X POST 'https://api.kameleoon.com/experiments/188308/results/share' \
-H 'Content-Type: application/json' \
-H 'Accept: */*' \
-H 'Authorization: Bearer eyJraWQiOiJLYW1lbGVvb24iLCJ0eXAiOiJKV1QiLCJhbGciOiJSUzI1NiJ9.eyJzdWIiOiIyOTgzMi1hYXRpcXVlK2F1dG9tYXRpb24ta2FtZWxlb29uLWNvbSIsImF1ZCI6IjI5ODMyLWFhdGlxdWUrYXV0b21hdGlvbi1rYW1lbGVvb24tY29tIiwiZG9tYWluIjoia2FtZWxlb29uIiwiaXNzIjoiaHR0cDpcL1wvYXBpLmthbWVsZW9vbi5jb21cL2FtXC9rYW1lbGVvb25cL29pZGMiLCJleHAiOjE3MzYyNzUyODcsImlhdCI6MTczNjI2ODA4NywianRpIjoiTHhsb1dMZDhXTGhlMEZLSVdXSnBSc0t3MXl2elloeUlsMzhLQzhJa29wOCIsInVzZXJuYW1lIjoiYWF0aXF1ZSthdXRvbWF0aW9uQGthbWVsZW9vbi5jb20ifQ.EqngEd1uA1W3_luq_Jb642Jjpu4UUgb3A-K06tGvFhYhkOYU51u88y9WyIcInTXb3ooT9nkkUasxpB_Qlr02DLA4TBwkNpKJOcoIsAbZfxNijxr3KZBRR-pUW86r2Z93dp8dB-yqunFeendJrsh3lM97nSQuBt0ukbRPQcHdaOK9HIUg3tPN0NObqXwag6O1jHJWAXaIz_y2nF8rNUJlQfIHlrzGocjIRXojpsPgd2DUe82mUqLHGAMts_c8Lq_3OYaLZsp9gcfpRQRWHyrBJRW37hP2leb4PwMh1h1yoA4Wv65xBvLVpzij0R_N6V8Dr7cjgZvida5c16M0SfLnAQ' \
--data-raw '{
"visitorData": false,
"sequentialTesting": true,
"referenceVariationId": "0",
"goalsIds": [
279599
]
}'
Response:
{
"method": "POST",
"url": "https://api.kameleoon.com/experiments/188308/results",
"path": "/experiments/188308/results",
"headers": {
"SharedToken": "eyJhbGciOiJIUzUxMiJ9.eyJleHAiOjE4NDQ5NDI5MTMsInNoYXJlZENvZGUiOiJhZmM1NzRjN2I4MjY4NTY3NzI2YjZhMDRhMTUxMjgyNjA3Zjk1ZGI4YjA1Y2RkZGY1ZDEwM2ExZDg5ZWM0MmZmIn0.oCECNm5mEhgIthc6eejo9BrfB7p8kEIrpoqtNb4JUiHK6nsxWHMvLc4hHYXCg3DgaBlVoKv6eEZHGty9c-VAoA",
"Content-Type": "application/json"
},
"payload": {
"visitorData": false,
"sequentialTesting": true,
"referenceVariationId": "0",
"goalsIds": [
279599
]
}
}
1. Retrieve the dataCode with the SharedToken
To retrieve the results of this experiment, you must send a POST request to the Request experiment's result endpoint using the SharedToken instead of the usual access token.
| Name | Type | Description |
|---|---|---|
experimentId | String | This is a mandatory field and must be included in the request’s header. |
goalId | Array | This example specifically focuses on the main goal's result. Therefore, you must only provide the ID for that particular goal in the request. |
referenceVariationId | Object | This ID is required as it will be used to compare the two variations against the original. |
sequentialTesting | String | This parameter is set to "true" because the experiment will employ this specific statistical method to determine the confidence interval. |
visitorData | String | It is set to “false” in this example to retrieve data based on each visit targeted by the experiment. If you need to retrieve data for each Visitor targeted by the experiment, you can set it to “true”. |
Example:
curl -L -X POST 'https://api.kameleoon.com/experiments/188308/results' \
-H 'Content-Type: application/json' \
-H 'Accept: */*' \
-H 'SharedToken: eyJhbGciOiJIUzUxMiJ9.eyJleHAiOjE4NDQ5NDI5MTMsInNoYXJlZENvZGUiOiJhZmM1NzRjN2I4MjY4NTY3NzI2YjZhMDRhMTUxMjgyNjA3Zjk1ZGI4YjA1Y2RkZGY1ZDEwM2ExZDg5ZWM0MmZmIn0.oCECNm5mEhgIthc6eejo9BrfB7p8kEIrpoqtNb4JUiHK6nsxWHMvLc4hHYXCg3DgaBlVoKv6eEZHGty9c-VAoA' \
--data-raw '{
"visitorData": false,
"sequentialTesting": true,
"referenceVariationId": "0",
"goalsIds": [
279599
]
}'
Response:
{"dataCode":"14443931880924098266207585267983330260134079899081739889989435434342588016765"}
2. Retrieve the experiment results
Endpoint:
After you have received the datacode, use the result endpoint.
Example:
curl -L -X GET 'https://api.kameleoon.com/results?dataCode=14443931880924098266207585267983330260134079899081739889989435434342588016765'
Response:
{
"status": "READY",
"data": {
"dataCode": "14443931880924098266207585267983330260134079899081739889989435434342588016765",
"variationData": {
"_reference": {
"breakdownData": {
"_reference": {
"intervalData": {},
"generalData": {
"visitCount": 126862,
"visitorCount": 0,
"goalsData": {
"279599": {
"conversionCount": 1898,
"convertedVisitCount": 1898,
"revenueCount": 235352.0,
"convertedVisitorCount": 0,
"revenuePerVisit": 1.86,
"revenuePerVisitor": 0.0,
"conversionRate": 0.014961138875313333,
"averageCart": 124.0,
"ratioValue": 0.0,
"outlierBounds": {
"lower": 0.1,
"upper": 99.0
}
}
}
}
}
}
},
"828220": {
"breakdownData": {
"_reference": {
"intervalData": {},
"generalData": {
"visitCount": 167938,
"visitorCount": 0,
"goalsData": {
"279599": {
"conversionCount": 7826,
"convertedVisitCount": 7826,
"revenueCount": 986076.0,
"convertedVisitorCount": 0,
"revenuePerVisit": 5.87,
"revenuePerVisitor": 0.0,
"reliability": 100.0,
"improvementRange": {
"min": 193.92,
"max": 229.03,
"half": 17.56
},
"improvementRate": 211.48,
"conversionRate": 0.04660053114840001,
"averageCart": 126.0,
"ratioValue": 0.0,
"continuousMetrics": {
"conversions": {
"reliability": 1.0,
"improvementRate": 2.114771645178252,
"halfInterval": 0.10445147824911348,
"lowerBound": 2.0103201669291386,
"upperBound": 2.219223123427365
},
"revenuePerVisit": {
"reliability": 1.0,
"improvementRate": 2.1650098975198366,
"halfInterval": 0.10613617951119597,
"lowerBound": 2.0588737180086407,
"upperBound": 2.2711460770310326
},
"averageCart": null
},
"outlierBounds": {
"lower": 0.1,
"upper": 99.0
}
}
}
}
}
}
},
"828221": {
"breakdownData": {
"_reference": {
"intervalData": {},
"generalData": {
"visitCount": 174194,
"visitorCount": 0,
"goalsData": {
"279599": {
"conversionCount": 1477,
"convertedVisitCount": 1477,
"revenueCount": 189056.0,
"convertedVisitorCount": 0,
"revenuePerVisit": 1.09,
"revenuePerVisitor": 0.0,
"reliability": 100.0,
"improvementRange": {
"min": -47.93,
"max": -38.72,
"half": 4.6
},
"improvementRate": -43.33,
"conversionRate": 0.00847905209134643,
"averageCart": 128.0,
"ratioValue": 0.0,
"continuousMetrics": {
"conversions": {
"reliability": 1.0,
"improvementRate": -0.4332615877700786,
"halfInterval": 0.027369693136479373,
"lowerBound": -0.46063128090655797,
"upperBound": -0.40589189463359926
},
"revenuePerVisit": {
"reliability": 1.0,
"improvementRate": -0.41497970350459723,
"halfInterval": 0.028252586463462577,
"lowerBound": -0.4432322899680598,
"upperBound": -0.38672711704113466
},
"averageCart": null
},
"outlierBounds": {
"lower": 0.1,
"upper": 99.0
}
}
}
}
}
}
}
},
"ventilationNames": null,
"cupedDataByGoalId": {}
}
}
3. Interpreting the results to determine the winning variation
To be considered a winner, a variation must demonstrate both a high reliability rate and improvement rate compared to the reference variation. By definition, a “reliable” variation is one that has a reliability rate exceeding 95%.
From the JSON object that was returned from the previous example’s cURL request (as shown in the code snippets below), we can see that both Redesign 1 and Redesign 2 have a reliability rate of 100%. However, Redesign 1 has an improvement rate of +211.48% while Redesign 2 has an improvement rate of -43.33%, therefore, Redesign 1 is the winning variation of this experiment:
Redesign 1
"828220": {
"breakdownData": {
"_reference": {
"intervalData": {},
"generalData": {
"visitCount": 167938,
"visitorCount": 0,
"goalsData": {
"279599": {
"conversionCount": 7826,
"convertedVisitCount": 7826,
"revenueCount": 986076.0,
"convertedVisitorCount": 0,
"revenuePerVisit": 5.87,
"revenuePerVisitor": 0.0,
"reliability": 100.0,
"improvementRange": {
"min": 193.92,
"max": 229.03,
"half": 17.56
},
"improvementRate": 211.48,
Redesign 2
"828221": {
"breakdownData": {
"_reference": {
"intervalData": {},
"generalData": {
"visitCount": 174194,
"visitorCount": 0,
"goalsData": {
"279599": {
"conversionCount": 1477,
"convertedVisitCount": 1477,
"revenueCount": 189056.0,
"convertedVisitorCount": 0,
"revenuePerVisit": 1.09,
"revenuePerVisitor": 0.0,
"reliability": 100.0,
"improvementRange": {
"min": -47.93,
"max": -38.72,
"half": 4.6
},
"improvementRate": -43.33,
4. Filter your experiment’s results
In addition to utilizing Kameleoon's Automation API to determine successful experiment variations, you can use it to obtain more specific results by using the breakdown and filter parameters. Breakdown offers a more in-depth view of your results, which encompasses data organized by factors such as new visitors, operating system, browser, day of the week, SDK, timezone, and more.
For example, if you wish to narrow the previous request to return only a breakdown of the visit results by browser only for new visitors, you must follow the same procedures outlined earlier. However this time, your cURL request to the Export experiment results endpoint must include the Breakdown parameter to specify that you are seeking results categorized by BROWSER, and you should filter it to show visits for NEW_VISITOR only, as demonstrated below:
curl -L -X POST 'https://api.kameleoon.com/experiments/188308/results' \
-H 'Content-Type: application/json' \
-H 'Accept: */*' \
-H 'Authorization: Bearer eyJraWQiOiJLYW1lbGVvb24iLCJ0eXAiOiJKV1QiLCJhbGciOiJSUzI1NiJ9.eyJzdWIiOiIyOTgzMi1hYXRpcXVlK2F1dG9tYXRpb24ta2FtZWxlb29uLWNvbSIsImF1ZCI6IjI5ODMyLWFhdGlxdWUrYXV0b21hdGlvbi1rYW1lbGVvb24tY29tIiwiZG9tYWluIjoia2FtZWxlb29uIiwiaXNzIjoiaHR0cDpcL1wvYXBpLmthbWVsZW9vbi5jb21cL2FtXC9rYW1lbGVvb25cL29pZGMiLCJleHAiOjE3NDgwMzE3NjAsImlhdCI6MTc0ODAyNDU2MCwianRpIjoiVmM1YmZvNkpUcXZFX3pfZ0ZiTHJqRzZUZ1hob3REeGU4T2JGX2hPOEl5ZyIsInVzZXJuYW1lIjoiYWF0aXF1ZSthdXRvbWF0aW9uQGthbWVsZW9vbi5jb20ifQ.Ekig7vPXobOY4Vfj4JJ80_Omrgw1jMv3QZ-r2p0KB0Nmw8sFPjuj0UWM-sVvfXPmkQx-YNzdd1u3Zfiik0mk6TE3wFHXxBQRfSP7vJV00PT1BZPJ5IG62QJyNagBM0mwDe0dzPWrGKYleRpL7m-uZCC7jBDvJkiwVEQOfVkFUKze3uzrul9iRdsbpw2mKkYJlb5c1raaXL8G9siLLbDnA52RN3Tra_UXxmSdDU2Xb-UhGCTdkBLIhR2Z-zmqHmhi8ElE0A6VDfESokJsspcfqC674fVoZRr5gkYVA2LQE7TlM75apX7hoIbs_k1ansdM0rjkG4eRxEeQA0zDWaQ6CA
' \
--data-raw '{
"visitorData": true,
"sequentialTesting": true,
"breakdown": {
"type": "BROWSER"
},
"referenceVariationId": "0",
"filters": [
{
"type":"NEW_VISITOR",
"visitorsType":"NEW_VISITORS",
"include":true
}],
"goalsIds": [
279599
]
}'
Once you receive the data code, send the following GET request:
curl -L -X GET 'https://api.kameleoon.com/results?dataCode=14443931880924098266207585267983330260134079899081739889989435434342588016765'
A successful request will provide a response that includes experiment results associated with new visitors from the Chrome browser (certain parts of the response have been truncated for clarity):
>....
}
},
"OTHERS": {
"intervalData": {},
"generalData": {
"visitCount": 0,
"visitorCount": 587206,
"goalsData": {
"279599": {
"conversionCount": 5410,
"convertedVisitCount": 0,
"revenueCount": 692480.0,
"convertedVisitorCount": 5359,
"revenuePerVisit": 0.0,
"revenuePerVisitor": 1.18,
"reliability": 100.0,
"improvementRange": {
"min": -53.0,
"max": -48.7,
"half": 2.15
},
"improvementRate": -50.85,
"conversionRate": 0.009126269145751235,
"averageCart": 129.22,
"ratioValue": 0.0
}
}
}
},
"CHROME": {
"intervalData": {},
"generalData": {
"visitCount": 0,
"visitorCount": 16,
"goalsData": {
"279599": {
"conversionCount": 0,
"convertedVisitCount": 0,
"revenueCount": 0.0,
"convertedVisitorCount": 0,
"revenuePerVisit": 0.0,
"revenuePerVisitor": 0.0,
"reliability": 50.0,
"improvementRange": {
"min": null,
"max": null,
"half": null
},
"improvementRate": 0.0,
"averageCart": 0.0,
"ratioValue": 0.0
}
}
}
}
}
}
},
"ventilationNames": null,
"cupedDataByGoalId": {}
}
}