Retrieving experiment results using the automation API

In our tutorial focused on code based experiment , we explored the essential operations that can be carried out using our Automation API, which covered the process of creating a new experiment, modifying an existing variant, associating specific goals with a targeted segment, and finally, launching the experiment. Once launched, the experiment generates valuable results that provide critical insights for determining which variant performs best. In this follow-up tutorial, we will delve into how to use the Automation API to request results of a given experiment and use some of the result parameters to determine the winning variation of that experiment.

Accessing the automation api

To use the automation API , you must first get an access token. Find in this link the steps on how to retrieve it programmatically.

Requesting experiment results

For the purpose of this tutorial, we will work with the following experiment:

This experiment, named Product Page Redesign, includes two variations in addition to the original version: Redesign 1 (ID 828220) and Redesign 2 (ID 828221). While the experiment has several objectives, this tutorial will focus solely on the main goal, which is to track insurance subscriptions through Click Tracking.

To retrieve the results of this experiment, we must first send a POST request to the Request experiment's result endpoint.

In this example, the payload consists of the following components:

• experimentId (188308): This is a mandatory field and must be included in the request’s header.
• goalId: For this example, we are specifically focusing on the results of the main goal. Therefore, you only need to provide the ID for that particular goal in the request.
• referenceVariationId (0): This ID is required as it will be used to compare the two variations against the original.
• sequentialTesting: This parameter is set to "true" because the experiment will employ this specific statistical method to determine the confidence interval.
• visitorData: It is set to “false” in this example to retrieve data based on each visit targeted by the experiment. If you need to retrieve data for each Visitor targeted by the experiment, you can set it to “true”.

The cURL request will be formatted as follows:

curl -L -X POST 'https://api.kameleoon.com/experiments/188308/results' \
-H 'Content-Type: application/json' \
-H 'Accept: */*' \
-H 'Authorization: Bearer eyJraWQiOiJLYW1lbGVvb24iLCJ0eXAiOiJKV1QiLCJhbGciOiJSUzI1NiJ9.eyJzdWIiOiIyOTgzMi1hYXRpcXVlK2F1dG9tYXRpb24ta2FtZWxlb29uLWNvbSIsImF1ZCI6IjI5ODMyLWFhdGlxdWUrYXV0b21hdGlvbi1rYW1lbGVvb24tY29tIiwiZG9tYWluIjoia2FtZWxlb29uIiwiaXNzIjoiaHR0cDpcL1wvYXBpLmthbWVsZW9vbi5jb21cL2FtXC9rYW1lbGVvb25cL29pZGMiLCJleHAiOjE3MzYyNzUyODcsImlhdCI6MTczNjI2ODA4NywianRpIjoiTHhsb1dMZDhXTGhlMEZLSVdXSnBSc0t3MXl2elloeUlsMzhLQzhJa29wOCIsInVzZXJuYW1lIjoiYWF0aXF1ZSthdXRvbWF0aW9uQGthbWVsZW9vbi5jb20ifQ.EqngEd1uA1W3_luq_Jb642Jjpu4UUgb3A-K06tGvFhYhkOYU51u88y9WyIcInTXb3ooT9nkkUasxpB_Qlr02DLA4TBwkNpKJOcoIsAbZfxNijxr3KZBRR-pUW86r2Z93dp8dB-yqunFeendJrsh3lM97nSQuBt0ukbRPQcHdaOK9HIUg3tPN0NObqXwag6O1jHJWAXaIz_y2nF8rNUJlQfIHlrzGocjIRXojpsPgd2DUe82mUqLHGAMts_c8Lq_3OYaLZsp9gcfpRQRWHyrBJRW37hP2leb4PwMh1h1yoA4Wv65xBvLVpzij0R_N6V8Dr7cjgZvida5c16M0SfLnAQ' \
--data-raw '{
  "visitorData": false,
  "sequentialTesting": true,
  "referenceVariationId": "0", 	 
  "goalsIds": [
	279599
  ]
}'

After making the cURL request, you will receive a data code, which will be used in another endpoint later to retrieve the detailed results. A successful response will be formatted as follows:

`{"dataCode":"14443931880924098266207585267983330260134079899081739889989435434342588016765"}`

After you have received the datacode, you will use this to send a GET request to the https://api.kameleoon.com/results?dataCodeendpoint.

The cURL request will be formatted as follows:

curl -L -X GET 'https://api.kameleoon.com/results?dataCode=14443931880924098266207585267983330260134079899081739889989435434342588016765'

A successful request will return a JSON object:

{
	"status": "READY",
	"data": {
		"dataCode": "14443931880924098266207585267983330260134079899081739889989435434342588016765",
		"variationData": {
			"_reference": {
				"breakdownData": {
					"_reference": {
						"intervalData": {},
						"generalData": {
							"visitCount": 126862,
							"visitorCount": 0,
							"goalsData": {
								"279599": {
									"conversionCount": 1898,
									"convertedVisitCount": 1898,
									"revenueCount": 235352.0,
									"convertedVisitorCount": 0,
									"revenuePerVisit": 1.86,
									"revenuePerVisitor": 0.0,
									"conversionRate": 0.014961138875313333,
									"averageCart": 124.0,
									"ratioValue": 0.0,
									"outlierBounds": {
										"lower": 0.1,
										"upper": 99.0
									}
								}
							}
						}
					}
				}
			},
			"828220": {
				"breakdownData": {
					"_reference": {
						"intervalData": {},
						"generalData": {
							"visitCount": 167938,
							"visitorCount": 0,
							"goalsData": {
								"279599": {
									"conversionCount": 7826,
									"convertedVisitCount": 7826,
									"revenueCount": 986076.0,
									"convertedVisitorCount": 0,
									"revenuePerVisit": 5.87,
									"revenuePerVisitor": 0.0,
									"reliability": 100.0,
									"improvementRange": {
										"min": 193.92,
										"max": 229.03,
										"half": 17.56
									},
									"improvementRate": 211.48,
									"conversionRate": 0.04660053114840001,
									"averageCart": 126.0,
									"ratioValue": 0.0,
									"continuousMetrics": {
										"conversions": {
											"reliability": 1.0,
											"improvementRate": 2.114771645178252,
											"halfInterval": 0.10445147824911348,
											"lowerBound": 2.0103201669291386,
											"upperBound": 2.219223123427365
										},
										"revenuePerVisit": {
											"reliability": 1.0,
											"improvementRate": 2.1650098975198366,
											"halfInterval": 0.10613617951119597,
											"lowerBound": 2.0588737180086407,
											"upperBound": 2.2711460770310326
										},
										"averageCart": null
									},
									"outlierBounds": {
										"lower": 0.1,
										"upper": 99.0
									}
								}
							}
						}
					}
				}
			},
			"828221": {
				"breakdownData": {
					"_reference": {
						"intervalData": {},
						"generalData": {
							"visitCount": 174194,
							"visitorCount": 0,
							"goalsData": {
								"279599": {
									"conversionCount": 1477,
									"convertedVisitCount": 1477,
									"revenueCount": 189056.0,
									"convertedVisitorCount": 0,
									"revenuePerVisit": 1.09,
									"revenuePerVisitor": 0.0,
									"reliability": 100.0,
									"improvementRange": {
										"min": -47.93,
										"max": -38.72,
										"half": 4.6
									},
									"improvementRate": -43.33,
									"conversionRate": 0.00847905209134643,
									"averageCart": 128.0,
									"ratioValue": 0.0,
									"continuousMetrics": {
										"conversions": {
											"reliability": 1.0,
											"improvementRate": -0.4332615877700786,
											"halfInterval": 0.027369693136479373,
											"lowerBound": -0.46063128090655797,
											"upperBound": -0.40589189463359926
										},
										"revenuePerVisit": {
											"reliability": 1.0,
											"improvementRate": -0.41497970350459723,
											"halfInterval": 0.028252586463462577,
											"lowerBound": -0.4432322899680598,
											"upperBound": -0.38672711704113466
										},
										"averageCart": null
									},
									"outlierBounds": {
										"lower": 0.1,
										"upper": 99.0
									}
								}
							}
						}
					}
				}
			}
		},
		"ventilationNames": null,
		"cupedDataByGoalId": {}
	}
}

Interpreting the results to determine the winning variation

To be considered a winner, a variation must demonstrate both a high reliability rate and a high improvement rate compared to the reference variant. By definition, a “reliable” variation is one that has a reliability rate exceeding 95%.

From the JSON object that was returned from the previous example’s cURL request (as shown in the code snippets below), we can see that both Redesign 1 and Redesign 2 have a reliability rate of 100%. However, Redesign 1 has an improvement rate of +211.48% while redesign 2 has an improvement rate of -43.33%, therefore, we can conclude that Redesign 1 is the winning variation of this experiment :

Redesign 1

"828220": {
				"breakdownData": {
					"_reference": {
						"intervalData": {},
						"generalData": {
							"visitCount": 167938,
							"visitorCount": 0,
							"goalsData": {
								"279599": {
									"conversionCount": 7826,
									"convertedVisitCount": 7826,
									"revenueCount": 986076.0,
									"convertedVisitorCount": 0,
									"revenuePerVisit": 5.87,
									"revenuePerVisitor": 0.0,
									"reliability": 100.0,
									"improvementRange": {
										"min": 193.92,
										"max": 229.03,
										"half": 17.56
									},
									"improvementRate": 211.48,

Redesign 2

"828221": {
				"breakdownData": {
					"_reference": {
						"intervalData": {},
						"generalData": {
							"visitCount": 174194,
							"visitorCount": 0,
							"goalsData": {
								"279599": {
									"conversionCount": 1477,
									"convertedVisitCount": 1477,
									"revenueCount": 189056.0,
									"convertedVisitorCount": 0,
									"revenuePerVisit": 1.09,
									"revenuePerVisitor": 0.0,
									"reliability": 100.0,
									"improvementRange": {
										"min": -47.93,
										"max": -38.72,
										"half": 4.6
									},
									"improvementRate": -43.33,

In this tutorial, we presented a simple example, but the first endpoint allows you to retrieve data in various ways, such as by device or browser. Users can utilize the breakdown parameter of the endpoint to access this data effectively.