Retrieving experiment results using the automation API

In our tutorial focused on code based experiment , we explored the essential operations that can be carried out using our Automation API, which covered the process of creating a new experiment, modifying an existing variant, associating specific goals with a targeted segment, and finally, launching the experiment. Once launched, the experiment generates valuable results that provide critical insights for determining which variant performs best. In this follow-up tutorial, we will delve into how to use the Automation API to request results of a given experiment and use some of the result parameters to determine the winning variation of that experiment.

Accessing the automation api

To use the automation API , you must first get an access token. Find in this link the steps on how to retrieve it programmatically.

Requesting experiment results

For the purpose of this tutorial, we will work with the following experiment:

Experiment_188308

This experiment, named Product Page Redesign, includes two variations in addition to the original version: Redesign 1 (ID 828220) and Redesign 2 (ID 828221). While the experiment has several objectives, this tutorial will focus solely on the main goal, which is to track insurance subscriptions through Click Tracking.

To retrieve the results of this experiment, we must first send a POST request to the Request experiment's result endpoint.

In this example, the payload consists of the following components:

experimentId (188308): This is a mandatory field and must be included in the request’s header.
goalId: For this example, we are specifically focusing on the results of the main goal. Therefore, you only need to provide the ID for that particular goal in the request.
referenceVariationId (0): This ID is required as it will be used to compare the two variations against the original.
sequentialTesting: This parameter is set to "true" because the experiment will employ this specific statistical method to determine the confidence interval.
visitorData: It is set to “false” in this example to retrieve data based on each visit targeted by the experiment. If you need to retrieve data for each Visitor targeted by the experiment, you can set it to “true”.

The cURL request will be formatted as follows:

curl -L -X POST 'https://api.kameleoon.com/experiments/188308/results' \
-H 'Content-Type: application/json' \
-H 'Accept: */*' \
-H 'Authorization: Bearer eyJraWQiOiJLYW1lbGVvb24iLCJ0eXAiOiJKV1QiLCJhbGciOiJSUzI1NiJ9.eyJzdWIiOiIyOTgzMi1hYXRpcXVlK2F1dG9tYXRpb24ta2FtZWxlb29uLWNvbSIsImF1ZCI6IjI5ODMyLWFhdGlxdWUrYXV0b21hdGlvbi1rYW1lbGVvb24tY29tIiwiZG9tYWluIjoia2FtZWxlb29uIiwiaXNzIjoiaHR0cDpcL1wvYXBpLmthbWVsZW9vbi5jb21cL2FtXC9rYW1lbGVvb25cL29pZGMiLCJleHAiOjE3MzYyNzUyODcsImlhdCI6MTczNjI2ODA4NywianRpIjoiTHhsb1dMZDhXTGhlMEZLSVdXSnBSc0t3MXl2elloeUlsMzhLQzhJa29wOCIsInVzZXJuYW1lIjoiYWF0aXF1ZSthdXRvbWF0aW9uQGthbWVsZW9vbi5jb20ifQ.EqngEd1uA1W3_luq_Jb642Jjpu4UUgb3A-K06tGvFhYhkOYU51u88y9WyIcInTXb3ooT9nkkUasxpB_Qlr02DLA4TBwkNpKJOcoIsAbZfxNijxr3KZBRR-pUW86r2Z93dp8dB-yqunFeendJrsh3lM97nSQuBt0ukbRPQcHdaOK9HIUg3tPN0NObqXwag6O1jHJWAXaIz_y2nF8rNUJlQfIHlrzGocjIRXojpsPgd2DUe82mUqLHGAMts_c8Lq_3OYaLZsp9gcfpRQRWHyrBJRW37hP2leb4PwMh1h1yoA4Wv65xBvLVpzij0R_N6V8Dr7cjgZvida5c16M0SfLnAQ' \
--data-raw '{
  "visitorData": false,
  "sequentialTesting": true,
  "referenceVariationId": "0", 	 
  "goalsIds": [
	279599
  ]
}'

After making the cURL request, you will receive a data code, which will be used in another endpoint later to retrieve the detailed results. A successful response will be formatted as follows:

`{"dataCode":"14443931880924098266207585267983330260134079899081739889989435434342588016765"}`

After you have received the datacode, you will use this to send a GET request to the https://api.kameleoon.com/results?dataCodeendpoint.

The cURL request will be formatted as follows:

curl -L -X GET 'https://api.kameleoon.com/results?dataCode=14443931880924098266207585267983330260134079899081739889989435434342588016765'

A successful request will return a JSON object:

{
	"status": "READY",
	"data": {
		"dataCode": "14443931880924098266207585267983330260134079899081739889989435434342588016765",
		"variationData": {
			"_reference": {
				"breakdownData": {
					"_reference": {
						"intervalData": {},
						"generalData": {
							"visitCount": 126862,
							"visitorCount": 0,
							"goalsData": {
								"279599": {
									"conversionCount": 1898,
									"convertedVisitCount": 1898,
									"revenueCount": 235352.0,
									"convertedVisitorCount": 0,
									"revenuePerVisit": 1.86,
									"revenuePerVisitor": 0.0,
									"conversionRate": 0.014961138875313333,
									"averageCart": 124.0,
									"ratioValue": 0.0,
									"outlierBounds": {
										"lower": 0.1,
										"upper": 99.0
									}
								}
							}
						}
					}
				}
			},
			"828220": {
				"breakdownData": {
					"_reference": {
						"intervalData": {},
						"generalData": {
							"visitCount": 167938,
							"visitorCount": 0,
							"goalsData": {
								"279599": {
									"conversionCount": 7826,
									"convertedVisitCount": 7826,
									"revenueCount": 986076.0,
									"convertedVisitorCount": 0,
									"revenuePerVisit": 5.87,
									"revenuePerVisitor": 0.0,
									"reliability": 100.0,
									"improvementRange": {
										"min": 193.92,
										"max": 229.03,
										"half": 17.56
									},
									"improvementRate": 211.48,
									"conversionRate": 0.04660053114840001,
									"averageCart": 126.0,
									"ratioValue": 0.0,
									"continuousMetrics": {
										"conversions": {
											"reliability": 1.0,
											"improvementRate": 2.114771645178252,
											"halfInterval": 0.10445147824911348,
											"lowerBound": 2.0103201669291386,
											"upperBound": 2.219223123427365
										},
										"revenuePerVisit": {
											"reliability": 1.0,
											"improvementRate": 2.1650098975198366,
											"halfInterval": 0.10613617951119597,
											"lowerBound": 2.0588737180086407,
											"upperBound": 2.2711460770310326
										},
										"averageCart": null
									},
									"outlierBounds": {
										"lower": 0.1,
										"upper": 99.0
									}
								}
							}
						}
					}
				}
			},
			"828221": {
				"breakdownData": {
					"_reference": {
						"intervalData": {},
						"generalData": {
							"visitCount": 174194,
							"visitorCount": 0,
							"goalsData": {
								"279599": {
									"conversionCount": 1477,
									"convertedVisitCount": 1477,
									"revenueCount": 189056.0,
									"convertedVisitorCount": 0,
									"revenuePerVisit": 1.09,
									"revenuePerVisitor": 0.0,
									"reliability": 100.0,
									"improvementRange": {
										"min": -47.93,
										"max": -38.72,
										"half": 4.6
									},
									"improvementRate": -43.33,
									"conversionRate": 0.00847905209134643,
									"averageCart": 128.0,
									"ratioValue": 0.0,
									"continuousMetrics": {
										"conversions": {
											"reliability": 1.0,
											"improvementRate": -0.4332615877700786,
											"halfInterval": 0.027369693136479373,
											"lowerBound": -0.46063128090655797,
											"upperBound": -0.40589189463359926
										},
										"revenuePerVisit": {
											"reliability": 1.0,
											"improvementRate": -0.41497970350459723,
											"halfInterval": 0.028252586463462577,
											"lowerBound": -0.4432322899680598,
											"upperBound": -0.38672711704113466
										},
										"averageCart": null
									},
									"outlierBounds": {
										"lower": 0.1,
										"upper": 99.0
									}
								}
							}
						}
					}
				}
			}
		},
		"ventilationNames": null,
		"cupedDataByGoalId": {}
	}
}

Interpreting the results to determine the winning variation

To be considered a winner, a variation must demonstrate both a high reliability rate and a high improvement rate compared to the reference variant. By definition, a “reliable” variation is one that has a reliability rate exceeding 95%.

From the JSON object that was returned from the previous example’s cURL request (as shown in the code snippets below), we can see that both Redesign 1 and Redesign 2 have a reliability rate of 100%. However, Redesign 1 has an improvement rate of +211.48% while redesign 2 has an improvement rate of -43.33%, therefore, we can conclude that Redesign 1 is the winning variation of this experiment :

Redesign 1

"828220": {
				"breakdownData": {
					"_reference": {
						"intervalData": {},
						"generalData": {
							"visitCount": 167938,
							"visitorCount": 0,
							"goalsData": {
								"279599": {
									"conversionCount": 7826,
									"convertedVisitCount": 7826,
									"revenueCount": 986076.0,
									"convertedVisitorCount": 0,
									"revenuePerVisit": 5.87,
									"revenuePerVisitor": 0.0,
									"reliability": 100.0,
									"improvementRange": {
										"min": 193.92,
										"max": 229.03,
										"half": 17.56
									},
									"improvementRate": 211.48,

Redesign 2

"828221": {
				"breakdownData": {
					"_reference": {
						"intervalData": {},
						"generalData": {
							"visitCount": 174194,
							"visitorCount": 0,
							"goalsData": {
								"279599": {
									"conversionCount": 1477,
									"convertedVisitCount": 1477,
									"revenueCount": 189056.0,
									"convertedVisitorCount": 0,
									"revenuePerVisit": 1.09,
									"revenuePerVisitor": 0.0,
									"reliability": 100.0,
									"improvementRange": {
										"min": -47.93,
										"max": -38.72,
										"half": 4.6
									},
									"improvementRate": -43.33,

In this tutorial, we presented a simple example, but the first endpoint allows you to retrieve data in various ways, such as by device or browser. Users can utilize the breakdown parameter of the endpoint to access this data effectively.

Filter your experiment’s results

In addition to utilizing our automation API to determine successful experiment variations, you can use it to obtain more specific results by using the breakdown and filter parameters. Breakdown offers a more in-depth view of your results, which encompasses data organized by factors such as: new visitors, operating system, browser, day of the week, SDK, timezone, and more.

For example, if you wish to narrow the previous request to return only a breakdown of the visit results by browser only for new visitors, you need to follow the same procedures outlined earlier. However this time, your cURL request to the Export experiment results endpoint must include the Breakdown parameter to specify that you are seeking results categorized by BROWSER, and you should filter it to show visits for NEW_VISITOR only, as demonstrated below:

curl -L -X POST 'https://api.kameleoon.com/experiments/188308/results' \
-H 'Content-Type: application/json' \
-H 'Accept: */*' \
-H 'Authorization: Bearer eyJraWQiOiJLYW1lbGVvb24iLCJ0eXAiOiJKV1QiLCJhbGciOiJSUzI1NiJ9.eyJzdWIiOiIyOTgzMi1hYXRpcXVlK2F1dG9tYXRpb24ta2FtZWxlb29uLWNvbSIsImF1ZCI6IjI5ODMyLWFhdGlxdWUrYXV0b21hdGlvbi1rYW1lbGVvb24tY29tIiwiZG9tYWluIjoia2FtZWxlb29uIiwiaXNzIjoiaHR0cDpcL1wvYXBpLmthbWVsZW9vbi5jb21cL2FtXC9rYW1lbGVvb25cL29pZGMiLCJleHAiOjE3NDgwMzE3NjAsImlhdCI6MTc0ODAyNDU2MCwianRpIjoiVmM1YmZvNkpUcXZFX3pfZ0ZiTHJqRzZUZ1hob3REeGU4T2JGX2hPOEl5ZyIsInVzZXJuYW1lIjoiYWF0aXF1ZSthdXRvbWF0aW9uQGthbWVsZW9vbi5jb20ifQ.Ekig7vPXobOY4Vfj4JJ80_Omrgw1jMv3QZ-r2p0KB0Nmw8sFPjuj0UWM-sVvfXPmkQx-YNzdd1u3Zfiik0mk6TE3wFHXxBQRfSP7vJV00PT1BZPJ5IG62QJyNagBM0mwDe0dzPWrGKYleRpL7m-uZCC7jBDvJkiwVEQOfVkFUKze3uzrul9iRdsbpw2mKkYJlb5c1raaXL8G9siLLbDnA52RN3Tra_UXxmSdDU2Xb-UhGCTdkBLIhR2Z-zmqHmhi8ElE0A6VDfESokJsspcfqC674fVoZRr5gkYVA2LQE7TlM75apX7hoIbs_k1ansdM0rjkG4eRxEeQA0zDWaQ6CA
' \
--data-raw '{
  "visitorData": true,
  "sequentialTesting": true,
  "breakdown": {
    "type": "BROWSER"
  },
  "referenceVariationId": "0",
  "filters": [
    
    {
        "type":"NEW_VISITOR",
        "visitorsType":"NEW_VISITORS",
        "include":true
    }],
  "goalsIds": [
        279599
  ]
}'

Once you receive the data code, send the following GET request:

curl -L -X GET 'https://api.kameleoon.com/results?dataCode=14443931880924098266207585267983330260134079899081739889989435434342588016765'

A successful request will provide a response that includes experiment results associated with new visitors from the Chrome browser (certain parts of the response have been truncated for clarity):

>....                                                                                                                                                              
                        }
                    },
                    "OTHERS": {
                        "intervalData": {},
                        "generalData": {
                            "visitCount": 0,
                            "visitorCount": 587206,
                            "goalsData": {
                                "279599": {
                                    "conversionCount": 5410,
                                    "convertedVisitCount": 0,
                                    "revenueCount": 692480.0,
                                    "convertedVisitorCount": 5359,
                                    "revenuePerVisit": 0.0,
                                    "revenuePerVisitor": 1.18,
                                    "reliability": 100.0,
                                    "improvementRange": {
                                        "min": -53.0,
                                        "max": -48.7,
                                        "half": 2.15
                                    },
                                    "improvementRate": -50.85,
                                    "conversionRate": 0.009126269145751235,
                                    "averageCart": 129.22,
                                    "ratioValue": 0.0
                                }
                            }
                        }
                    },
                    "CHROME": {
                        "intervalData": {},
                        "generalData": {
                            "visitCount": 0,
                            "visitorCount": 16,
                            "goalsData": {
                                "279599": {
                                    "conversionCount": 0,
                                    "convertedVisitCount": 0,
                                    "revenueCount": 0.0,
                                    "convertedVisitorCount": 0,
                                    "revenuePerVisit": 0.0,
                                    "revenuePerVisitor": 0.0,
                                    "reliability": 50.0,
                                    "improvementRange": {
                                        "min": null,
                                        "max": null,
                                        "half": null
                                    },
                                    "improvementRate": 0.0,
                                    "averageCart": 0.0,
                                    "ratioValue": 0.0
                                }
                            }
                        }
                    }
                }
            }
        },
        "ventilationNames": null,
        "cupedDataByGoalId": {}
    }
}

Accessing the automation api​

Requesting experiment results​

Interpreting the results to determine the winning variation​

Filter your experiment’s results​

Accessing the automation api

Requesting experiment results

Interpreting the results to determine the winning variation

Filter your experiment’s results