{"id":595,"date":"2023-08-10T08:00:00","date_gmt":"2023-08-10T08:00:00","guid":{"rendered":"http:\/\/www.diveintoaccessibility.com\/?p=595"},"modified":"2024-10-14T18:02:38","modified_gmt":"2024-10-14T18:02:38","slug":"running-a-page-speed-test-monitoring-vs-measuring","status":"publish","type":"post","link":"http:\/\/www.diveintoaccessibility.com\/index.php\/2023\/08\/10\/running-a-page-speed-test-monitoring-vs-measuring\/","title":{"rendered":"Running A Page Speed Test: Monitoring vs. Measuring"},"content":{"rendered":"<p>              <title>Running A Page Speed Test: Monitoring vs. Measuring<\/title><\/p>\n<article>\n<header>\n<h1>Running A Page Speed Test: Monitoring vs. Measuring<\/h1>\n<address>Geoff Graham<\/address>\n<p>                  2023-08-10T08:00:00+00:00<br \/>\n                  2024-10-14T17:35:03+00:00<br \/>\n                <\/header>\n<p>This article is sponsored by <b>DebugBear<\/b><\/p>\n<p>There is no shortage of ways to measure the <em>speed<\/em> of a webpage. The tooling to get a report with details from the time it takes to establish a server connection to the time it takes for the full page to render is out there. In fact, there\u2019s great tooling right under the hood of most browsers in DevTools that can do many things that a tried-and-true service like WebPageTest offers, complete with recommendations for improving specific metrics.<\/p>\n<figure class=\"\n  \n    break-out article__image\n  \n  \n  \"><\/p>\n<p>    <a href=\"https:\/\/files.smashing.media\/articles\/running-page-speed-test-monitoring-versus-measuring\/debug-devtools-lighthouse.png\"><\/p>\n<p>    <img loading=\"lazy\" width=\"800\" height=\"686\" src=\"https:\/\/res.cloudinary.com\/indysigner\/image\/fetch\/f_auto,q_80\/w_400\/https:\/\/files.smashing.media\/articles\/running-page-speed-test-monitoring-versus-measuring\/debug-devtools-lighthouse.png\" alt=\"Chrome\u2019s DevTools includes Google Lighthouse features for measuring Web Core Vitals.\" \/><\/p>\n<p>    <\/a><figcaption class=\"op-vertical-bottom\">\n      Lighthouse results. (<a href=\"files.smashing.media\/articles\/running-page-speed-test-monitoring-versus-measuring\/debug-devtools-lighthouse.png\">Large preview<\/a>)<br \/>\n    <\/figcaption><\/figure>\n<p>I don\u2019t know about you, but it often feels like I\u2019m missing something when measuring page speed performance. Even with all of the available tools at my disposal, I still find myself reaching for several of them. Certain tools are designed for certain metrics with certain assumptions that produce certain results. So, what I have is a hodgepodge of reports that needs to be collected, combined, and crunched before I have clear picture of what\u2019s going on.<\/p>\n<figure class=\"\n  \n    break-out article__image\n  \n  \n  \"><\/p>\n<p>    <a href=\"https:\/\/files.smashing.media\/articles\/running-page-speed-test-monitoring-versus-measuring\/debug-tools-collage.png\"><\/p>\n<p>    <img loading=\"lazy\" width=\"800\" height=\"491\" src=\"https:\/\/res.cloudinary.com\/indysigner\/image\/fetch\/f_auto,q_80\/w_400\/https:\/\/files.smashing.media\/articles\/running-page-speed-test-monitoring-versus-measuring\/debug-tools-collage.png\" alt=\"Collage of open windows with performance results.\" \/><\/p>\n<p>    <\/a><figcaption class=\"op-vertical-bottom\">\n      Not the best way to get a high-level view of performance. (<a href=\"https:\/\/files.smashing.media\/articles\/running-page-speed-test-monitoring-versus-measuring\/debug-tools-collage.png\">Large preview<\/a>)<br \/>\n    <\/figcaption><\/figure>\n<p>The folks at <a href=\"https:\/\/www.debugbear.com\/?utm_campaign=sm-1\">DebugBear<\/a> understand this situation all too well, and they were kind enough to give me an account to poke around their site speed and core web vitals reporting features. I\u2019ve had time to work with DebugBear and thought I\u2019d give you a peek at it with some notes on my experience using it to monitor performance. If you\u2019re like me, it\u2019s hard to invest in a tool &mdash; particularly a paid one &mdash; before seeing how it actually works and fits into my work.<\/p>\n<h2 id=\"monitoring-vs-measuring\">Monitoring vs. Measuring<\/h2>\n<p>Before we actually log in and look at reports, I think it\u2019s worth getting a little semantic. The key word here is \u201cmonitoring\u201d performance. After using DebugBear, I began realizing that what I\u2019ve been doing all along is \u201cmeasuring\u201d performance. And the difference between \u201cmonitoring\u201d and \u201cmeasuring\u201d is big.<\/p>\n<p>When I\u2019m <em>measuring<\/em> performance, I\u2019m only getting a snapshot at a particular time and place. There\u2019s no context about page speed performance before or after that snapshot because it stands alone. Think of it like a single datapoint on a line chart &mdash; there are no surrounding points to compare my results to which keeps me asking, <em>Is this a good result or a bad result?<\/em> That\u2019s the \u201cthing\u201d I\u2019ve been missing in my performance efforts.<\/p>\n<p>There are ways around that, of course. I could capture that data and feed it into a spreadsheet so that I have a record of performance results over time that can be used to spot where performance is improving and, conversely, where it is failing. That seems like a lot of work, even if it adds value. The other issue is that the data I\u2019m getting back is based on lab simulations where I can add throttling, determine the device that\u2019s used, and the network connection, among other simulated conditions.<\/p>\n<p>On that note, it\u2019s worth calling out that there are multiple flavors of network throttling. One is powered by Lighthouse, which observes data by testing on a fast connection and <strong>estimates the amount of time it takes to load on different connections<\/strong>. This is the type of network throttling you will find in PageSpeed Insights, and it is the default method in Lighthouse. <a href=\"https:\/\/www.debugbear.com\/blog\/simulated-throttling?utm_campaign=sm-1\">DebugBear explains this nicely<\/a> in its blog:<\/p>\n<blockquote><p>Simulated throttling provides low variability and makes test quick and cheap to run. However, it can also lead to inaccuracies as Lighthouse doesn&#8217;t fully replicate all browser features and network behaviors.<\/p><\/blockquote>\n<p>In contrast, tools like DebugBear and WebPageTest use more realistic throttling that accurately reflects network round trips on a higher-latency connection.<\/p>\n<p><em>Real<\/em> usage data would be better, of course. And we can get that with <strong>real-user monitoring<\/strong> (RUM) where a snippet of code on my site collects real data based on from real network conditions coming from real users is sent to a server and parsed for reporting.<\/p>\n<p>That\u2019s where a tool like DebugBear makes a lot of sense. It <em>measures<\/em> performance on an automated schedule (no more manual runs, but you can <a href=\"https:\/\/www.debugbear.com\/test\/website-speed?utm_campaign=sm-1\">still do that with their free tool<\/a>) and <em>monitors<\/em> the results by keeping an eye on the historical results (no more isolated data points). And in both cases, I know I\u2019m working with high-quality, realistic data.<\/p>\n<p>From there, DebugBear <em>notifies<\/em> me when it spots an outlier in the results so I am always in the know.<\/p>\n<h2 id=\"the-debugbear-dashboard\">The DebugBear Dashboard<\/h2>\n<p>This is probably what you want to see first, right? All I had to do to set up performance monitoring for a page is provide DebugBear with a URL and data flowed in immediately with subsequent automated tests running on a four-hour basis, which is configurable.<\/p>\n<p>Once that was in place, DebugBear produced a dashboard of results. And kept doing that over time.<\/p>\n<figure class=\"\n  \n    break-out article__image\n  \n  \n  \"><\/p>\n<p>    <a href=\"https:\/\/files.smashing.media\/articles\/running-page-speed-test-monitoring-versus-measuring\/debug-dashboard.png\"><\/p>\n<p>    <img loading=\"lazy\" width=\"800\" height=\"601\" src=\"https:\/\/res.cloudinary.com\/indysigner\/image\/fetch\/f_auto,q_80\/w_400\/https:\/\/files.smashing.media\/articles\/running-page-speed-test-monitoring-versus-measuring\/debug-dashboard.png\" alt=\"DebugBear dashboard of mini-charts.\" \/><\/p>\n<p>    <\/a><figcaption class=\"op-vertical-bottom\">\n      (<a href=\"https:\/\/files.smashing.media\/articles\/running-page-speed-test-monitoring-versus-measuring\/debug-dashboard.png\">Large preview<\/a>)<br \/>\n    <\/figcaption><\/figure>\n<p>You can probably look at that screenshot and see the immediate value of this high-level view of page performance. You get big score numbers, mini charts for a variety of web vital metrics, and a filmstrip of the page rendering with annotations identifying where those metrics sit in the process, among other great pieces of information.<\/p>\n<p>But I\u2019d like to call out a few especially nice affordances that have made my performance efforts easier and, more importantly, more insightful.<\/p>\n<h2 id=\"working-with-page-speed-data\">Working With Page Speed Data<\/h2>\n<p>I\u2019ve learned along the way that there are actually multiple kinds of data used to inform testing assumptions.<\/p>\n<p>One type is called <strong>lab data<\/strong>. It, in turn, has its own subset of data types. One is <em>observed data<\/em> where CPU and network throttling conditions are applied to the test environment before opening the page &mdash; \u201capplied throttling\u201d as it were. Another is <em>simulated data<\/em> which describes the Lighthouse method mentioned earlier where tests are done on a high-powered CPU with a highspeed network connection and then estimates how \u201cfast\u201d a page would load on lower-powered devices. Observed data is the high-quality type of lab data used by tools like DebugBear and WebPageTest. Simulated data, on the other hand, might be convenient and fast, but also can be innacurate.<\/p>\n<p>A second type of data is called <strong>real-user<\/strong> <strong>data<\/strong>. This is high-quality data from actual website visitors, for example <a href=\"https:\/\/www.debugbear.com\/blog\/chrome-user-experience-report?utm_campaign=sm-1\">based on<\/a> <strong>Google\u2019s Chrome User Experience (CrUX) Report<\/strong>. The report, released in 2017, provides network data from sessions collected from real Chrome users. This is high-quality data, for sure, but it comes with its own set of limitations. For example, the data is limited to Chrome users who are logged into their Google account, so it\u2019s not completely representative of all users. Plus, the data is aggregated over 28 days, which means it may not be not the freshest data.<\/p>\n<p>Alongside the CrUX report, we also have the <strong>RUM approach<\/strong> to data that we discussed earlier. It\u2019s another type of real-user monitoring takes real traffic from your site and sends the information over for extremely accurate results.<\/p>\n<p>So, having both a \u201creal user\u201d score and a \u201clab\u201d score in DebugBear is sort of like having my cake and eating it.<\/p>\n<figure class=\"\n  \n    break-out article__image\n  \n  \n  \"><\/p>\n<p>    <a href=\"https:\/\/files.smashing.media\/articles\/running-page-speed-test-monitoring-versus-measuring\/debug-real-lab-scores.png\"><\/p>\n<p>    <img loading=\"lazy\" width=\"800\" height=\"109\" src=\"https:\/\/res.cloudinary.com\/indysigner\/image\/fetch\/f_auto,q_80\/w_400\/https:\/\/files.smashing.media\/articles\/running-page-speed-test-monitoring-versus-measuring\/debug-real-lab-scores.png\" alt=\"Real user and lab data scores.\" \/><\/p>\n<p>    <\/a><figcaption class=\"op-vertical-bottom\">\n      (<a href=\"https:\/\/files.smashing.media\/articles\/running-page-speed-test-monitoring-versus-measuring\/debug-real-lab-scores.png\">Large preview<\/a>)<br \/>\n    <\/figcaption><\/figure>\n<p>This way, I can establish a \u201cbaseline\u201d set of conditions for DebugBear to use in my automated reports and view them alongside actual user data while keeping a historical record of the results.<\/p>\n<h2 id=\"comparing-tests\">Comparing Tests<\/h2>\n<p>Notice how I can dig into the data by opening up any test at a specific point in time and compare it to other tests at different points in time.<\/p>\n<figure class=\"\n  \n    break-out article__image\n  \n  \n  \"><\/p>\n<p>    <a href=\"https:\/\/files.smashing.media\/articles\/running-page-speed-test-monitoring-versus-measuring\/debug-compare-pages.png\"><\/p>\n<p>    <img loading=\"lazy\" width=\"800\" height=\"506\" src=\"https:\/\/res.cloudinary.com\/indysigner\/image\/fetch\/f_auto,q_80\/w_400\/https:\/\/files.smashing.media\/articles\/running-page-speed-test-monitoring-versus-measuring\/debug-compare-pages.png\" alt=\"Comparing test results of different pages.\" \/><\/p>\n<p>    <\/a><figcaption class=\"op-vertical-bottom\">\n      (<a href=\"https:\/\/files.smashing.media\/articles\/running-page-speed-test-monitoring-versus-measuring\/debug-compare-pages.png\">Large preview<\/a>)<br \/>\n    <\/figcaption><\/figure>\n<p>The fact that I can add any experiment on any page &mdash; and as many of them as I need &mdash; is just plain awesome. It\u2019s especially great for our team here at Smashing Magazine because different articles use different assets that affect performance, and the ability to compare the same article at different points in time or compare it to other pages is incredibly helpful to see exactly what is weighing down a specific page.<\/p>\n<p>DebugBear\u2019s comparison feature goes beyond mini charts by providing larger charts that evaluate more things than I can possibly print for you here.<\/p>\n<figure class=\"\n  \n    break-out article__image\n  \n  \n  \"><\/p>\n<p>    <a href=\"https:\/\/files.smashing.media\/articles\/running-page-speed-test-monitoring-versus-measuring\/debug-large-chart.png\"><\/p>\n<p>    <img loading=\"lazy\" width=\"800\" height=\"528\" src=\"https:\/\/res.cloudinary.com\/indysigner\/image\/fetch\/f_auto,q_80\/w_400\/https:\/\/files.smashing.media\/articles\/running-page-speed-test-monitoring-versus-measuring\/debug-large-chart.png\" alt=\"Detailed line chart with a dropdown of metric options.\" \/><\/p>\n<p>    <\/a><figcaption class=\"op-vertical-bottom\">\n      (<a href=\"https:\/\/files.smashing.media\/articles\/running-page-speed-test-monitoring-versus-measuring\/debug-large-chart.png\">Large preview<\/a>)<br \/>\n    <\/figcaption><\/figure>\n<h2 id=\"running-page-test-experiments\">Running Page Test Experiments<\/h2>\n<p>Sometimes I have an idea to optimize page speed but find I need to deploy the changes to production first so that a reporting tool can re-evaluate the page for me to compare the results. It would be a lot cooler to know whether those changes are effective <em>before<\/em> hitting production.<\/p>\n<p>That\u2019s what you can do with DebugBear\u2019s Experiments feature &mdash; tweak the code of the page being measured and run a test you can compare to other live results.<\/p>\n<figure class=\"\n  \n    break-out article__image\n  \n  \n  \"><\/p>\n<p>    <a href=\"https:\/\/files.smashing.media\/articles\/running-page-speed-test-monitoring-versus-measuring\/debug-test-changes.png\"><\/p>\n<p>    <img loading=\"lazy\" width=\"800\" height=\"667\" src=\"https:\/\/res.cloudinary.com\/indysigner\/image\/fetch\/f_auto,q_80\/w_400\/https:\/\/files.smashing.media\/articles\/running-page-speed-test-monitoring-versus-measuring\/debug-test-changes.png\" alt=\"\u201cPrettify Code\u201d option\" \/><\/p>\n<p>    <\/a><figcaption class=\"op-vertical-bottom\">\n      See the \u201cPrettify Code\u201d option? \ud83d\ude0d(<a href=\"https:\/\/files.smashing.media\/articles\/running-page-speed-test-monitoring-versus-measuring\/debug-test-changes.png\">Large preview<\/a>)<br \/>\n    <\/figcaption><\/figure>\n<p>This is the kind of thing I would definitely expect from a paid service. It really differentiates DebugBear from something like a standard Lighthouse report, giving me more control as well as tools to help me gain deeper insights into my work.<\/p>\n<h2 id=\"everything-in-one-place\">Everything In One Place<\/h2>\n<p>Having all of my reports in a central one-stop shop is worth the price of admission alone. I can\u2019t stand the clutter of having multiple windows open to get the information I need. With DebugBear, I have everything that a mish-mash of DevTools, WebPageTest, and other tools provides, but in one interface that is as clean as it gets. There\u2019s no hunting around trying to remember which window has my <abbr title=\"Time to First Byte\">TTFB<\/abbr> score for one experiment or which has the filmstrip of another experiment I need.<\/p>\n<p>But what you might not expect is a set of actionable recommendations to improve page speed performance right within reach.<\/p>\n<figure class=\"\n  \n    break-out article__image\n  \n  \n  \"><\/p>\n<p>    <a href=\"https:\/\/files.smashing.media\/articles\/running-page-speed-test-monitoring-versus-measuring\/debug-recs.png\"><\/p>\n<p>    <img loading=\"lazy\" width=\"800\" height=\"699\" src=\"https:\/\/res.cloudinary.com\/indysigner\/image\/fetch\/f_auto,q_80\/w_400\/https:\/\/files.smashing.media\/articles\/running-page-speed-test-monitoring-versus-measuring\/debug-recs.png\" alt=\"Filmstrip and video of page rendering.\" \/><\/p>\n<p>    <\/a><figcaption class=\"op-vertical-bottom\">\n      (<a href=\"https:\/\/files.smashing.media\/articles\/running-page-speed-test-monitoring-versus-measuring\/debug-recs.png\">Large preview<\/a>)<br \/>\n    <\/figcaption><\/figure>\n<p>Let me be clear that I am no performance expert. There are plenty of situations where I don\u2019t know what I don\u2019t know, and performance is one of them. Performance can easily be a profession and full-time job by itself, just as design, accessibility, and other specializations. So, having a list of things I can do to improve performance is incredibly helpful for me. It\u2019s like having a performance consultant in the room giving me directions.<\/p>\n<h2 id=\"wrapping-up\">Wrapping Up<\/h2>\n<p>Again, this is merely a peek at some of the things that DebugBear can do and what I enjoy about it. The fact is that it does <em>so many things<\/em> that I\u2019ve either glossed over or simply lack the space to show you.<\/p>\n<p>The best thing you can do is <a href=\"https:\/\/www.debugbear.com\/signup?utm_campaign=sm-1\">create a free DebugBear account<\/a> and play around with it yourself. Seriously, there\u2019s no credit card required. You set up a username and password, then it\u2019s off to the races.<\/p>\n<p>And when (not if!) you get your account, I\u2019d love to know what stands out to you. Performance means a lot of things to different people and we all have our ways of approaching it. I\u2019m keen to know how you would use a suite of features like this in your own work.<\/p>\n<div class=\"signature\">\n  <img src=\"https:\/\/www.smashingmagazine.com\/images\/logo\/logo--red.png\" alt=\"Smashing Editorial\" width=\"35\" height=\"46\" loading=\"lazy\" \/><br \/>\n  <span>(gg, il)<\/span>\n<\/div>\n<\/article>\n","protected":false},"excerpt":{"rendered":"<p>Running A Page Speed Test: Monitoring vs. Measuring Running A Page Speed Test: Monitoring vs. Measuring Geoff Graham 2023-08-10T08:00:00+00:00 2024-10-14T17:35:03+00:00 This article is sponsored by [&hellip;]<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":[],"categories":[11],"tags":[],"_links":{"self":[{"href":"http:\/\/www.diveintoaccessibility.com\/index.php\/wp-json\/wp\/v2\/posts\/595"}],"collection":[{"href":"http:\/\/www.diveintoaccessibility.com\/index.php\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"http:\/\/www.diveintoaccessibility.com\/index.php\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"http:\/\/www.diveintoaccessibility.com\/index.php\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"http:\/\/www.diveintoaccessibility.com\/index.php\/wp-json\/wp\/v2\/comments?post=595"}],"version-history":[{"count":1,"href":"http:\/\/www.diveintoaccessibility.com\/index.php\/wp-json\/wp\/v2\/posts\/595\/revisions"}],"predecessor-version":[{"id":596,"href":"http:\/\/www.diveintoaccessibility.com\/index.php\/wp-json\/wp\/v2\/posts\/595\/revisions\/596"}],"wp:attachment":[{"href":"http:\/\/www.diveintoaccessibility.com\/index.php\/wp-json\/wp\/v2\/media?parent=595"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"http:\/\/www.diveintoaccessibility.com\/index.php\/wp-json\/wp\/v2\/categories?post=595"},{"taxonomy":"post_tag","embeddable":true,"href":"http:\/\/www.diveintoaccessibility.com\/index.php\/wp-json\/wp\/v2\/tags?post=595"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}