OpenMetrics Wizard
The OpenMetrics Wizard is a new way to create DataSources via wizard using OpenMetrics (a Prometheus Exposition format). These OpenMetrics DataSources monitor OpenMetrics-formatted data, including metrics in the Prometheus exposition format. This feature was one of the first products that I designed for LogicMonitor’s APM suite. I was the sole designer on this project, where I was in charge of designing and researching the prototype.
June 2020 to December 2020
Dates
Worked with…
Tools
Sketch, InVision
Teammates
Chris Sternberg (Product Manager), Greg Nudelman (Design Advisor), Peter Stradinger (Team Lead, UI Engineering)
The Problem
OpenMetrics monitoring can get complicated - While OpenMetrics can be helpful to ensure that all systems can ingest and emit data, setting up monitoring for it can be complicated. Most monitoring solutions that already allows users to monitor OpenMetrics and/or Prometheus require users to go into a text editor in order to set up their monitoring (as seen by examples from NewRelic and DataDog). These methods are typically used by expert users and have a high learning curve.
70% of DataDog’s (LogicMonitor competitor) income comes from Custom Metrics - Right now, DataDog charges $5 per 100 custom metrics per month. This price doesn’t seem expensive, until you consider the fact that companies typically use around 14k to 30k custom metrics. So, if you’re a company using 30k custom metrics at most, you’d get charged around $1500 per month. That’s around the average rent in Austin! To beat out DataDog, LogicMonitor needed a way to monitor custom metrics that was easy and inexpensive for customers. LogicMonitor offers DataSources which come at no extra charge, but…
DataSources can be a pain to create - Currently, it takes around 30 minutes to 3 hours on average for an ITOps developer to add in a DataSource into LogicMonitor. If they want to add extra customizations to their DataSources, it can take them up to two weeks to create one. Typically, only senior/principal level ITOps developers create DataSources, due to how complex the current process is. As a result, it can cause a bottleneck within their company, if their ITOps teams want to create multiple DataSources and are understaffed…
To provide a streamlined creation of DataSources based off of an endpoint from OpenMetrics so that a non-expert user can create and customize a DataSource in the fraction of the time.
The Goal
Ideation
The first thing that I did was sit down and do ideation with Chris, the product manager I was working with. One of the first things that we had to figure out was where exactly the feature lived and what format it could take. I started out some initial ideas with where the OpenMetric DataSource creator could live. We initially thought it could live in the Settings page, as that was where users were most familiar with adding and configuring their DataSources.
We were considering, due to development time (as the UI refresh of the Settings page had not been completed yet), we implement it as something within Settings. I initially tried to go with a style that was familiar with our users, as most of our users who worked in DataSources used long forms within the settings page.
I initially tested this design with around 5 users, 4 out of 5 who were familiar with creating DataSources. I asked them to walk through the steps of the form and tell me what steps they needed to take in order to create a OpenMetrics DataSource. They were able to navigate it ok, as it was familiar to what they had worked with in the past. However, they were unfamiliar with Prometheus and OpenMetrics terms within the form.
Even though the straight form shown above had performed well in user testing, we ended up deciding to go with a Wizard style flow. The reason why we ended up going with a wizard style flow was due to users struggling to understand some of the terms within the form, even though a majority of the users had experience working with DataSources in the past. We ended up deciding to reuse some of the design patterns from LM Cloud Onboarding, as they had performed well with both beginner and expert users in previous usability studies. We wanted the OpenMetrics DataSource creator to be as intuitive as possible and predicted that by reusing the wizard, it would meet our goal of creating a streamlined and easy experience for all types of users to create OpenMetrics DataSources.
We also ended up deciding that that the creator should live in LM Exchange. The reason why we ended up deciding on the Exchange, was due to product strategy. Our product team was planning to migrate all of the editing and creation of DataSources from the Settings page to LM Exchange so that a user could have one central location when it came to managing DataSources. To prevent engineering from doing extra work, we decided on creating a wizard in LM Exchange.
After deciding where exactly the wizard should live, I worked with Chris to map out the flow in Balsamiq, in order to determine what steps we needed to take and what the user needed to do in each step. We made sure to make the steps virtually identical to making a DataSource, but with an additional step thanks to the OpenMetrics exposition. A user would need a way to load up the metrics from an existing resource, so that they could select those and then turn them into datapoints.
Once the flow was nailed down, it was time to turn them into a vision prototype. We ended up going for a vision at first instead of building incrementally in order ensure we got every feature nailed down.
Creating the Vision
The first stage of the prototyping process was to build a vision prototype. This vision prototype included all of the features that could go into the production once the prototype was complete. The reason why I ended up creating a vision first was because I could map out every single feature ahead of time and get it validated by customers before it got stripped into an MVP. By doing so, it would save me time in the future once we wanted to add back in features, as the designs would already be validated.
For the first pass of the Vision, I focused mostly on creating the most important flow - adding in the OpenMetrics DataSource. I also focused on the first step, which was filtering and finding the metrics that the user wanted to create datapoints out of.
First Pass of Usability Testing
This first round of usability testing was done in order to determine if the OpenMetrics Wizard Intuitive to users and what features needed to be added and improved upon. In this initial test, the design was tested with nine users, three of them having experience creating DataSources. In the usability test, users were asked to filter out and use the wizard to create an OpenMetrics DataSource. I also focused on one step in particular in regards to creating a DataSource, which was adding and editing datapoints. The main findings from this first round of usability testing were:
The overall flow made sense to all of the users - including the one who didn’t have a lot of experience creating DataSources. All users were also able to understand it, despite not having a large understanding of OpenMetrics/Prometheus.
Users wanted the ability to compare metric groups together - in the first version, if a user wanted to see what metrics were inside of a metric group, they would have to go back and forth between modals, which proved to be annoying.
Users need more context and help - considering that most of the users did not have experience using OpenMetrics, they wanted more in-line support documentation to help them
Improvements to Vision
After the first round of usability testing, I started making a second version of the vision prototype. The following features were added/improved upon:
The ability to compare metric groups and move metrics from one group to another
The ability to include and exclude metrics from a metric group
A bulk action where a user could select multiple metric groups and then “loop” the wizard to create multiple DataSources in one go
A table view that shows ungrouped metrics
Taking the load step out of the wizard, so it was a completely separate step - serving as a pseudo-home page for the OpenMetrics DataSources
This improved upon design was tested with eight different users, all who had experience working with DataSources. Like the users in the first round of usability testing, they were asked to load the metrics and to create a DataSource. The main findings from this round of usability testing were:
Moving metrics from one label to another made sense - Users were able to move metrics between metric groups with little to no trouble. However, users stated that they wouldn’t be moving metrics between groups that often and would usually would do this in their own text editors.
Separating the load metrics step out of the wizard made no sense - Users were confused why the list of DataSources that they had already created disappeared after they loaded the metrics. Users were also unsure of how they could check to see if they already created a DataSource after loading the metrics.
MVP
One of the last steps for the OpenMetrics wizard, as it was close to the start of development was to implement some changes to the vision - which included:
Putting the load metrics step back in wizard flow as the second step. This change was made after talking with engineering, and it turns out that it was easier to capture the metrics in the DataSource after the user had loaded them up after setting up the DataSource.
Moving the table of created DataSources to their own tab in the LM Exchange page for further differentiation (for now)
Adding an in-line help panel
Adding the ability to add instance filters to control what information populated as instances.
Then, I sat down with Peter and Chris to discuss what and what couldn’t be in the initial MVP. We ended up taking out the metric groups, the ability to compare, include, exclude, and move metrics from groups, the ability to “loop” the wizard, and some advanced filtering. We decided that we really wanted to focus on the primary function of the wizard, which was to add in DataSources.
Overall, during the beta, there were customer testimonies that helped us ensure that the design was meeting its goal. Below is an example of a customer story.
“One managed service provider customer had been trying to use the scripted DataSource templates to create custom DataSources for his customers, but couldn’t get them to work. They reached out to support and they were unable to help him. After being introduced to the OpenMetrics wizard, he was able to create a DataSource that collected data from a custom application that a customer had created. He was happy that he was able to finally get a DataSource running, as his customers were getting upset with him. The OpenMetrics wizard makes it much easier to meet demanding requests from customers.”
When LogicMonitor’s APM package was released, this product was the most well-received out of all of the products.
Post-Release Feedback
Post-Release Improvements
Even though, the overall feedback from this feature was positive, a majority of customers wanted the ability to load up metrics that weren’t coming from a preexisting resource. Though customers were fine setting up a resource where they could load metrics from, they wanted then ability to load up metrics that weren’t coming from a pre-existing resource, so that they could do everything in the wizard.
In order to address this change, we added in an additional step where users could add in their mandatory fields and then load then metrics using those. We also added secondary fields that were optional, as we had heard from customers that they wanted to get granular with their metrics using fields such as these.
Also, we got additional feedback from customers and the LMX team that the OpenMetrics DataSources felt disconnected from the rest of the platform. In order to address this, we decided to do away with the tab separating the two. The OpenMetrics DataSources would be treated like any other DataSource. However, a user could still filter down to see only OpenMetrics DataSources. This filtering functionality existed in the current LMX design in order to see other formats. We also included the ability to create a new OpenMetrics DataSource in the add modal.
Edit Flow
One of the next things that I needed to design after making improvements was the new edit flow. The edit flow was to fulfill the requirement of allowing the user to edit any of their OpenMetrics DataSources. In order to create a consistent experience, we ended up using then same pattern as the Cloud Resource editors for the OpenMetrics edit flow. Like the wizard, this pattern had already been validated in previous user tests, so I was confident that this would work. However, I ran it against 10 users who had experience building DataSources in a separate usability test to ensure that they would understand it.
My hypothesis ended up being correct, the usability test showed that all of the users were able to understand how to edit their OpenMetrics DataSources. They cited how it was similar to the Wizard and the Cloud Resource editor as primary reasons for the ease of use. Engineering was also able to get it done with no changes to the prototype due to my reuse of components. This feature is now live in the product.
Like most products at LogicMonitor, this product will go through incremental changes over time in order to meet evolving needs of the customer. Here are some initiatives we are working on.
Future Changes - The team and I are revisiting the ability to load up metrics via RegEx within the wizard. We’re working on figuring out when it should be implemented and whether design changes need to be made.
Reusing the design to make wizards for other LogicModules - A followup study was done by two members of my team who are focused on LM Exchange (Will and Craig) in order to determine whether users would want wizards to expedite the process. As predicted by my team and I, users of varying expertise wanted a wizard, as it would allow more junior members to create DataSources, thus allowing their organizations to spread work around easily. Right now Craig is working on proposed designs for the SNMP and WMI wizards, using this prototype as a base.