JMeter, from Apache, is a load and performance testing tool.
$0
Maze
Score 8.0 out of 10
N/A
Maze is a rapid user testing platform from Maze.design in Paris, designed to give users actionable user insights, in a matter of hours. The vendor states that with it, users can test remotely, autonomously, and collaboratively.
$75
per month
Optimizely Web Experimentation
Score 8.7 out of 10
N/A
Whether launching a first test or scaling a sophisticated experimentation program, Optimizely Web Experimentation aims to deliver the insights needed to craft high-performing digital experiences that drive engagement, increase conversions, and accelerate growth.
JMeter is well suited for Java applications where the user can script the scenario once and make changes to accommodate for as many numbers of users for load test execution. The image and selection of any files or exporting files scenario is handled well.
It is less appropriate to test Ajax applications where it is required to script click per use.
Maze User Testing is great if you're interested in doing user research from the comfort of your own desk. You can easily setup usability tests, surveys, card sorting and tree tests among other things to get a better understanding of how customers use your product. The only limitation at the moment with Maze that I can identify is only being able to do unmoderated tests, so if you'd like to be able to ask follow up questions in the moment, Maze is not the tool for you.
I think it can serve the whole spectrum of experiences from people who are just getting used to web experimentation. It's really easy to pick up and use. If you're more experienced then it works well because it just gets out of the way and lets you really focus on the experimentation side of things. So yeah, strongly recommend. I think it is well suited both to small businesses and large enterprises as well. I think it's got a really low barrier to entry. It's very easy to integrate on your website and get results quickly. Likewise, if you are a big business, it's incrementally adoptable, so you can start out with one component of optimizing and you can build there and start to build in things like data CMS to augment experimentation as well. So it's got a really strong a pathway to grow your MarTech platform if you're a small company or a big company.
Easy of use - in generate load like HTTP requests, and processing/analyzing the responses. No coding is necessary at the basic level, just need to understand load testing and the infrastructure being tested.
Automatic management of things like cookies to help with session state support - so you don't specifically have to worry about it or handle it
Lots of testing/configuration options to suit your needs in making the right load generation (sampling requests), and analyzing the results, including any pre and post processing of the results first. Things like the Beanshell/BSF pre/post processors, response assertion, regular expression extractor, XPath extractor, CSV data set config
There is a JMeter cloud service called BlazeMeter that I think would be useful for those that need to scale up high load without provisioning their own systems. I've not personally tried it though, but I recently attended a meetup presentation that highlighted nice useful features that BlazeMeter provides. One should evaluate the service if they are considering JMeter and need to expand beyond existing hardware resources.
The Platform contains drag-and-drop editor options for creating variations, which ease the A/B tests process, as it does not require any coding or development resources.
Establishing it is so simple that even a non-technical person can do it perfectly.
It provides real-time results and analytics with robust dashboard access through which you can quickly analyze how different variations perform. With this, your team can easily make data-driven decisions Fastly.
Jmeter requires many tweaks with respect to its configuration file and thread properties. users need to edit theses files themselves. There could be some interface where we can edit this fields.
Jmeter cannot handle more threads and hangs up when we increase the number of threads. This causes lot of inconvenience. In these situations, user can be notified that such change would be lead to slow performance so that user can do as required. The same appears when we try to view huge files on graph listener.
Jmeter should optimize the read and write access to output csv since it acts as overhead to the I/O performance. This affects our test results for the application which we are testing.
Price, Wiki and user sharing. Having access to the information provided by the developers and other open source providers is key for me. The ability to share information and get answers directly is very important to success in software testing. And the price of this product currently is amazing. Too many companies charge way too much money for products that are far behind in their value and pertinence
I rated this question because at this stage, Optimizely does most everything we need so I don't foresee a need to migrate to a new tool. We have the infrastructure already in place and it is a sizeable lift to pivot to another tool with no guarantee that it will work as good or even better than Optimizely
The purpose related to performance and load testing through Apache JMeter works fine but the usability of the tool should be improved quite a lot. If someone starts with the Jmeter fresh without prior experience, they need to put more efforts in understanding the tool. The UI is not that great which is the main reason not to give high rating on usability.
Maze is easy to use most of the times. It is easy to integrate with Figma, It is easy to find testers worldwide with required filters. Maze gives recorded videos which are helpful in debugging and understanding the problem with flows. A/B testing is easy to add and test. Overall Maze is very easy to use
Optimizely Web Experimentation's visual editor is handy for non-technical or quick iterative testing. When it comes to content changes it's as easy as going into wordpress, clicking around, and then seeing your changes live--what you see is what you get. The preview and approval process for sharing built experiments is also handy for sharing experiments across teams for QA purposes or otherwise.
I would rate Optimizely Web Experimentation's availability as a 10 out of 10. The software is reliable and does not experience any application errors or unplanned outages. Additionally, the customer service and technical support teams are always available to help with any issues or questions.
I would rate Optimizely Web Experimentation's performance as a 9 out of 10. Pages load quickly, reports are complete in a reasonable time frame, and the software does not slow down any other software or systems that it integrates with. Additionally, the customer service and technical support teams are always available to help with any issues or questions.
I have been using JMeter for the last year. By using this tool, you can make sure the system will work under varied loads. It helps us to simulate real time scenarios by creating required virtual users and make sure the application will work under load. Perform load, stress, and stability testing using JMeter.
They always are quick to respond, and are so friendly and helpful. They always answer the phone right away. And [they are] always willing to not only help you with your problem, but if you need ideas they have suggestions as well.
The tool itself is not very difficult to use so training was not very useful in my opinion. It did not also account for success events more complex than a click (which my company being ecommerce is looking to examine more than a mere click).
In retrospect: - I think I should have stressed more demo's / workshopping with the Optimizely team at the start. I felt too confident during demo stages, and when came time to actually start, I was a bit lost. (The answer is likely I should have had them on-hand for our first install.. they offered but I thought I was OK.) - Really getting an understanding / asking them prior to install of how to make it really work for checkout pages / one that uses dynamic content or user interaction to determine what the UI does. Could have saved some time by addressing this at the beginning, as some things we needed to create on our site for Optimizely to "use" as a trigger for the variation test. - Having a number of planned/hoped-for tests already in-hand before working with Optimizely team. Sharing those thoughts with them would likely have started conversations on additional things we needed to do to make them work (rather than figuring that out during the actual builds). Since I had development time available, I could have added more things to the baseline installation since my developers were already "looking under the hood" of the site.
I have used LoadRunner and Silkperformer, and so far Jmeter turns out be the easiest to use of all these. While each of them have their own ROI, Jmeter can be picked by anyone in hours and start testing within a day. While with other tools, we need to get license, install them (takes a while) and setup tests and firewalls, etc.
A Lookback is an alternative option if you think Maze User Testing is quite expensive for you, but look back has a different approach to Maze User Testing. Lookback focuses on qualitative usability testing instead of quantitative UserTesting. And also, Maze User Testing has a free option but Lookback doesn't have it, but Lookback has a cheaper option at $19/month than Maze.
The ability to do A/B testing in Optimizely along with the associated statistical modelling and audience segmentation means it is a much better solution than using something like Google Analytics were a lot more effort is required to identify and isolate the specific data you need to confidently make changes
We can use it flexibly across lines of business and have it in use across two departments. We have different use cases and slightly different outcomes, but can unify our results based on impact to the bottom line. Finally, we can generate value from anywhere in the org for any stakeholders as needed.
We're able to share definitive annualized revenue projections with our team, showing what would happen if we put a test into Production
Showing the results of a test on a new page or feature prior to full implementation on a site saves developer time (if a test proves the new element doesn't deliver a significant improvement.
Making a change via the WYSIWYG interface allows us to see multiple changes without developer intervention.
