Open-source IoT Platform

Device management, data collection, processing and visualization
for your IoT projects
Learn about Thingsboard

Thursday, February 9, 2017

Load testing of MQTT servers using Java, Maven and Gatling

The rapid growth of IoT market dramatically increased the popularity of MQTT protocol. MQTT is supported by the most popular IoT platforms and is used for data collection, push notifications, real-time messaging and other.

We would like to share our experience with running performance tests against MQTT servers in this article. We are going to concentrate on the tools that we used to generate load on the server. The results of the performance tests for our platform will be published as a separate post.

Gatling

We have chosen Gatling as a framework to run our test due to a number of reasons:

  • Generates beautiful reports out-of-the-box.
  • High performance and low overhead.
  • Easy setup of load scenarios.
  • Scalability of simulations.

Unfortunately Gatling.io framework doesn’t support MQTT protocol out-of-the-box. At the same time Gatling is an open-source framework and we have found an unofficial MQTT plugin.

Gatling MQTT Plugin

Gatling-MQTT plugin was developed by Muneyuki Noguchi and at the moment it is hosted on GitHub under Apache 2.0 License. We have started implementation of Thingsboard performance testing project using Gatling and Gatling-MQTT plugin. Some time later we have realized that the plugin doesn’t support scenarios that we would like to verify and the behaviour of default scenario is not something that we have expected.

The problem with the default scenario of the unofficial Gatling-MQTT plugin is that each time when some data is published, client waits for reply from server and sends MQTT disconnect. So, the message sequence looks like this:

image

This approach is very resource consuming and gives minimal benefits comparing to HTTP or other protocols. This behaviour is normal for HTTP requests-response model, but not for MQTT. The typical MQTT session is maintained for certain time and multiple MQTT publish messages are sent and received between client and MQTT broker. Of course, there are other types of MQTT messages, but they were out of scope for our tests. In our scenario, load testing of the IOT platform must be done in the following way:

image

So once we have connected a device to the IOT platform that acts as an MQTT broker, we will reuse session and publish MQTT messages using the same session. Sure, the session could be recreated at some point, but not every time we would like to publish a message to the server.

In order to support this scenario, we have decided not to implement something new from the scratch, but rather to use Gatling-MQTT plugin as a base and considering fact that this is open-source we are free to modify the software as we wish to meet our needs.

Gatling MQTT Plugin Fork

We have done the fork of Gatling-MQTT plugin, spent some time on investigation how the plugin was constructed, modified it and added Connect, Publish and Disconnect actions as separate steps.

Now we were able to support the expected scenario. The extended version of Gatling-MQTT plugin is located here Extended Gatling-MQTT.

Finally, we were able to implement the scenario that suits our needs. The Gatling simulation below will create separate MQTT session for 10 simulated devices and will send 100 publish messages for each session.

MqttSimulation.scala

class MqttSimulation extends Simulation {
    val mqttConfiguration = mqtt
        // MQTT broker
        .host("tcp://localhost:1883")
    
    val connect = exec(mqtt("connect")
        .connect())
    
    // send 100 publish MQTT messages
    val publish = repeat(100) {
        exec(mqtt("publish")
            // topic: "foo"
            // payload: "Hello"
            // QoS: AT_LEAST_ONCE (1)
            // retain: false
            .publish("foo", "Hello", QoS.AT_LEAST_ONCE, retain = false))
            // 1 seconds pause between sending messages
            .pause(1000 milliseconds)
        }
    
    val disconnect = exec(mqtt("disconnect")
      .disconnect())
    
    val scn = scenario("MQTT Test")
      .exec(connect, publish, disconnect)
    
    setUp(scn
        // linearly connect 10 devices over 1 second 
        // and send 100 publish messages
        .inject(rampUsers(10) over (1 seconds))
    ).protocols(mqttConfiguration)
}

Performance tests project

Our performance test project is hosted on GitHub. It is mainly written in Java and uses Maven as a build tool. Gatling and Gatling-MQTT plugin are written in Scala, and use SBT tool for building sources and running tests. But here, at Thingsboard, we are more Java guys than Scala, that’s why we have implemented custom Java code that connects to the platform, creates a device, warms it up and provides the credentials id string back:

MqttSimulation.scala

// get the device credential ids of the created devices
val deviceCredentialsIds: Array[String] = MqttStressTestTool.createDevices(testParams).asScala.toArray

MqttStressTestTool.java

RestClient restClient = new RestClient(params.getRestApiUrl());
// login to the Thingsboard server
restClient.login(params.getUsername(), params.getPassword());
for (int i = 0; i < params.getDeviceCount(); i++) {
    // create device using REST API
    Device device = restClient.createDevice("Device " + UUID.randomUUID());
    // get credentials from the created device
    DeviceCredentials credentials = restClient.getCredentials(device.getId());
    // store in the array that eventually will be used by Simulation class    
    deviceCredentialsIds.add(credentials.getCredentialsId());
    String[] mqttUrls = params.getMqttUrls();
    String mqttURL = mqttUrls[i % mqttUrls.length];
    MqttStressTestClient client = new MqttStressTestClient(results, mqttURL, credentials.getCredentialsId());
    // connect to the server and do the warm up 
    client.connect().waitForCompletion();
    client.warmUp(data);
    client.disconnect();
}
Thread.sleep(1000);

The list of credential ids is used in MqttSimulation.scala file to do the stress test itself:

// get the device credential ids of the created devices
val deviceCredentialsIds: Array[String] = MqttStressTestTool.createDevices(testParams).asScala.toArray

// provide device credential id as username during the connection phase to the Thingsboard server
val mqttConf = mqtt
    .host(testParams.getMqttUrls.head)
    .userName("${deviceCredentialsId}")

val connect = exec(mqtt("connect").connect())

val publish = repeat(testParams.getPublishTelemetryCount.toInt) {
    exec(mqtt("publish") 
        // publish single message and verify that it was delivered at least once
        .publish("v1/devices/me/telemetry", "{\"temp\":73.2}", QoS.AT_LEAST_ONCE, retain = false))
        .pause(testParams.getPublishTelemetryPause milliseconds)
}

val disconnect = exec(mqtt("disconnect").disconnect())
// create map of device credential ids of the connected devices  
// and use it as a feeder in the scenario
val deviceCredentialsIdsFeeder = deviceCredentialsIds.map( x => {Map("deviceCredentialsId" -> x)})

val scn = scenario("Scenario Name")
    // go over the map and take column deviceCredentialsId as username
    .feed(deviceCredentialsIdsFeeder)
    .exec(connect, publish, disconnect)

setUp(scn
    .inject(rampUsers(deviceCredentialsIds.length) over (1 seconds))
).protocols(mqttConf)

For the guys who prefer Java and Maven there is a maven plugin - gatling-maven-plugin:

<plugin>
    <groupId>io.gatling</groupId>
    <artifactId>gatling-maven-plugin</artifactId>
</plugin>

This plugin finds Simulation files in your project, compiles and runs them. Results will be stored inside of the target folder in a pretty format that you are able to inspect after the run:

image

image

image

To run the tests you simply can type:

mvn clean install gatling:execute

Summary

In general, we have described our approach how we have generated high load on our IoT platform and verified that it provides good results.

We will share the results of Thingsboard IoT platform performance tests in the next post.

Also, we’ll describe code changes, improvements and instances tuning that we have done to achieve processing of more than 1 million MQTT messages per minute. These are basically the first steps for us in the direction of performance testing of the platform, and any feedback regarding this approach is more than welcome.

We hope that the experience provided will help you to verify your solutions against load specifications that you expect to see in the production. Stay tuned by subscribing to our twitter, blog or by starring our project on Github.

Your feedback

If you found this article interesting, please leave your feedback in the comments section, post questions or feature requests on the forum and “star” our project on the github in order to stay tuned for new releases and tutorials.

73 comments :

  1. Excellent post, it has been of great help to me. But perhaps you can help me out with one thing. With the adjusted version created by yourself I have been struggling with using custom feeders that change during every message publication. For example I need to simulate messages that contain the actual date and time of the creation of the message. Any ideas how I can cope with this? I have only been using gatling for one week now so I am still new to it.

    Any help is appreciated

    ReplyDelete
    Replies
    1. IEEE Final Year projects Project Centers in Chennai are consistently sought after. Final Year Students Projects take a shot at them to improve their aptitudes, while specialists like the enjoyment in interfering with innovation. For experts, it's an alternate ball game through and through. Smaller than expected IEEE Final Year project centers ground for all fragments of CSE & IT engineers hoping to assemble. Final Year Projects for CSE It gives you tips and rules that is progressively critical to consider while choosing any final year project point.

      Spring Framework has already made serious inroads as an integrated technology stack for building user-facing applications. Spring Framework Corporate TRaining the authors explore the idea of using Java in Big Data platforms.

      Specifically, Spring Framework provides various tasks are geared around preparing data for further analysis and visualization. Spring Training in Chennai

      The Angular Training covers a wide range of topics including Components, Angular Directives, Angular Services, Pipes, security fundamentals, Routing, and Angular programmability. The new Angular TRaining will lay the foundation you need to specialise in Single Page Application developer. Angular Training

      Delete
  2. hi Unknown,

    If I got you correctly you are looking for something like this:

    def getMessageBody: io.gatling.core.session.Expression[scala.Predef.String] = {
    val currentTime = System.currentTimeMillis() / 1000
    val result = "{\"tc\":" + currentTime + ", \"temp\":73.2}"
    result
    }

    val publish = repeat(testParams.getPublishTelemetryCount.toInt) {
    exec(mqtt("publish")
    .publish("v1/devices/me/telemetry", getMessageBody, QoS.AT_LEAST_ONCE, retain = false))
    .pause(testParams.getPublishTelemetryPause milliseconds)
    }

    ReplyDelete
  3. That was a great message in my carrier, and It's wonderful commands like mind relaxes with understand words of knowledge by information's Load Test Website

    ReplyDelete
  4. Well Said, you have furnished the right information that will be useful to anyone at all time. Thanks for sharing your Ideas.

    software testing course in chennai

    ReplyDelete
  5. If your life has stopped giving you pleasure due to some unknown reasons than our Escorts Service is the right place it is still not too late to make it exciting. It is just your, who are such buddies, who can make you worth living if you spend a few moments with them. There is no such nudging in availing yourself of their companionship.


    Call Girls Near IGI Airport Delhi
    Escorts Service in Aerocity
    Russian Escorts in Ahmedabad
    Body Massage in Dharuhera
    Body Massage in Bhiwadi
    Massage Parlor in Dharuhera

    ReplyDelete
  6. The blog was having very informative content and very useful for me. Well done post and keep it up... Thanks for sharing such a Useful info.tableau automation

    ReplyDelete
  7. Hiya very cool web site!! Man ...Superb .. I’ll bookmark your site and take the feeds additionally?
    McAfee is quick and easy to install.
    McAfee programming gives the digital security services to ensure your hardware information.Install mcafee from mcafee.com/activate |
    How to Uninstall or disable McAfee WebAdvisor ?
    How to redeem Redeem McAfee LiveSafe ?
    How to know when my McAfee Subscription Expire ?
    Why do you need McAfee Security ?

    ReplyDelete
  8. I look forward to fresh updates and will share this website with my Facebook group!
    Find your HP printer support
    Easy way for How to connect hp printer to wifi
    Why is my HP printer offline

    ReplyDelete
  9. It's really a nice and useful piece of info.
    Setup your ms office from office.com/setup |
    What is the use of Windows 10 media creation tool
    How to get microsoft Office 365 for free ?
    Do you know you can Install office without account .
    How to Create Windows 10 recovery USB

    ReplyDelete
  10. I am really impressed with the information you have provided in your post. Looking forward to visiting more.
    mcafee.com/activate | Geek Squad Appointment

    ReplyDelete
  11. Several factors matter when you choose the Microsoft 365 subscription. Depending on your requirements on the respective productivity platforms, these products are different in price. microsoft365.com/setup

    ReplyDelete
  12. To activate Trend micro, make sure you already have an activation code that you’ll probably enter on the trendmicro activate site. www.trendmciro.com/activate | www.trendmicro/activate | trend micro activation

    ReplyDelete
  13. Set up Microsoft 365 through microsoft365.com/setup and use Microsoft apps such as Word, PowerPoint, and Excel, online storage, cloud-connected features, and more. If you have a Microsoft account associated with Outlook, OneDrive, Xbox Live, or Skype, you can proceed with Microsoft 365 login
    | microsoft365.com/setup | www.hp.com/go/wirelessprinting

    ReplyDelete
  14. Such a great blog site: There's magic in your words. Whenever you write a blog, you grab the attention of your audience. Your efforts should be appreciated. Eligible citizens are required to complete the Turkish Visit Visa online application form to enter the country.

    ReplyDelete
  15. A new purchased Canon printer needs to be set up. First, ensure you’ve installed the latest drivers and software from a valid siteThe wireless and wired models require drivers and software; therefore, each has to undergo one important installation process.
    https //ij.start.cannon,
    http //ij.start.canon

    ReplyDelete
  16. The site is based on the computer operating system where you can find an online manual and guide. ij.start.canon |
    ij.start.cannon

    ReplyDelete
  17. Visit canon’s official site – ij.start.canon/ts3122 and download appropriate software and drivers one windows PC. Or if you have a CD, install it. ij.start canon is the manufacturer's site to download Canon printer drivers. Install and set up Canon Printer from ij.start.canon and obtain high-quality printing documents straightforwardly.

    ReplyDelete
  18. The setup process for every Canon model is almost similar, however the download through https //ij.start.cannon or http //ij.start.cannon and installation process may differ.

    ReplyDelete
  19. The wireless and wired models require drivers and software; therefore, each has to undergo one important installation process. If your printer model isn’t set up yet, you can get started now.
    https //ij.start.cannon,
    http //ij.start.canon,

    ReplyDelete
  20. You can easily download the driver according to your model as ij start canon is designed with a user-friendly interface. canonij printer provides an answer for all type of canon printer problems by which you can undoubtedly figure out how to set up from ij.start.canon and improve insight.

    ij.start.canon ,
    ij.start canon

    ReplyDelete
  21. I want to say thanks for beautiful blog sharing with us. Your blog really great resource to update my knowledge 야한동영상

    Also feel free to visit may webpage check this link
    야설

    ReplyDelete
  22. What a good blog you have here. Please update it more often. This topics is my interest. Thank you. 일본야동

    Also feel free to visit may webpage check this link
    한국야동

    ReplyDelete
  23. This is a smart blog. I mean it. You have so much knowledge about this issue, and so much passion. You also know how to make people rally behind it, obviously from the responses. 한국야동닷컴

    Also feel free to visit may webpage check this link
    국산야동

    ReplyDelete
  24. I am browsing this website daily and get good facts from here all the time. Aw, this was a really nice post. Kenya welcomes foreigners to the country for various purposes. Obtaining a Kenyan visa is quite convenient and reading about Kenyan visa invitation letter.

    ReplyDelete
  25. Thanks for sharing this! All the best!

    ReplyDelete
  26. When I read your article on this topic, the first thought seems profound and difficult. There is also a bulletin board for discussion of articles and photos similar to this topic on my site, but I would like to visit once when I have time to discuss this topic. sòng bạc


    ReplyDelete
  27. Canon IJ Network Tool is a toolkit software with the options to keep a check on most of your Canon printer network settings and adjust them according to your requirements. The Canon IJ Network tool will get you through the network settings uninterruptedly. Canon IJ Printer Utility is a complete software package meant to adjust and modify the configurations of your printer device as per the requirement.


    ReplyDelete
  28. Canon IJ Network Tool is a toolkit software with the options to keep a check on most of your Canon printer network settings and adjust them according to your requirements.

    Canon IJ Printer Utility is a complete software package meant to adjust and modify the configurations of your printer device as per the requirement. It is one of the essential utility software offered by Canon to ease your printer configuration alteration using a few clicks.

    ReplyDelete

  29. Canon.com/ijsetup/mg2500
    is the Canon support link to get you the latest printer drivers and software to set up your PIXMA MG2500 or MG2520.

    ReplyDelete
  30. It is one of the best information which you are providing. I love your blog because it totally full of informative posts. You are nice one and thanks for sharing. It is very nice and informative blog through which we gain a lot information for our some purpose. you did a great job and i appreciate your effort. Very nice share and keep on sharing. Get best App Developer Dubai service you visit here site for more info.

    ReplyDelete
  31. The wireless and wired models require drivers and software; therefore, each has to undergo one important installation process. If your printer model isn’t set up yet, you can get started now.
    http //ij.start.canon,
    https //ij.start.cannon,

    ReplyDelete
  32. You can easily download the driver according to your model as ij start canon is rinter problems by which you can undoubtedly figure out how to set up from ij.start.canon and improve insight.

    ij.start.canon ,
    ij.start canon

    ReplyDelete
  33. The setup process for every Canon model is almost similar, however the download through https //ij.start.cannon or http //ij.start.cannon and installation process may differ. All-in-one Canon Inkjet printers are suitable for home, business, school, and others to improve productivity. Depending on your requirement, it offers a type printer including PIXMA, SELPHY, MAXIFY, etc. Some factors need to be in mind while choosing an inkjet printer for you. Later, you can easily set up your Canon printer through drivers from canon.com/ijsetup , wireless connection, USB, and a few components. | ij.start.cannon


    ReplyDelete
  34. Wow.. Very informative article thanks for sharing please keep it up.. India visa price is changed in case of emergency. Foreign travelers should check the visa price.

    ReplyDelete
  35. Wow, that's what I was looking for, what stuff! Present here on this website, thanks to the admin of this website.. Turkish businessperson visa , You can apply for a Turkish visa online through the e Visa Turkey website. It takes 3-5 minutes to fill in the online application.

    ReplyDelete
  36. Decent data, profitable and phenomenal outline, as offer well done with smart thoughts and ideas, bunches of extraordinary data and motivation, both of which I require, on account of offer such an accommodating data here 토토사이트

    ReplyDelete

  37. This is really an inspiring and helpful article. I am fully satisfied with your effort .There is no need to meet any embassy, especially to apply for Indian visa from Australia. You can easily apply online and easily get your evisa on your email id

    ReplyDelete
  38. I love what you guys do too. Much clever work and reporting! Keep up the good work guys... Turkey visa for Iraq Citizens, Iraq Citizens can be apply online Turkey visa via Turkish visa website.

    ReplyDelete
  39. It is my first visit to your blog, and I am very impressed with the articles that you serve. Give adequate knowledge for me. Thank you for sharing useful material. I will be back for the more great post. 먹튀검증사이트 But by chance looking at your post solved my problem! I will leave my blog, so when would you like to visit it?!

    ReplyDelete
  40. Good post. I learn something new and challenging on blogs I stumbleupon on a daily basis. 토토사이트

    ReplyDelete
  41. The information which you have provided is very helpful and precise. 바카라사이트닷컴

    ReplyDelete
  42. Absolutely love all your blogs and enjoy reading the story behind each. 파워볼사이트닷컴

    ReplyDelete
  43. This is new knowledge for me, I am excited about it. thanks....tourist visa India, You can get an online tourist visa for India and visit the beautiful religious places of India etc.

    ReplyDelete
  44. Nice...
    It's a very powerful article. I really like this post. Thank you so much for sharing this.

    Canon IJ Scan Utility
    ij start canon ts3122

    ReplyDelete
  45. Make sure the surface where you are placing the printer is clean and close to Pc or laptop.Check the shipped components with your inkjet printer
    http //ij.start.canon,
    https //ij.start.cannon

    ReplyDelete
  46. Thank you very much for writing this amazing article. I hope you will share more in the future. With the help of the government of India, getting an India tourist visa for US citizens has been easy and convenient done online.

    ReplyDelete
  47. Canon IJ Network Tool you to watch out for the points mentioned below considerably:
    If you want a LAN setup for your printer, you should have the required connectivity equipment such as Network Access Point (Router/ Modem), LAN Cable etc.To get your IJ Network tool to work, you first need to check your printer’s connectivity via LAN connection. Next, check your LAN cable carefully.Canon IJ Printer Utility you can alter your printer’s ink cartridge settings according to your requirements. If any ink cartridges get empty, you can continue to print with other available cartridges after changing the settings.

    ReplyDelete
  48. Canon IJ Network Tool is a toolkit software with the options to keep a check on most of your Canon printer network settings and adjust them according to your requirements.

    Canon IJ Printer Utility is a complete software package meant to adjust and modify the configurations of your printer device as per the requirement. It is one of the essential utility software offered by Canon to ease your printer configuration alteration using a few clicks.


    Canon.com/ijsetup/mg2500
    is the Canon support link to get you the latest printer drivers and software to set up your PIXMA MG2500 or MG2520.

    ReplyDelete
  49. Visit ij.start canon | ij.start.cannon and find out the best way to download Canon printer drivers. Canon printers are ideal for every situation wherever you need a document, paper, or photo print or even if you wish to scan, fax, and do more.

    All-in-one Canon Inkjet printers are suitable for home, business, school, and others to improve productivity. You can easily set up your Canon printer through drivers from Canon.com/ijsetup | canon.come/ijsetup , wireless connection, USB, and a few components.

    ReplyDelete
  50. ij.start canon helps to set up canon printer. It’s the online support platform to download and install canon printer drivers, firmware, and software ij.start.canon , Canon IJ Network Tool

    ReplyDelete
  51. Hey sir, This great article has really piqued my interest. Thank you! How much does an e visa cost for India & how much is an e visa for India? India visa cost depends on your nationality and your visa type.

    ReplyDelete
  52. Thank you.. Get the azerbaijan visas through online e visa application to travel to Azerbaijan. Just follow 3 steps, fill application, upload document and make online payment for Azerbaijan e visa.

    ReplyDelete
  53. I read your excellent post. After reading this post, i really appreciate your effort and my request is to please share us more post in future. Keep it up. It is one of the best information which you are providing. I love your blog because it totally full of informative posts. You are nice one and thanks for sharing. Get best SEO Company Dubai service of marketing you visit here site for more info.

    ReplyDelete
  54. Certainly a fantastic piece of work. It has relevant information. Thanks for posting this. Your blog is so interesting and very informative. Thanks sharing. Definitely a great piece of work. Thanks for your work. Get best Freelance Jobs UAE provide good services visit here site for more info from Smart Job.

    ReplyDelete
  55. Cricut Design Space The Cricut Design Space download is available in Apple’s app store as well as Android’s Google Play Store. For Windows and Mac, the Cricut Design Space download is possible via cricut Cricut Design Space Download

    ReplyDelete
  56. Visit ij.start.cannon | ij.start canon and find out the best way to download Canon printer drivers. Canon printers are ideal for every situation wherever you need a document, paper, or photo print or even if you wish to scan, fax, and do more.

    Use dragon naturally speaking to turn speech into text quickly. Download the latest version and work in Voice mode and be Hands-Free. Dictate Documents and complete the transcription faster.

    ReplyDelete
  57. The tour bus was packed with teenage girls heading toward their next adventure.ThingsyoudoforbeautyEvery manager should be able to recite at least ten nursery rhymes backward.INDIA'S BEST OFF SITE SEO PROVIDER CHEAPEST AND FASTEST OFF SITE SEO SERVICE IN INDIA infocompanion educatijhnI'm confused: when people ask me what's up, and I point, they groan.

    ReplyDelete
  58. https://docs.google.com/document/d/19CZXabVzQ-1SK3DcWDBv8UBP5EdQMIRRqsebFPVUfl8/edit?usp=sharing
    https://docs.google.com/presentation/d/1MXPImkOapg4iH3n3vF37YaMnJUM6rwo-VbRIiO5Ouzw/edit?usp=sharing
    https://docs.google.com/forms/d/e/1FAIpQLSfl3U4Xo-jspkxoTf0jqhKP5fXNKpD2l6y23nJZKjax30_PIw/viewform
    https://docs.google.com/spreadsheets/d/161S6GlaDhexbyRaI_Ua19G9mLPURwfvVb-ofjgh6M1c/edit?usp=sharing

    ReplyDelete
  59. I trust all the ideas that you have offered for your post. And thank you sir. We have good news for foreign travelers who have been wanting to visit India for a long time. Citizens of 156 countries can plan their travel as per the new visa guideline standards. You can apply now online but firstly do you know India tourist visa covid requirement. You can read tourist visa India covid requirement via our Indian visa blog.

    ReplyDelete
  60. Thanks for sharing information related to Java. It is brilliant blog. I like the way you express information to us. It is very useful for me. You are doing great job and thanks for sharing. Get best SEO Company Dubai service of marketing you visit here site for more info.

    ReplyDelete