Open-source IoT Platform

Device management, data collection, processing and visualization
for your IoT projects
Learn about Thingsboard

Wednesday, September 16, 2020

Deleting timeseries of deleted ThingsBoard entities

    As of ThingsBoard v.2.5.4/3.1.1, telemetry and attributes of entities are not automatically deleted when entities themselves are deleted, which causes performance degradation (because of decreasing query speed) and filling up of the disk space . Lower are queries for the PostgreSQL database to delete telemetry and attributes.

    Deleting timeseries of previously deleted entities:


 DELETE * FROM ts_kv WHERE entity_id NOT IN (SELECT id FROM device) AND entity_id NOT IN (SELECT id FROM asset) AND entity_id NOT IN (SELECT id FROM customer) AND entity_id NOT IN (SELECT id FROM tenant) AND entity_id NOT IN (SELECT id FROM entity_group) AND entity_id NOT IN (SELECT id FROM integration) AND entity_id NOT IN (SELECT id FROM converter) AND entity_id NOT IN (SELECT id FROM entity_view) AND entity_id NOT IN (SELECT id FROM tb_user) AND entity_id NOT IN (SELECT id FROM dashboard) AND entity_id NOT IN (SELECT id FROM rule_chain) AND entity_id NOT IN (SELECT id FROM role); 

    Deleting attributes of previously deleted entities:


 DELETE * FROM attribute_kv WHERE entity_id NOT IN (SELECT id FROM device) AND entity_id NOT IN (SELECT id FROM asset) AND entity_id NOT IN (SELECT id FROM customer) AND entity_id NOT IN (SELECT id FROM tenant) AND entity_id NOT IN (SELECT id FROM entity_group) AND entity_id NOT IN (SELECT id FROM integration) AND entity_id NOT IN (SELECT id FROM converter) AND entity_id NOT IN (SELECT id FROM entity_view) AND entity_id NOT IN (SELECT id FROM tb_user) AND entity_id NOT IN (SELECT id FROM dashboard) AND entity_id NOT IN (SELECT id FROM rule_chain) AND entity_id NOT IN (SELECT id FROM role); 

    However, in PostgreSQL deleted rows are not physically removed from the table, they remain present until the VACUUM command is done, as described here: https://www.postgresql.org/docs/11/sql-vacuum.html. In order to reclaim the disk space VACUUM commands with ANALYZE parameter (for statistics refreshing) needs to be executed:


 VACUUM ANALYZE attribute_kv; 

    And if reclaiming space is critical, the parameter FULL should be added (please note that it requires as much free disk space as the table currently takes up):


 VACUUM FULL ANALYZE attribute_kv; 

    In order to find the list of telemetry tables to be VACUUM’ed, the following command needs to be executed:

SELECT table_name FROM information_schema.tables WHERE table_schema = 'public' AND table_name like 'ts_kv%' ORDER BY table_name; 

    Then the resulting list of tables is put in the VACUUM ANALYZE command:

    In this example the command looks like this:

VACUUM FULL ANALYZE ts_kv, ts_kv_2004_01, ts_kv_2020_08, ts_kv_2020_09, ts_kv_dictionary, ts_kv_indefinite, ts_kv_latest; 

    Please be aware that:

1.You need to put the list of tables from the output of the previous SELECT query, not just copy the previous VACUUM command.

2.Tables are not available when you perform a VACUUM command (all incoming read and write queries are put in a queue), so the best solution would be to perform it in the (at least relative) downtime.



Your feedback

If you found this article interesting, please leave your feedback in the comments section, post questions or feature requests on the forum and “star” our project on the github in order to stay tuned for new releases and tutorials.

17 comments :

  1. I would love this command for Cassandra Database! Really useful.

    ReplyDelete

  2. Escorts Service in Connaught Place, here are incredibly evolved and they appreciate the customer’s needs since this city has every sort of individual and the fight is there for everyone. Our escort knows this reality, so they give you 100% satisfaction guaranteed. We Provide bestCall Girls in Vasant Kunj that will you never forget and come again and again. Our Agency maintain the quality and an immense collection which contains a range of categories of escort girls.
    Russian Escorts in Delhi
    Foreigner Call Girls in Gurgaon
    Housewife Escorts in Mahipalpur
    Call Girls in Mahipalpur

    ReplyDelete
  3. Attractive section of content. I simply stumbled upon your blog and in accession capital to assert that I acquire in fact enjoyed
    account your blog posts. Anyway I’ll be subscribing 풀싸롱


    on your augment or even I success you access constantly quickly.

    ReplyDelete
  4. It has fully emerged to crown Singapore's southern shores and undoubtedly placed her on the global map of residential landmarks. I still scored the more points than I ever have in a season for GS. I think you would be hard pressed to find somebody with the same consistency I have had over the years so I am happy with that. 먹튀검증

    ReplyDelete
  5. Thanks for your post. The article is neatly organized with the information I want, so there are many things to refer to. Bookmark this site and visit often in the future. Thanks again.^^ keonhacai

    ReplyDelete
  6. 192.168.1.1 is a private IP address used to login the admin panel of a router. 192.168.l.l is pre-specified by the router companies as the default gateway address and can be used to make different changes to the router’s settings. 192.168.0.1

    ReplyDelete

  7. Marvelous, what a web site it is! This weblog provides helpful data to us, keep it up. Feel free to visit my website;
    야설

    ReplyDelete
  8. I was searching some blogs to read on google and found this blog post page. I must say it is very informative as well as interesting. Thanks to the author of this post/page for writing such wonderful lines. Feel free to visit my website; 한국야동

    ReplyDelete
  9. Its like you read my mind! You seem to know a lot about this, like you wrote the book in it or something. I think that you can do with a few pics to drive the message home a bit, but instead of that, this is excellent blog. A great read. I’ll definitely be back. Feel free to visit my website;
    국산야동

    ReplyDelete

  10. Great info. Thank you for sharing this fascinating information together. I am so blessed to discover this. Feel free to visit my website;
    일본야동

    ReplyDelete

  11. A very awesome blog post. We are really grateful for your blog post. You will find a lot of approaches after visiting your post Also visit my website:
    일본야동

    ReplyDelete
  12. I trully appretiate your work and tips given by you is helpful to me. I will share this information with my family & friends. This is a great website, keep the positive reviews coming. This is a great inspiring .pay to write my assignment I am pretty much pleased with your good work. You put really very helpful information. I am looking to reading your next post. !!!!

    ReplyDelete
  13. Drift Hunters game is a racing game with unllimited money. You can free to buy all the items on the shop. This game will be easy for you. Enjoy the game!

    ReplyDelete
  14. people genuinely need to take a gander at this and understand this side of your story. It's surprising you're not more standard since you undeniably have the gift Thanks for bestowing this best stuff to us! Keep on sharing! I'm new in blog writing.All types composes and posts are not helpful for the readers.Here the author is giving worthy insights and thoughts to each and every perusers through this article. Quality of the substance is the essential help with theory uk segment of the blog and this is the technique for yellowstone blue coat forming and presenting. It was an amazing post without a doubt. I totally liked scrutinizing it in my early afternoon. coat
    Will come and visit this blog even more consistently. Thankful for sharing. There are such endless fun and empowering exercises and experiences all through the planet that I thought I 'd set up a summary of most adored exercises

    ReplyDelete