Datadog standards
Because multiple business units will utilize Datadog, the Datadog Standards document has been created to maintain consistency across the organization. If you are a business unit implementing Datadog, please be sure you are conforming to these standards.
How do you access it?
You can access us5.datadoghq.com using the “Datadog-US5” tile in Okta or follow this direct link. A new account will be automatically provisioned for you with a standard JH Standard User access role.
Links for training
A short video going over the basics here Passcode is 9f0rs+?Z
Although not training, Useful Datadog Searches is helpful when using Datadog.
Rehydrating logs
By default, only 15 days are kept indexed in Datadog. All logs sent to Datadog (even if they are excluded from being indexed and aren’t searchable) are also sent to Banno’s own Azure storage accounts. These storage accounts currently do not have any retention set. It is possible to re-index those stored logs that are past the 15 day retention back into Datadog.
As we add more business units to Datadog, it’s likely logs will reside in different storage accounts. Contact a member of @Banno/datadog-administrators for more information on setting up a new archive.
Cost of rehydrating logs
The cost of rehydrating logs is similiar to ingesting the log in the first place. Costs are based on the scan size as well as how many events are added back to the index. Scans are priced at $0.10/GB. Adding the logs to the index costs as well, but this is a negotiated rate. Try to not add too many things back to the index (try to stick to <1000 per restore if possible).
An example:
I have a request ID, and institution ID, and a user ID. I don’t know when the event happened, so I’ll just use a 4 day window.
In the above case, I can hit the “estimate” button on the “archive scan size” section of the rehydration window to see an approximate of how much data will be scanned. If the “4 day window” returns a 12TB index, that’s at least $1200 to restore the required logs. Limiting time windows as much as possible is a great way to reduce costs.
For Banno: Archived logs only start at July 29, 2021; no logs before then can be rehydrated.
Datadog has their own documentation on how to rehdyrate logs
Some things to note:
- After submitting the request to Datadog to rehydrate logs from an archive into a historical view, by default Datadog will email you when it’s complete. I have found that Gmail will put the email into my spam label. Make sure to check there and create a rule so that emails from
dtdg.coare not marked as spam. - Historical views will only stick around for 15 days after creation. If it’s needed after 15 days, the historical view will have to be recreated.
- After the historical view is created, you can either use the link in the email (easiest) or you have to manually select the historical view to search it via selecting the Index pane on the left sidebar of the Log search view.
For guided tour on how to rehdrate logs and what to expect, a video is here Passcode is %WTF1adG.
FAQ
- How long do we keep our logs indexed? Most of our logs are indexed for 15 days on the “main” index. As more business units join Datadog, we will likely increase the amount of indices. To see a list of current indices, visit this page. Contact @Banno/datadog-administrators to add a new index.