Why Storing Server Metadata Immutably Has Become A Must

In the world of software development and system administration, storing who did what and when is crucial information that needs to be handled with care. When things go wrong, or when data gets stolen, auditors, regulators, and law enforcement will ask for this information, and the number one question is: can you trust these logs?

One important aspect of data management is ensuring its immutability. Immutability refers to the state of data that cannot be changed once it is created. In this blog post, we will explore why it is crucial to store certain types of data, such as login metadata, configuration change metadata, and database access information, in an immutable manner.

Server Access Metadata (Login Data)

Login metadata includes information about user logins, such as timestamps, IP addresses, and user agents. Storing this data immutably provides several benefits. Firstly, it helps with security and auditing. By storing login metadata in an immutable database like our immudb Vault offering, any unauthorized changes or tampering attempts can be easily detected. Additionally, untampered login metadata can be used for forensic analysis in case of security incidents or investigations. It provides a reliable source of information that cannot be altered or manipulated.

Configuration Change Metadata

Configuration changes are an integral part of system administration. They involve modifying settings, parameters, or configurations of various software components. Storing configuration change metadata immutably is crucial for maintaining a reliable audit trail. Immutable metadata allows system administrators to track and review changes made to the system over time. It helps troubleshoot issues, identify the root cause of problems, and revert to previous configurations if necessary. You can even go one step further and store the actual changes as a new version. That allows for the correlation of changes and events.

Database Access

Database access information, such as user credentials and connection details, should be stored immutably to ensure data integrity and security. Furthermore, database metadata, such as when schemas are changed, tables dropped, etc., is all information of the highest importance that needs to be stored in a tamper-proof way. Immudb Vault makes it easy to store this information in a tamper-proof way. Immutable storage prevents unauthorized modifications to the access information, reducing the risk of data breaches or unauthorized access to sensitive data. It also helps in maintaining a clear record of who accessed the database and when. This can be useful for compliance purposes or investigating any suspicious activities.

Other Important Meta-Data

Similar data types, such as system logs, error logs, or application logs, should also be stored immutably. Immutable storage ensures that these logs cannot be modified or tampered with, preserving their integrity and reliability. Immutable logs are essential for troubleshooting issues, analyzing system behavior, and identifying patterns or trends. They provide a historical record of events that can be invaluable in diagnosing problems or understanding system behavior over time.

Examples

We are pointing out in this blog post two examples of system extensions that store important metadata in a tamper-proof, immutable cloud database like immudb Vault:

This repository shows an approach to store all access data in immudb Vault

By running the install script, a new rule will be set up in the rsyslog configuration of your Linux server (works with RHEL, Ubuntu, and Debian) which then stores SSH login data in the immudb Vault.

To change the configuration of rsyslogd so that every SSH login is sent with curl to a remote cloud database, you can follow these steps:

1. Open the rsyslog configuration file using a text editor. The file is typically located at `/etc/rsyslog.conf`.

2. Locate the section that starts with `# The authpriv file has restricted access`. This section handles the logging of authentication-related messages.

3. Uncomment the line that starts with `authpriv.*`. Remove the leading `#` to enable the line.

4. Add the following line below the uncommented `authpriv.*` line to specify the action to be taken for SSH login events:

   authpriv.* action(type="omprog" binary="/usr/bin/curl -X POST -d @-" template="RSYSLOG_TraditionalFileFormat")
   
   This line uses the `omprog` action to execute the `curl` command with the specified options. It sends the log message to a remote cloud database using a POST request.

5. Save the changes to the rsyslog configuration file.

6. Restart the rsyslog service to apply the changes with:

   sudo systemctl restart rsyslog

The proper rsyslog template would be:

module(load="omprog")

template(name="json-template" type="list") {
    constant(value="{")
    constant(value="\"timestamp\":\"") property(name="timereported" dateFormat="rfc3339")
    constant(value="\",\"message\":\"") property(name="msg" format="json")
    constant(value="\",\"host\":\"") property(name="hostname")
    constant(value="\",\"severity\":\"") property(name="syslogseverity-text")
    constant(value="\",\"facility\":\"") property(name="syslogfacility-text")
    constant(value="\",\"syslogtag\":\"") property(name="syslogtag")
    constant(value="\"}\n")
}

authpriv.* action(type="omprog"
            binary="/usr/bin/forward-logs.sh"
            template="json-template")

`/usr/bin/forward-logs.sh`:

while read line; do

curl -X 'PUT' 'https://vault.immudb.io/ics/api/v1/ledger/default/collection/default/document' -H 'accept:
application/json' -H 'X-API-Key: default.-uiriTRuJVQsKACSuXhMZQ.LXUeKJ8XCCFNMkDIsi-tvUn-OUWzVOfOMt7VpP_ln6Ix9w9Q' -H 'Content-Type: application/json' -d "$line"

Conclusion

This article has shed light upon the significance of immutably storing specific data types, such as login metadata, configuration change data, and database access information. Immutable storage not only fortifies security and auditability but also enables forensic analysis, system troubleshooting, and historical event correlation. Embracing this approach is not merely a best practice; it's an essential cornerstone of modern data management, safeguarding the integrity and trustworthiness of vital system records.

Explore immudb Vault to learn more about immutable data storage.